>>The cube presents, Coon and cloud native con Europe 22, brought to you by the cloud native computing foundation. >>Welcome to Valencia Spain in co con cloud native con Europe, 2022. I'm Keith Townsend with my cohos on Rico senior. Etti senior it analyst at gig home. Exactly 7,500 people I'm told en Rico. What's the flavor of the show so far, >>It's a fantastic mood. I mean, I found a lot of people wanting to track talk about what they're doing with Kubernetes, sharing their, you know, stories, some word stories that meet tough. And you know, this is where you learn actually, because we had a lot of zoom calls, webinar and stuff, but it is when you talk a video, oh, I did it this way and it didn't work out very well. So, and, and you start a conversation like this that is really different from learning from zoom. When, you know, everybody talks about things that working well, they did it, right. No, it's here that you learn from other experiences. >>So we're talking to amazing people the whole week, talking about those experiences here on the queue, fresh on the queue for the first time, Chris Vos, senior software engineer at Microsoft Xbox, Chris, welcome to the queue. >>Thank you so much for having >>Me. So first off, give us a high level picture of the environment that you're running at Microsoft. >>Yeah. So, you know, we've got 20, well probably close to 30 clusters at this point around the globe, you know, 700 to a thousand pods per cluster, roughly. So about 22,000 pods total. So yeah, it's pretty pretty sizable footprint and yeah. So we've been running on Kubernetes since 2018 and well actually might be 2017, but anyways, so yeah, that, that's kind of our, our footprint. >>Yeah. So all of that, let's talk about the basics, which is security across multiple I'm assuming containers, work, microservices, et cetera. Why did you and the team settle on link or do >>Yeah, so previously we had our own kind of solution for managing TLS certs and things like that. And we found it to be pretty painful pretty quickly. And so we knew, you know, we wanted something that was a little bit more abstracted away from the developers and, and things like that that allowed us to move quickly. And so we began investigating, you know, solutions to that. And a few of our colleagues went to Cuban in San Diego in 2019 cloud native con as well. And basically they just, you know, sped it all up. And actually funny enough, my, my old manager was one of the people who was there and he went to the link D booth and they had a thing going that was like, Hey, get set up with MTLS in five minutes. And he was like, this is something we want to do, why not check this out? And he was able to do it. And so that, that put it on our radar. And so yeah, we investigated several others and Leer D just perfectly fit exactly what we needed. >>So, so in general, we are talking about, you know, security at scale. So how you manage security to scale and also flexibility, right. But you know, what is the you, this there, you told us about the five minutes to start using there, but you know, again, we are talking about word stories. We talk about, you know, all these. So what, what, what kind of challenges you found at the beginning when you start adopting this technology? >>So the biggest ones were around getting up and running with like a new service, especially in the beginning, right. We were, you know, adding a new service almost every day. It felt like. And so, you know, basically it took someone going through a whole bunch of different repos, getting approvals from everyone to get the SEARCHs minted, all that fun stuff, getting them put into the right environments and in the right clusters to make sure that, you know, everybody is talking appropriately. And just the amount of work that, that took alone was just a huge headache and a huge barrier to entry for us to, you know, quickly move up the number of services we have. So, >>So I'm, I'm trying to wrap my head around the scale of the challenge. When I think about certification or certificate management, I have to do it on a small scale and the, the, every now and again, when a certificate expires, it is just a troubleshooting pain. Yes. So as I think about that, it costs, it's not just certificates across 22,000 pods or it's certificates across 22,000 pods in multiple applications. How were you doing that before link D like, what was the, what and what were the pain points? Like? What happens when a certificate either fails or expired up not, not updated? >>So, I mean, to be completely honest, the biggest thing is we're just unable to make the calls, you know, out or, or in, based on yeah. What is failing basically. But, you know, we saw essentially an uptick in failures around a certain service and pretty quickly, I pretty quickly, we got used to the fact that it was like, oh, it's probably a cert expiration issue. And so we tried, you know, a few things in order to make that a little bit more automated and things like that, but we never came to a solution that like didn't require every engineer on the team to know essentially quite a bit about this, just to get into it, which was a huge issue. >>So talk about day two after you've deployed link D how did this alleviate software engineers and what was like the, the benefits of now having this automated way of managing >>Certs? So the biggest thing is like, there is no touch from developers, everyone on our team. Well, I mean, there are a lot of people who are familiar with security and certs and all of that stuff, but no one has to know it. Like it's not a requirement. Like for instance, I knew nothing about it when I joined the team. And even when I was setting up our newer clusters, I knew very little about it. And I was still able to really quickly set up blinker D, which was really nice. And, and it's been, you know, essentially we've been able to just kind of set it and not think about it too much. Obviously, you know, there are parts of it that you have to think about. We monitor it and all that fun stuff, but, but yeah, it's been pretty painless almost day one. It took a lot, a long time to trust it for developers. You know, anytime there was a failure, it's like, oh, could this be link or D you know, but after a while, like now we don't have that immediate assumption because people have built up that trust, but >>Also you have this massive infrastructure, I mean, 30 cluster. So I guess that it's quite different to manage a single cluster and 30. So what are the, you know, consideration that you have to do to install this software on, you know, 30 different cluster manage different, you know, versions probably etcetera, etcetera, et cetera. >>So, I mean, you know, the, the, as far as like, I guess, just to clarify, are you asking specifically with Linky or are you just asking in more in general? Well, >>I mean, you, you can take the, the question in the, in two ways, so, okay. Yeah. Yes. Link in particular, but the 30 cluster also quite interesting. >>Yeah. So, I mean, you know, more generally, you know, how we manage our clusters and things like that. We have, you know, a CLI tool that we use in order to like, change context very quickly and switch and communicate with whatever cluster we're trying to connect to and, you know, are we debugging or getting logs, whatever. And then, you know, with link D it's nice because again, you know, we, we, aren't having to worry about like, oh, how is this cert being inserted in the right node or, or not the right node, but in the right cluster or things like that. Whereas with link D we don't, we don't really have that concern when we spin up our, our clusters, essentially we get the root certificate and, and everything like that packaged up, passed along to link D on installation. And then essentially there's not much we have to do after that. >>So talk to me about your upcoming coming section here at Q con what's the, what's the high level talking points? Like what, what will attendees learn? >>Yeah. So it's, it's a journey. Those are the sorts of talks that I find useful. Having not been, you know, I, I'm not a deep Kubernetes expert from, you know, decades or whatever of experience, but I think >>Nobody is >>Also true. That's another story. That's a, that's, that's a job posting decades of requirements for >>Of course. Yeah. But so, you know, it, it's a journey it's really just like, Hey, what made us decide on a service mesh in the first place? What made us choose link D and then what are the ways in which, you know, we, we use link D so what are those, you know, we use some of the extra plugins and things like that. And then finally, a little bit about more, what we're gonna do in the future. >>Let's talk about not just necessarily the future as in two or three days from now, or two or three years from now. Well, the future after you immediately solve the, the low level problems with link D what were some of the, the surprises, because link D in service me in general has have side benefits. Do you experience any of those side benefits as well? >>Yeah, it's funny, you know, writing the, the blog post, you know, I hadn't really looked at a lot of the data in years on, you know, when we did our investigations and things like that. And we had seen that we like had very low latency and low CPU utilization and things like that. And looking at some of that, I found that we were actually saving time off of requests. And I couldn't really think of why that was, and I was talking with someone else and the biggest, unfortunately, all that data's gone now, like the source data. So I can't go back and verify this, but it, it makes sense, you know, there's the availability zone routing that linker D supports. And so I think that's actually doing it where, you know, essentially if a node is closer to another node, it's essentially, you know, routing to those ones. So when one service is talking to another service and maybe on they're on the same node, you know, it, it short circuits that, and allows us to gain some, some time there. It's not huge, but it adds up after, you know, 10, 20 calls down the line. Right. >>In general. So you are saying that it's smooth operations in, in ATS, very, you know, simplifying your life. >>And again, we didn't have to really do anything for that. It, it, it handled that for it was there. Yeah. Yep. Yeah, exactly. >>So we know one thing when I do it on my laptop, it works fine when I do it with across 22,000 pods, that's a different experience. What were some of the lessons learned coming out of KU con 2018 in San Diego was there? I wish I would've ran to the microphone folks, but what were some of the hard lessons learned scaling link D across the 22,000 nodes? >>So, you know, the, the first one, and this seems pretty obvious, but was just not something I knew about was the high availability mode of link D so obviously makes sense. You would want that in a, you know, a large scale environment. So like, that's one of the big lessons that like, we didn't ride away. No. Like one of the mistakes we made in, in one of our pre-production clusters was not turning that on. And we were kind of surprised. We were like, whoa, like all of these pods are spinning up, but they're having issues like actually getting injected and things like that. And we found, oh, okay. Yeah, you need to actually give it some, some more resources, but it's still very lightweight considering, you know, they have high availability mode, but it's just a few instances still. >>So from, even from a, you know, binary perspective and running link D how much overhead is it? >>That is a great question. So I don't remember off the top of my head, the numbers, but it's very lightweight. We, we evaluated a few different service missions and it was the lightest weight that we encountered at that point. >>And then from a resource perspective, is it a team of link D people? Is it a couple of people, like how >>To be completely honest for a long time, it was one person, Abraham who actually is the person who proposed this talk. He couldn't make it to Valencia, but he essentially did probably 95% of the work to get a into production. And then this was before we even had a team dedicated to our infrastructure. And so we have, now we have a team dedicated, we're all kind of Linky folks, if not Linky experts, we at least can troubleshoot basically. And things like that. So it's, I think a group of six people on our team, and then, you know, various people who've had experience with it >>On other teams, but I'm not dedicated just to that. >>I mean, >>No one is dedicated just to it. No, it's pretty like pretty light touch once it's, once it's up and running, it took a very long time for us to really understand it and, and to, you know, get like, not getting started, but like getting to where we really felt comfortable letting it go in production. But once it was there, like, it is very, very light touch. >>Well, I really appreciate you stopping by Chris. It's been an amazing conversation to hear how Microsoft is using a open source project. Exactly. At scale. It's just a few years ago, when you would've heard the concept of Microsoft and open source together and like, oh, that's just, you know, but >>They have changed a lot in the last few years now, there are huge contributors. And, you know, if you go to Azure, it's full of open source stuff, every >>So, yeah. Wow. The Cuban 2022, how the world has changed in so many ways from Licia Spain, I'm Keith Townsend, along with a Rico senior, you're watching the, the leader in high tech coverage.

>>The cube presents, Coon and cloud native con Europe 22, brought to you by the cloud native computing foundation. >>Welcome to vincia Spain in Coon cloud native con Europe, 2022. I'm Keith towns alongside en Rico senior. Etti senior it analyst for giong welcome back to the show en >>Rico. Thank you again for having me here. >>First impressions of QAN. >>Well, great show. As, as I mentioned before, I think that we are really in this very positive mode of talking with each other and people wanting to see, you know, the projects, people that build the projects at it's amazing. I mean, a lot of interesting conversation in the show floor and in the various sessions, very positive move. >>So this is gonna be a fun one. We have some amazing builders on the show this week, and none other than William Morgan, CEO of buoyant. What's your role in the link D project? >>So I was one of the original creators of link D but at this point I'm just the, the beautiful face of the project. >>Speaking of beautiful face of the project, linker D just graduated from as a CNCF project. >>Yeah, that's right. So last year we, we became the first service mesh to graduate in the CNCF. Very proud of that. And that's thanks, you know, largely to the incredible community around Linky that is just excited about the project and, you know, wants to talk about it and wants to be involved. >>So let's talk about the significance of that link D not the only service mesh project out there. Talk to me about the level effort to get it to the point that it's graduated. That's you don't see too many projects graduating CNCF in general. So let's talk about kind of the work needed to get Nier D to this point. >>Yeah. So, you know, the, the, the bar is high and it's mostly a measure, not necessarily of like the, the project being technically good or bad or anything, but it's really a measure of maturity of the community around it. So is it being adopted by organizations that are really relying on it in a critical way? Is it, you know, being adopted across industries, you know, is it having kind of a significant impact on the cloud native community? And so for us, you know, there was the, the work involved in that was really not any different from the work involved in, in kind of maintaining ity and growing the community in the first place, which is you try and make it really useful. You try and make it really easy to get started with you, try and be supportive and to, you know, have a, a friendly and welcoming community. And if you do those things and, you know, you kind of naturally get yourself to the point where it's a, it's a really strong community full of people who are excited about it. >>So from the of view of, you know, users adopting the, this technology, so we are talking about everybody, or do you see really, you know, large organization, large Kubernetes yeah. Clusters infrastructure adopting it. >>Yeah. So that's the answer to that is changed a little bit over time. But at this point we see Linky adoption across industries, across verticals, and we see it from very small companies to very large ones. So, you know, one of the talks I'm really excited about at this conference is from the folks at Xbox cloud gaming, who talked about, who are gonna talk about how they deployed Linky across, you know, 22,000 pods around the world to serve, you know, basically on demand video games, never a use case I would ever have imagined for Linky. And at the previous Kuan, you know, virtually Kuan EU, we had a whole keynote about how Linky was used to combat COVID 19. So all sorts of uses. And it really doesn't, you know, whether, whether it's a small cluster or large cluster it's equally applicable. >>Wow. So as we talk about link D service match, we obviously are gonna talk about security application control, etcetera. But in this climate Software supply chain is critical, right. And as we think about open source software supply chain, talk to us about the recent security audit of link dealer. >>Yeah. So one of the things that we do as part of a CNCF project, and also as part of, I, I think our relationship with our community is we have regular security audits, you know, where we, we engage security professionals who are very thorough and, you know, dig into all the details. Of course the source code is all out there, you know, so anyone can read through the code, but they'll build threat model analyses and things like that. And then we take their, their report and we publish it. We say, Hey, look, here's, you know, here's the situation. So we have earlier reports online, and this newest one was done by a company called trail of bits. And they built a whole threat model and looked through all the different ways that Linky could go wrong. And they always find issues. Of course, you know, it's, it would be very scary, I think, to get a report that was like, no, we didn't find yeah. Earth clean, you know? Yeah. Everything's fine. You know, should be okay. I don't know. Right. But they, you know, they did not find anything critical. They found some issues that we rapidly addressed and then, you know, everything gets written up in the report and, and then we publish it, you know, as part of an open source artifact >>Are, you let's say, you know, do they give you and add something? So if something happens so that you can act on the code before, you know, somebody else discovers the >>Yeah, yeah. They'll give you a preview of what they found. And then often, you know, it's not like you're going before the judge and the judge makes a judgment and then like off the jail, right. It's, it's a dialogue because they don't necessarily understand the project. Well, they definitely don't understand it as well as you do. So you are helping them, you know, understand which parts and, and your, you know, are, are interesting to look at from the security perspective, which parts are not that interesting. They do their own investigation of course, but it's a dialogue the entire time. So you do have an opportunity to say, oh, you told me that was a, a, a minor issue. I actually think that's larger or, or vice versa. You know, you, you think that's a big problem. Actually, we thought about that, and it's not a big problem because of whatever. So it's a collaborative process. >>So link D been around, like, when I first learned about service me link D was the project that I learned about. Yeah. It's been there for a long time, but just mentioned 22,000 clusters. That's just mind boggling pod, 22,000 pods, the pods. Okay. >>Clusters would be >>Great. Yeah. Yeah. Clusters would be great too, but filled 22 thousands pods, big deployment. That's the big deployment of link D but all the way down to the small, smallest set of pods as well. What are some of the recent project updates from of the learnings you bought back from the community and updated the, the project as a result? >>Yeah. So a big one for us, you know, on the topic of security link, a big driver of link adoption is security and, and less on the supply chain side and more on the traffic, like live traffic security. So things like mutual TLS. So you can encrypt the communication between pods and make sure it's authenticated. One of the recent feature additions is authorization policy. So you can lock down connections between services and you can say service a is only allowed to talk to service B. And I wanna do that. Not based on network identity, you know, and not based on like IP addresses, cuz those are spoof. And you know, we've kind of like as an industry moved, moved, we've gotten a little more advanced from that, but actually based on the workload identity, you know, as captured by the mutual TLS certificate exchange. So we give you the ability now to, to, to restrict the types of communication that are allowed to happen on your cluster. >>So, okay. This is what happened. What about the future? Can you give us, you know, into suggestion of what is going to happen in the medium and long term? >>I think we're done, you know, we graduated, so we're just gonna >>Stop there's >>What else is there to do? There's no grad school, you know? No, no. So for us, there's a clear roadmap ahead, continuing down the, the security realm, for sure. We've given you kind of the very first building block, which at the service level, but coming up in, in the two point 12 release, we'll have route based policy as well, as you can say, this service is only allowed to call these three, you know, routes on this end point and we'll be working later to do things like mesh expansion so we can run the data plane outside of Kubernetes. You know, so the control plane will stay in in Kubernetes, but the data plane will, you'll be able to run that on VMs and, and, and things like that. And then of course in the, you know, we're also starting to look at things like I like to make a fun of WAM a lot, but we are actually starting to look at WAM in, in the ways that that might actually be useful for Linky users. >>So we talk a lot about the flexibility of a project, like link D you can do amazing things with it from a security perspective, but we're talking still to a DevOps type cloud of, of, of developers who are spread thin across their skillset. How do you help balance the need for the flexibility, which usually becomes more nerd knobs and servicing a crowd that wants even higher levels of abstraction and simplicity. >>Yeah. Yeah. That's a great question. And this is, this is what makes Linky so unique in the service mesh spaces. We have a laser focus on simplicity and especially on operational simplicity. So our audience, you know, we can make it easy to install Linky, but what we really care about is when you're running it and you're on call for it and it's sitting in this critical, vulnerable part of your infrastructure, do you feel confident in that? Do you feel like you understand it? Do you feel like you can observe it? Do you feel like you can predict what it's gonna do? And so every aspect of Linky is designed to be as operationally simple as possible. So when we deliver features, you know, that's always our, our primary consideration is, you know, we have to reject the urge. You know, we have an urge as, as engineers to like want to build everything, you know, it's an ultimate platform to solve all problems and we have to really be disciplined and say, we're not gonna do that. >>We're gonna look at solving the minimum possible problem with a minimum set of features because we need to keep things simple. And, and then we need to look at the human aspect to that. And I think that's been a part of, of Link's success. And then on the buoyant side, of course, you know, I don't just work on link day. I also work on, on buoyant, which helps organizations adopt Linky and, and increasingly large organizations that are not service mesh experts don't wanna be service mesh experts that, you know, they wanna spend their time and energy developing their business, right. And, and building the business logic that powers their company. So for them, we have actually re recently introduced, fully managed. Linky where we can take on, even though Linky has to run on your cluster, right? The, the, the, the sidecar proxies has to be alongside your application. We can actually take on the operational burden of, of upgrades and trust, anchor rotation, and installation. And you can effectively treat it as a utility, right. And, and, and have a, a hosted, like, experience, even though the, the actual bits, at least most of them, not all of them, most of 'em have to live on your cluster. >>I love the focus of most CNCF projects, you know, it's, it's peanut butter or jelly, not peanut butter. Yeah. Trying to be become jelly. Right. What's the, what's the, what's the peanut butter to link D's jelly. Like where does link D stop and some of the things that customers should really consider yeah. When looking at service mesh. >>Yeah. No, that's a great way of looking at it. And I, I actually think that that philosophy comes from Kubernetes. I think Kubernetes itself, one of the reasons it was so successful is because it had some clearly delineated, it said, this is what we're gonna do. Right. And this is what we're not gonna do. So we're gonna do layer three, four networking. Right. But we're gonna stop there. We're not gonna do anything with layer seven. And that allowed the service mesh. So I guess if I were to go down the, the bread, the bread of the sandwich has Kubernetes, and then Linky is the, is the peanut butter, I guess, and then the jelly, you know, so I think the jelly is every other aspect of, of building a platform. Right. So if you are the, the audience for Linky, most of the time, it's a platform owners, right. They're building a platform, an internal platform for their developers to write code. And so, as part of that, of course, you've got Kubernetes, you've got Linky, but you've also got a C I CD system. You've also got a, you know, a code repository, if it's GitLab or, or GitHub or wherever you've got, you know, other kind of tools that are enforcing various other constraints. All of that is the jelly, you know, in the, this is, analogy's getting complicated now. And like the, the platform sandwich that, you know, that you're serving. >>So talk to us about trans and service mesh from the, from the, as we think of the macro. >>Yeah. Yeah. So, you know, it's been an interesting space because we were talking a little bit about, you know, about this before the show, but the, there was so much buzz, you know, and then what we, what we saw was basically it took two years for that buzz to become actual adoption, you know, and now a lot of the buzz is off on other exciting things. And the people who remain in the Linky space are, are very focused on, oh, I actually have a, a real problem that I need to solve and I need to solve it now. So that's been great. So in terms of broader trends, you know, I think one thing we've seen for sure is the service mesh space is kind of notorious for complexity, you know, and a lot of what we've been doing on the Linky side has been trying to, to reverse that, that, that idea, you know, because it doesn't actually have to be complex. There's interesting stuff you can do, especially when you get into the way we handle the sidecar model. It's actually really, it's a wonderful model operationally. It's really, it feels weird at first. And then you're like, oh, actually this makes my operations a lot easier. So a lot of the trends that I see at least for Linky is doubling down on the sidecar model, trying to make side cards as small and as thin as possible and try and make them, you know, kind of transparent to the rest of the application. So >>Well, William Morgan, one of the coolest Twitter handles I've seen at WM on Twitter, that's actually a really cool Twitter handle. Thank you, CEO of buoyant. Thank you for joining the cube again. Cube alum from Valencia Spain. I'm Keith towns, along with en Rico, and you're watching the cube, the leader in high tech coverage.