>>The cube presents, Coon and cloud native con Europe, 2022, brought to you by red hat, the cloud native computing foundation and its ecosystem partners. >>Welcome to Valencia Spain in Coon cloud native con 2022 Europe. I'm Keith Townsend, along with my cohot on Rico senior, Etti senior it analyst at gig home. We are talking to amazing people, creators people contributing to all these open source projects. Speaking of open source on Rico. Talk to me about the flavor of this show versus a traditional like vendor show of all these open source projects and open source based companies. >>Well, first of all, I think that the real difference is that this is a real conference. Hmm. So real people talking about, you know, projects about, so the, the open source stuff, the experiences are, you know, on stage and there are not really too many product pitches. It's, it's about, it's about the people. It's about the projects. It's about the, the challenges they had, how they, you know, overcome some of them. And, uh, that's the main difference. I mean, it's very educative informative and the kind of people is different. I mean, developers, you know, SREs, you know, you find ends on people. I mean, people that really do stuff that that's a real difference. I mean, uh, quite challenginghow discussing with them, but really, I mean, because they're really opinionated, but >>So we're gonna get talked to, to a company that has boosts on the ground doing open source since the, almost the start mm-hmm <affirmative> Kirsten newcomer, director of hybrid platform security at red hat and, uh, Connor Gorman, senior principal software engineer at red hat. So Kirsten, we're gonna start with you security and Kubernetes, you know, is Kubernetes. It's a, it's a race car. If I wanted security, I'd drive a minivan. <laugh> >>That's, that's a great frame. I think, I think though, if we stick with your, your car analogy, right, we have seen cars in cars and safety in cars evolve over the years to the point where you have airbags, even in, you know, souped up cars that somebody's driving on the street, a race car, race cars have safety built into, right. They do their best to protect those drivers. So I think while Kubernetes, you know, started as something that was largely, you know, used by Google in their environment, you know, had some perimeter based security as Kubernetes has become adopted throughout enterprises, as people. And especially, you know, we've seen the adoption accelerate during the pandemic, the move to both public cloud, but also private cloud is really accelerated. Security becomes even more important. You can't use Kubernetes in banking without security. You can't use it, uh, in automotive without security telco. >>And Kubernetes is, you know, Telco's adoption, Telco's deploying 5g on Kubernetes on open shift. Um, and, and this is just so the security capabilities have evolved over time to meet the customers and the adopters really red hat because of our enterprise customer base, we've been investing in security capabilities and we make those contributions upstream. We've been doing that really from the beginning of our adoption of Kubernetes, Kubernetes 1.0, and we continue to expand the security capabilities that we provide. And which is one of the reasons, you know, the acquisition of stack rocks was, was so important to us. >>And, and actually we are talking about security at different levels. I mean, so yeah, and different locations. So you are securing an edge location differently than a data center or, or, or maybe, you know, the cloud. So there are application level security. So there are so many angles to take this. >>Yeah. And, and you're right. I mean, I, there are the layers of the stack, which starts, you know, can start at the hardware level, right. And then the operating system, the Kubernetes orchestration all the services, you need to have a complete Kubernetes solution and application platform and then the services themselves. And you're absolutely right. That an edge deployment is different than a deployment, uh, on, you know, uh, AWS or in a private da data center. Um, and, and yet, because there is this, if you, if you're leveraging the heart of Kubernetes, the declarative nature of Kubernetes, you can do Kubernetes security in a way that can be consistent across these environments with the need to do some additions at the edge, right? You may, physical security is more important at the edge hardware based encryption, for example, whereas in a, in a cloud provider, your encryption might be at the cloud provider storage layer rather than hardware. >>So how do you orchestrate, because we are talking about orchestration all day and how do you orchestrate all these security? >>Yep. So one of the things, one of the evolutions that we've seen in our customer base in the last few years is we used to have, um, a small number of large clusters that our customers deployed and they used in a multi-tenant fashion, right? Multiple teams from within the organization. We're now starting to see a larger number of smaller clusters. And those clusters are in different locations. They might be, uh, customers are both deploying in public cloud, as well as private, you know, on premises, um, edge deployments, as you mentioned. And so we've invested in, uh, multi cluster management and, or, you know, sort of that orchestration for orchestrators, right? The, and because again of the declarative nature of Kubernetes, so we offer, uh, advanced cluster management, red hat, advanced cluster management, which we open sourced as the multi cluster engine CE. Um, so that component is now also freely available, open source. We do that with everything. So if you need a way to ensure that you have managed the configuration appropriately across all of these clusters in a declarative fashion, right. It's still YAML, it's written in YAML use ACM use CE in combination with a get ops approach, right. To manage that, uh, to ensure that you've got that environment consistent. And, and then, but then you have to monitor, right. You have to, I'm wearing >>All of these stack rocks >>Fits in. I mean, yeah, sure. >>Yeah. And so, um, you know, we took a Kubernetes native approach to securing all of this. Right. And there's kind of, uh, we have to say, there's like three major life cycles. You have the build life cycle, right. You're building these imutable images to go deployed to production. Right. That should never change that are, you know, locked at a point in time. And so you can do vulnerability scanning, you can do compliance checks at that point right. In the build phase. But then you put those in a registry, then those go and be deployed on top of Kubernetes. And you have the configuration of your application, you know, including any vulnerabilities that may exist in those images, you have the R back permissions, right. How much access does it have to the cluster? Is it exposed on the internet? Right. What can you do there? >>And then finally you have, the runtime perspective of is my pod is my container actually doing what I think it's supposed to do. Is it accessing all the right things? Is it running all the right processes? And then even taking that runtime information and influencing the configuration through things like network policies, where we have a feature called process baselining that you can say exactly what processes are supposed to run in this pod. Um, and then influencing configuration in that way to kind of be like, yeah, this is what it's doing. And let's go stamp this, you know, declaratively so that when you deploy it the next time you already have security built in at the Kubernetes level. >>So as we've talked about a couple of different topics, the abstraction layers, I have security around DevOps. So, you know, I have multi tendency, I have to deal with, think about how am I going to secure the, the, the Kubernetes infrastructure itself. Then I have what seems like you've been talking about here, Connor, which is dev SecOps mm-hmm <affirmative> and the practice of securing the application through policy. Right. Are customers really getting what's under the hood of dev SecOps? >>Do you wanna start or yeah. >>I mean, I think yes and no. I think, um, you know, we've, some organizations are definitely getting it right. And they have teams that are helping build things like network policies, which provide network segmentation. I think this is huge for compliance and multi-tenancy right. Just like containers, you know, one of the main benefits of containers, it provides this isolation between your applications, right? And then everyone's familiar with the network firewall, which is providing network segmentation, but now in between your applications inside Kubernetes, you can create, uh, network segmentation. Right. And so we have some folks that are super, super far along that path and, and creating those. And we have some folks who have no network policies except the ones that get installed with our products. Right. And then we say, okay, how can we help you guys start leveraging these things and, and creating maybe just basic name, space isolation, or things like that. And then trying to push that back into more the declarative approach. >>So some of what I think we hear from, from what Connor just te teed up is that real DevSecOps requires breaking down silos between developers, operations and security, including network security teams. And so the Kubernetes paradigm requires, uh, involvement actually, in some ways, it, it forces involvement of developers in things like network policy for the SDN layer, right? You need to, you know, the application developer knows which, what kinds of communication he or she, his app or her app needs to function. So they need to define, they need to figure out those network policies. Now, some network security teams, they're not familiar with YAML, they're not necessary familiar with software development, software defined networking. So there's this whole kind of, how do we do the network security in collaboration with the engineering team? And when people, one of the things I worry about, so DevSecOps it's technology, but it's people in process too. >>Right. And one of the things I think people are very comfortable adopting vulnerability scanning early on, but they haven't yet started to think about the network security angle. This is one area that not only do we have the ability in ACS stack rocks today to recommend a network policy based on a running deployment, and then make it easy to deploy that. But we're also working to shift that left so that you can actually analyze app deployment data prior to it being deployed, generate a network policy, tested out in staging and, and kind of go from the beginning. But again, people do vulnerability analysis shift left, but they kind of tend to stop there and you need to add app config analysis, network communication analysis, and then we need appropriate security gates at deployment time. We need the right automation that helps inform the developers. Not all developers have security expertise, not all security people understand a C I C D pipeline. Right. So, so how, you know, we need the right set of information to the right people in the place they're used to working in order to really do that infinity loop. >>Do you see this as a natural progression for developers? Do they really hit a wall before, you know, uh, finding out that they need to progress in, in this, uh, methodology? Or I know >>What else? Yeah. So I think, I think initially there's like a period of transition, right? Where there's sometimes there's opinion, oh, I, I ship my application. That's what I get paid for. That's what I do. Right. <laugh> um, and, and, but since, uh, Kubernetes has basically increased the velocity of developers on top, you know, of the platform in order to just deploy their own code. And, you know, we have every, some people have commits going to production, you know, every commitment on the repo goes to production. Right. Um, and so security is even more at the forefront there. So I think initially you hit a little bit of a wall security scans in CI. You could get some failures and some pushback, but as long as these are very informative and actionable, right. Then developers always wanna do the right thing. Right. I mean, we all want to ship secure code. >>Um, and so if you can inform you, Hey, this is why we do this. Or, or here's the information about this? I think it's really important because I'm like, right, okay. Now when I'm sending my next commits, I'm like, okay, these are some constraints that I'm thinking about, and it's sort of like a mindset shift, but I think through the tooling that we like know and love, and we use on top of Kubernetes, that's the best way to kind of convey that information of, you know, honestly significantly smaller security teams than the number of developers that are really pushing all of this code. >>So let's scale out what, talk to me about the larger landscape projects like prime cube, Litner, OPPI different areas of investment in, in, in security. Talk to me about where customers are making investments. >>You wanna start with coup linter. >>Sure. So coup linter was a open source project, uh, when we were still, uh, a private company and it was really around taking some of our functionality on our product and just making it available to everyone, to basically check configuration, um, both bridging DevOps and SecOps, right? There's some things around, uh, privileged containers, right? You usually don't wanna deploy those into your environment unless you really need to, but there's other things around, okay, do I have anti affinity rules, right. Am I running, you know, you can run 10 replicas of a pod on the same node, and now your failure domain is a single node. Now you want them on different nodes, right. And so you can do a bunch of checks just around the configuration DevOps best practices. And so we've actually seen quite a bit of adoption. I think we have like almost 2000 stars on, uh, and super happy to see people just really adopt that and integrate it into their pipelines. It's a single binary. So it's been super easy for people to take it into their C I C D and just, and start running three things through it and get, uh, you know, valuable insights into, to what configurations they should change. Right. >>And then if you're, if you were asking about things like, uh, OPPA, open policy agent and OPPA gatekeeper, so one of the things happening in the community about OPPA has been around for a while. Uh, they added, you know, the OPPA gatekeeper as an admission controller for Cobe. There's also veno another open source project that is doing, uh, admission as the Kubernetes community has, uh, kind of is decided to deprecate pod security policies, um, which had a level of complexity, but is one of the key security capabilities and gates built into Kubernetes itself. Um, OpenShift is gonna continue to have security context constraints, very similar, but it prevents by default on an OpenShift cluster. Uh, not a regular user cannot deploy a privileged pod or a pod that has access to the host network. Um, and there's se Linux configuration on by default also protects against container escapes to the file system or mitigates them. >>So pod security policies were one way to ensure that kind of constraint on what the developer did. Developers might not have had awareness of what was important in terms of the level of security. And so again, the cube and tools like that can help to inform the developer in the tools they use, and then a solution like OPPA, gatekeeper, or SCCs. That's something that runs on the cluster. So if something got through the pipeline or somebody's not using one of these tools, those gates can be leveraged to ensure that the security posture of the deployment is what the organization wants and OPPA gatekeeper. You can do very complex policies with that. And >>Lastly, talk to me about Falco and Claire, about what Falco >>Falco and yep, absolutely. So, um, Falco, great runtime analysis have been and something that stack rocks leveraged early on. So >>Yeah, so yeah, we leveraged, um, some libraries from Falco. Uh, we use either an EB P F pro or a kernel module to detect runtime events. Right. And we, we primarily focus on network and process activity as, um, as angles there. And then for Claire, um, it's, it's now within red hat again, <laugh>, uh, through the acquisition of cores, but, uh, we've forked in added a bunch of things around language vulnerabilities and, and different aspects that we wanted. And, uh, and you know, we're really interested in, I think, you know, the code bases have diversion a little bit Claire's on V4. We, we were based off V2, but I think we've both added a ton of really great features. And so I'm really looking forward to actually combining all of those features and kind of building, um, you know, we have two best of best of breed scanners right now. And I'm like, okay, what can we do when we put them together? And so that's something that, uh, I'm really excited about. >>So you, you somehow are aiming at, you know, your roadmap here now putting everything together. And again, orchestrated well integrated yeah. To, to get, you know, also a simplified experience, because that could be the >>Point. Yeah. And, and as you mentioned, you know, it's sort of that, that orchestration of orchestrators, like leveraging the Kubernetes operator principle to, to deliver an app, an opinionated Kubernetes platform has, has been one of the key things we've done. And we're doing that as well for security out of the box security policies, principles based on best practices with stack rocks that can be leveraged in the community or with red hat, advanced cluster security, combining our two scanners into one clear based scanner, contributing back, contributing back to Falco all of these things. >>Well, that speaks to the complexity of open source projects. There's a lot of overlap in reconciling. That is a very difficult thing. Kirsten Connor, thank you for joining the cube Connor. You're now a cube alone. Welcome to main elite group. Great. From Valencia Spain, I'm Keith Townsend, along with en Rico senior, and you're watching the cue, the leader in high tech coverage.

>>The cube presents, Coon and cloud native con Europe 22, brought to you by the cloud native computing foundation. >>Welcome to Melissa Spain. And we're at cuon cloud native con Europe, 2022. I'm Keith Townsend. And my co-host en Rico senior Etti en Rico's really proud of me. I've called him en Rico and said IK, every session, senior it analyst giga, O we're talking to fantastic builders at Cuban cloud native con about the projects and the efforts en Rico up to this point, it's been all about provisioning insecurity. What, what conversation have we been missing? >>Well, I mean, I, I think, I think that, uh, uh, we passed the point of having the conversation of deployment of provisioning. You know, everybody's very skilled, actually everything is done at day two. They are discovering that, well, there is a security problem. There is an observability problem. And in fact, we are meeting with a lot of people and there are a lot of conversation with people really needing to understand what is happening. I mean, in their classroom, what, why it is happening and all the, the questions that come with it. I mean, and, uh, the more I talk with, uh, people in the, in the show floor here, or even in the, you know, in the various sessions is about, you know, we are growing, the, our clusters are becoming bigger and bigger. Uh, applications are becoming, you know, bigger as well. So we need to know, understand better what is happening. It's not only, you know, about cost it's about everything at the >>End. So I think that's a great set up for our guests, max, Provo, founder, and CEO of storm for forge and Patrick Britton, Bergstrom, Brookstone. Yeah, I spelled it right. I didn't say it right. Berg storm CTO. We're at Q con cloud native con we're projects are discussed, built and storm forge. I I've heard the pitch before, so forgive me. And I'm, I'm, I'm, I'm, I'm, I'm kind of torn. I have service mesh. What do I need more like, what problem is storm for solving? >>You wanna take it? >>Sure, absolutely. So it it's interesting because, uh, my background is in the enterprise, right? I was an executive at United health group. Um, before that I worked at best buy. Um, and one of the issues that we always had was, especially as you migrate to the cloud, it seems like the CPU dial or the memory dial is your reliability dial. So it's like, oh, I just turned that all the way to the right and everything's hunky Dory. Right. Uh, but then we run into the issue like you and I were just talking about where it gets very, very expensive, very quickly. Uh, and so my first conversations with Matt and the storm forge group, and they were telling me about the product and, and what we're dealing with. I said, that is the problem statement that I have always struggled with. And I wish this existed 10 years ago when I was dealing with EC two costs, right? And now with Kubernetes, it's the same thing. It's so easy to provision. So realistically, what it is is we take your raw telemetry data and we essentially monitor the performance of your application. And then we can tell you using our machine learning algorithms, the exact configuration that you should be using for your application to achieve the results that you're looking for without over provisioning. So we reduce your consumption of CPU of memory and production, which ultimately nine times outta 10, actually I would say 10 out of 10 reduces your cost significantly without sacrificing reliability. >>So can your solution also help to optimize the application in the long run? Because yes, of course, yep. You know, the lowing fluid is, you know, optimize the deployment. Yeah. But actually the long term is optimizing the application. Yes. Which is the real problem. >>Yep. So we actually, um, we're fine with the, the former of what you just said, but we exist to do the latter. And so we're squarely and completely focused at the application layer. Um, we are, uh, as long as you can track or understand the metrics you care about for your application, uh, we can optimize against it. Um, we love that we don't know your application. We don't know what the SLA and SLO requirements are for your app. You do. And so in, in our world, it's about empowering the developer into the process, not automating them out of it. And I think sometimes AI and machine learning sort of gets a bad wrap from that standpoint. And so, uh, we've at this point, the company's been around, you know, since 2016, uh, kind of from the very early days of Kubernetes, we've always been, you know, squarely focused on Kubernetes using our core machine learning, uh, engine to optimize metrics at the application layer, uh, that people care about and, and need to need to go after. And the truth of the matter is today. And over time, you know, setting a cluster up on Kubernetes has largely been solved. Um, and yet the promise of, of Kubernetes around portability and flexibility, uh, downstream when you operationalize the complexity, smacks you in the face. And, uh, and that's where, where storm forge comes in. And so we're a vertical, you know, kind of vertically oriented solution. Um, that's, that's absolutely focused on solving that problem. >>Well, I don't want to play, actually. I want to play the, uh, devils advocate here and, you know, >>You wouldn't be a good analyst if you didn't. >>So the, the problem is when you talk with clients, users, they, there are many of them still working with Java with, you know, something that is really tough. Mm-hmm <affirmative>, I mean, we loved all of us loved Java. Yeah, absolutely. Maybe 20 years ago. Yeah. But not anymore, but still they have developers. They are porting applications, microservices. Yes. But not very optimized, etcetera. C cetera. So it's becoming tough. So how you can interact with these kind of yeah. Old hybrid or anyway, not well in generic applications. >>Yeah. We, we do that today. We actually, part of our platform is we offer performance testing in a lower environment and stage. And we like Matt was saying, we can use any metric that you care about and we can work with any configuration for that application. So the perfect example is Java, you know, you have to worry about your heap size, your garbage collection tuning. Um, and one of the things that really struck, struck me very early on about the storm forage product is because it is true machine learning. You remove the human bias from that. So like a lot of what I did in the past, especially around SRE and, and performance tuning, we were only as good as our humans were because of what they knew. And so we were, we kind of got stuck in these paths of making the same configuration adjustments, making the same changes to the application, hoping for different results. But then when you apply machine learning capability to that, the machine will recommend things you never would've dreamed of. And you get amazing results out of >>That. So both me and an Rico have been doing this for a long time. Like I have battled to my last breath, the, the argument when it's a bare metal or a VM. Yeah. Look, I cannot give you any more memory. Yeah. And the, the argument going all the way up to the CIO and the CIO basically saying, you know what, Keith you're cheap, my developer resources expensive, my bigger box. Yep. Uh, buying a bigger box in the cloud to your point is no longer a option because it's just expensive. Talk to me about the carrot or the stick as developers are realizing that they have to be more responsible. Where's the culture change coming from? So is it, that is that if it, is it the shift in responsibility? >>I think the center of the bullseye for us is within those sets of decisions, not in a static way, but in an ongoing way, especially, um, especially as the development of applications becomes more and more rapid. And the management of them, our, our charge and our belief wholeheartedly is that you shouldn't have to choose, you should not have to choose between costs or performance. You should not have to choose where your, you know, your applications live, uh, in a public private or, or hybrid cloud environment. And so we want to empower people to be able to sit in the middle of all of that chaos and for those trade-offs and those difficult interactions to no, no longer be a thing. You know, we're at, we're at a place now where we've done, you know, hundreds of deployments and never once have we met a developer who said, I'm really excited to get outta bed and come to work every day and manually tune my application. <laugh> One side, secondly, we've never met, uh, you know, uh, a manager or someone with budget that said, uh, please don't, you know, increase the value of my investment that I've made to lift and shift us over mm-hmm <affirmative>, you know, to the cloud or to Kubernetes or, or some combination of both. And so what we're seeing is the converging of these groups, um, at, you know, their happy place is the lack of needing to be able to, uh, make those trade offs. And that's been exciting for us. So, >>You know, I'm listening and looks like that your solution is right in the middle in application per performance management, observability. Yeah. And, uh, and monitoring. So it's a little bit of all of this. >>So we, we, we, we want to be, you know, the Intel inside of all of that, mm-hmm, <affirmative>, we don't, you know, we often get lumped into one of those categories. It used to be APM a lot. We sometimes get a, are you observability or, and we're really not any of those things in and of themselves, but we, instead of invested in deep integrations and partnerships with a lot of those, uh, with a lot of that tooling, cuz in a lot of ways, the, the tool chain is hardening, uh, in a cloud native and, and Kubernetes world. And so, you know, integrating in intelligently staying focused and great at what we solve for, but then seamlessly partnering and not requiring switching for, for our users who have already invested likely in a APM or observability. >>So to go a little bit deeper. Sure. What does it mean integration? I mean, do you provide data to this, you know, other applications in, in the environment or are they supporting you in the work that you >>Yeah, we're, we're a data consumer for the most part. Um, in fact, one of our big taglines is take your observability and turn it into actionability, right? Like how do you take the it's one thing to collect all of the data, but then how do you know what to do with it? Right. So to Matt's point, um, we integrate with folks like Datadog. Um, we integrate with Prometheus today. So we want to collect that telemetry data and then do something useful with it for you. >>But, but also we want Datadog customers. For example, we have a very close partnership with, with Datadog, so that in your existing data dog dashboard, now you have yeah. This, the storm for capability showing up in the same location. Yep. And so you don't have to switch out. >>So I was just gonna ask, is it a push pull? What is the developer experience? When you say you provide developer, this resolve ML, uh, learnings about performance mm-hmm <affirmative> how do they receive it? Like what, yeah, what's the, what's the, what's the developer experience >>They can receive it. So we have our own, we used to for a while we were CLI only like any good developer tool. Right. Uh, and you know, we have our own UI. And so it is a push in that, in, in a lot of cases where I can come to one spot, um, I've got my applications and every time I'm going to release or plan for a release or I have released, and I want to take, pull in, uh, observability data from a production standpoint, I can visualize all of that within the storm for UI and platform, make decisions. We allow you to, to set your, you know, kind of comfort level of automation that you're, you're okay with. You can be completely set and forget, or you can be somewhere along that spectrum. And you can say, as long as it's within, you know, these thresholds, go ahead and release the application or go ahead and apply the configuration. Um, but we also allow you to experience, uh, the same, a lot of the same functionality right now, you know, in Grafana in Datadog, uh, and a bunch of others that are coming. >>So I've talked to Tim Crawford who talks to a lot of CIOs and he's saying one of the biggest challenges, or if not, one of the biggest challenges CIOs are facing are resource constraints. Yeah. They cannot find the developers to begin with to get this feedback. How are you hoping to address this biggest pain point for CIOs? Yeah. >>Development? >>Just take that one. Yeah, absolutely. That's um, so like my background, like I said, at United health group, right. It's not always just about cost savings. In fact, um, the way that I look about at some of these tech challenges, especially when we talk about scalability, there's kind of three pillars that I consider, right? There's the tech scalability, how am I solving those challenges? There's the financial piece, cuz you can only throw money at a problem for so long. And it's the same thing with the human piece. I can only find so many bodies and right now that pool is very small. And so we are absolutely squarely in that footprint of, we enable your team to focus on the things that they matter, not manual tuning like Matt said. And then there are other resource constraints that I think that a lot of folks don't talk about too. >>Like we were, you were talking about private cloud for instance. And so having a physical data center, um, I've worked with physical data centers that companies I've worked for have owned where it is literally full wall to wall. You can't rack any more servers in it. And so their biggest option is, well, I could spend 1.2 billion to build a new one if I wanted to. Or if you had a capability to truly optimize your compute to what you needed and free up 30% of your capacity of that data center. So you can deploy additional name spaces into your cluster. Like that's a huge opportunity. >>So either out of question, I mean, may, maybe it, it doesn't sound very intelligent at this point, but so is it an ongoing process or is it something that you do at the very beginning mean you start deploying this. Yeah. And maybe as a service. Yep. Once in a year I say, okay, let's do it again and see if something changes. Sure. So one spot 1, 1, 1 single, you know? >>Yeah. Um, would you recommend somebody performance tests just once a year? >>Like, so that's my thing is, uh, previous at previous roles I had, uh, my role was you performance test, every single release. And that was at a minimum once a week. And if your thing did not get faster, you had to have an executive exception to get it into production. And that's the space that we wanna live in as well as part of your C I C D process. Like this should be continuous verification every time you deploy, we wanna make sure that we're recommending the perfect configuration for your application in the name space that you're deploying >>Into. And I would be as bold as to say that we believe that we can be a part of adding, actually adding a step in the C I C D process that's connected to optimization and that no application should be released monitored and sort of, uh, analyzed on an ongoing basis without optimization being a part of that. And again, not just from a cost perspective, yeah. Cost end performance, >>Almost a couple of hundred vendors on this floor. You know, you mentioned some of the big ones, data, dog, et cetera. But what happens when one of the up and comings out of nowhere, completely new data structure, some imaginable way to click to elementry data. Yeah. How do, how do you react to that? >>Yeah. To us it's zeros and ones. Yeah. Uh, and you know, we're, we're, we're really, we really are data agnostic from the standpoint of, um, we're not, we we're fortunate enough to, from the design of our algorithm standpoint, it doesn't get caught up on data structure issues. Um, you know, as long as you can capture it and make it available, uh, through, you know, one of a series of inputs, what one, one would be load or performance tests, uh, could be telemetry, could be observability if we have access to it. Um, honestly the messier, the, the better from time to time, uh, from a machine learning standpoint, um, it, it, it's pretty powerful to see we've, we've never had a deployment where we, uh, where we saved less than 30% while also improving performance by at least 10%. But the typical results for us are 40 to 60% savings and, you know, 30 to 40% improvement in performance. >>And what happens if the application is, I, I mean, yes, Kubernetes is the best thing of the world, but sometimes we have to, you know, external data sources or, or, you know, we have to connect with external services anyway. Mm-hmm <affirmative> yeah. So can you, you know, uh, can you provide an indication also on, on, on this particular application, like, you know, where the problem could >>Be? Yeah, yeah. And that, that's absolutely one of the things that we look at too, cuz it's um, especially when you talk about resource consumption, it's never a flat line, right? Like depending on your application, depending on the workloads that you're running, um, it varies from sometimes minute to minute, day to day, or it could be week to week even. Um, and so especially with some of the products that we have coming out with what we want to do, you know, partnering with, uh, you know, integrating heavily with the HPA and being able to handle some of those bumps and not necessarily bumps, but bursts and being able to do it in a way that's intelligent so that we can make sure that, like I said, it's the perfect configuration for the application regardless of the time of day that you're operating in or what your traffic patterns look like. Um, or you know, what your disc looks like, right? Like cuz with our, our low environment testing, any metric you throw at us, we can, we can optimize for. >>So Madden Patrick, thank you for stopping by. Yeah. Yes. We can go all day. Because day two is I think the biggest challenge right now. Yeah. Not just in Kubernetes, but application replatforming and re and transformation. Very, very difficult. Most CTOs and S that I talked to, this is the challenge space from Valencia Spain. I'm Keith Townsend, along with my host en Rico senior. And you're watching the queue, the leader in high tech coverage.

>>The cube presents, Coon and cloud native con Europe 22, brought to you by the cloud native computing foundation. >>Welcome to Valencia Spain in co con cloud native con Europe, 2022. I'm Keith Townsend with my cohos on Rico senior. Etti senior it analyst at gig home. Exactly 7,500 people I'm told en Rico. What's the flavor of the show so far, >>It's a fantastic mood. I mean, I found a lot of people wanting to track talk about what they're doing with Kubernetes, sharing their, you know, stories, some word stories that meet tough. And you know, this is where you learn actually, because we had a lot of zoom calls, webinar and stuff, but it is when you talk a video, oh, I did it this way and it didn't work out very well. So, and, and you start a conversation like this that is really different from learning from zoom. When, you know, everybody talks about things that working well, they did it, right. No, it's here that you learn from other experiences. >>So we're talking to amazing people the whole week, talking about those experiences here on the queue, fresh on the queue for the first time, Chris Vos, senior software engineer at Microsoft Xbox, Chris, welcome to the queue. >>Thank you so much for having >>Me. So first off, give us a high level picture of the environment that you're running at Microsoft. >>Yeah. So, you know, we've got 20, well probably close to 30 clusters at this point around the globe, you know, 700 to a thousand pods per cluster, roughly. So about 22,000 pods total. So yeah, it's pretty pretty sizable footprint and yeah. So we've been running on Kubernetes since 2018 and well actually might be 2017, but anyways, so yeah, that, that's kind of our, our footprint. >>Yeah. So all of that, let's talk about the basics, which is security across multiple I'm assuming containers, work, microservices, et cetera. Why did you and the team settle on link or do >>Yeah, so previously we had our own kind of solution for managing TLS certs and things like that. And we found it to be pretty painful pretty quickly. And so we knew, you know, we wanted something that was a little bit more abstracted away from the developers and, and things like that that allowed us to move quickly. And so we began investigating, you know, solutions to that. And a few of our colleagues went to Cuban in San Diego in 2019 cloud native con as well. And basically they just, you know, sped it all up. And actually funny enough, my, my old manager was one of the people who was there and he went to the link D booth and they had a thing going that was like, Hey, get set up with MTLS in five minutes. And he was like, this is something we want to do, why not check this out? And he was able to do it. And so that, that put it on our radar. And so yeah, we investigated several others and Leer D just perfectly fit exactly what we needed. >>So, so in general, we are talking about, you know, security at scale. So how you manage security to scale and also flexibility, right. But you know, what is the you, this there, you told us about the five minutes to start using there, but you know, again, we are talking about word stories. We talk about, you know, all these. So what, what, what kind of challenges you found at the beginning when you start adopting this technology? >>So the biggest ones were around getting up and running with like a new service, especially in the beginning, right. We were, you know, adding a new service almost every day. It felt like. And so, you know, basically it took someone going through a whole bunch of different repos, getting approvals from everyone to get the SEARCHs minted, all that fun stuff, getting them put into the right environments and in the right clusters to make sure that, you know, everybody is talking appropriately. And just the amount of work that, that took alone was just a huge headache and a huge barrier to entry for us to, you know, quickly move up the number of services we have. So, >>So I'm, I'm trying to wrap my head around the scale of the challenge. When I think about certification or certificate management, I have to do it on a small scale and the, the, every now and again, when a certificate expires, it is just a troubleshooting pain. Yes. So as I think about that, it costs, it's not just certificates across 22,000 pods or it's certificates across 22,000 pods in multiple applications. How were you doing that before link D like, what was the, what and what were the pain points? Like? What happens when a certificate either fails or expired up not, not updated? >>So, I mean, to be completely honest, the biggest thing is we're just unable to make the calls, you know, out or, or in, based on yeah. What is failing basically. But, you know, we saw essentially an uptick in failures around a certain service and pretty quickly, I pretty quickly, we got used to the fact that it was like, oh, it's probably a cert expiration issue. And so we tried, you know, a few things in order to make that a little bit more automated and things like that, but we never came to a solution that like didn't require every engineer on the team to know essentially quite a bit about this, just to get into it, which was a huge issue. >>So talk about day two after you've deployed link D how did this alleviate software engineers and what was like the, the benefits of now having this automated way of managing >>Certs? So the biggest thing is like, there is no touch from developers, everyone on our team. Well, I mean, there are a lot of people who are familiar with security and certs and all of that stuff, but no one has to know it. Like it's not a requirement. Like for instance, I knew nothing about it when I joined the team. And even when I was setting up our newer clusters, I knew very little about it. And I was still able to really quickly set up blinker D, which was really nice. And, and it's been, you know, essentially we've been able to just kind of set it and not think about it too much. Obviously, you know, there are parts of it that you have to think about. We monitor it and all that fun stuff, but, but yeah, it's been pretty painless almost day one. It took a lot, a long time to trust it for developers. You know, anytime there was a failure, it's like, oh, could this be link or D you know, but after a while, like now we don't have that immediate assumption because people have built up that trust, but >>Also you have this massive infrastructure, I mean, 30 cluster. So I guess that it's quite different to manage a single cluster and 30. So what are the, you know, consideration that you have to do to install this software on, you know, 30 different cluster manage different, you know, versions probably etcetera, etcetera, et cetera. >>So, I mean, you know, the, the, as far as like, I guess, just to clarify, are you asking specifically with Linky or are you just asking in more in general? Well, >>I mean, you, you can take the, the question in the, in two ways, so, okay. Yeah. Yes. Link in particular, but the 30 cluster also quite interesting. >>Yeah. So, I mean, you know, more generally, you know, how we manage our clusters and things like that. We have, you know, a CLI tool that we use in order to like, change context very quickly and switch and communicate with whatever cluster we're trying to connect to and, you know, are we debugging or getting logs, whatever. And then, you know, with link D it's nice because again, you know, we, we, aren't having to worry about like, oh, how is this cert being inserted in the right node or, or not the right node, but in the right cluster or things like that. Whereas with link D we don't, we don't really have that concern when we spin up our, our clusters, essentially we get the root certificate and, and everything like that packaged up, passed along to link D on installation. And then essentially there's not much we have to do after that. >>So talk to me about your upcoming coming section here at Q con what's the, what's the high level talking points? Like what, what will attendees learn? >>Yeah. So it's, it's a journey. Those are the sorts of talks that I find useful. Having not been, you know, I, I'm not a deep Kubernetes expert from, you know, decades or whatever of experience, but I think >>Nobody is >>Also true. That's another story. That's a, that's, that's a job posting decades of requirements for >>Of course. Yeah. But so, you know, it, it's a journey it's really just like, Hey, what made us decide on a service mesh in the first place? What made us choose link D and then what are the ways in which, you know, we, we use link D so what are those, you know, we use some of the extra plugins and things like that. And then finally, a little bit about more, what we're gonna do in the future. >>Let's talk about not just necessarily the future as in two or three days from now, or two or three years from now. Well, the future after you immediately solve the, the low level problems with link D what were some of the, the surprises, because link D in service me in general has have side benefits. Do you experience any of those side benefits as well? >>Yeah, it's funny, you know, writing the, the blog post, you know, I hadn't really looked at a lot of the data in years on, you know, when we did our investigations and things like that. And we had seen that we like had very low latency and low CPU utilization and things like that. And looking at some of that, I found that we were actually saving time off of requests. And I couldn't really think of why that was, and I was talking with someone else and the biggest, unfortunately, all that data's gone now, like the source data. So I can't go back and verify this, but it, it makes sense, you know, there's the availability zone routing that linker D supports. And so I think that's actually doing it where, you know, essentially if a node is closer to another node, it's essentially, you know, routing to those ones. So when one service is talking to another service and maybe on they're on the same node, you know, it, it short circuits that, and allows us to gain some, some time there. It's not huge, but it adds up after, you know, 10, 20 calls down the line. Right. >>In general. So you are saying that it's smooth operations in, in ATS, very, you know, simplifying your life. >>And again, we didn't have to really do anything for that. It, it, it handled that for it was there. Yeah. Yep. Yeah, exactly. >>So we know one thing when I do it on my laptop, it works fine when I do it with across 22,000 pods, that's a different experience. What were some of the lessons learned coming out of KU con 2018 in San Diego was there? I wish I would've ran to the microphone folks, but what were some of the hard lessons learned scaling link D across the 22,000 nodes? >>So, you know, the, the first one, and this seems pretty obvious, but was just not something I knew about was the high availability mode of link D so obviously makes sense. You would want that in a, you know, a large scale environment. So like, that's one of the big lessons that like, we didn't ride away. No. Like one of the mistakes we made in, in one of our pre-production clusters was not turning that on. And we were kind of surprised. We were like, whoa, like all of these pods are spinning up, but they're having issues like actually getting injected and things like that. And we found, oh, okay. Yeah, you need to actually give it some, some more resources, but it's still very lightweight considering, you know, they have high availability mode, but it's just a few instances still. >>So from, even from a, you know, binary perspective and running link D how much overhead is it? >>That is a great question. So I don't remember off the top of my head, the numbers, but it's very lightweight. We, we evaluated a few different service missions and it was the lightest weight that we encountered at that point. >>And then from a resource perspective, is it a team of link D people? Is it a couple of people, like how >>To be completely honest for a long time, it was one person, Abraham who actually is the person who proposed this talk. He couldn't make it to Valencia, but he essentially did probably 95% of the work to get a into production. And then this was before we even had a team dedicated to our infrastructure. And so we have, now we have a team dedicated, we're all kind of Linky folks, if not Linky experts, we at least can troubleshoot basically. And things like that. So it's, I think a group of six people on our team, and then, you know, various people who've had experience with it >>On other teams, but I'm not dedicated just to that. >>I mean, >>No one is dedicated just to it. No, it's pretty like pretty light touch once it's, once it's up and running, it took a very long time for us to really understand it and, and to, you know, get like, not getting started, but like getting to where we really felt comfortable letting it go in production. But once it was there, like, it is very, very light touch. >>Well, I really appreciate you stopping by Chris. It's been an amazing conversation to hear how Microsoft is using a open source project. Exactly. At scale. It's just a few years ago, when you would've heard the concept of Microsoft and open source together and like, oh, that's just, you know, but >>They have changed a lot in the last few years now, there are huge contributors. And, you know, if you go to Azure, it's full of open source stuff, every >>So, yeah. Wow. The Cuban 2022, how the world has changed in so many ways from Licia Spain, I'm Keith Townsend, along with a Rico senior, you're watching the, the leader in high tech coverage.

>>The cube presents, Coon and cloud native con Europe 22, brought to you by the cloud native computing foundation. >>Welcome to vincia Spain in Coon cloud native con Europe, 2022. I'm Keith towns alongside en Rico senior. Etti senior it analyst for giong welcome back to the show en >>Rico. Thank you again for having me here. >>First impressions of QAN. >>Well, great show. As, as I mentioned before, I think that we are really in this very positive mode of talking with each other and people wanting to see, you know, the projects, people that build the projects at it's amazing. I mean, a lot of interesting conversation in the show floor and in the various sessions, very positive move. >>So this is gonna be a fun one. We have some amazing builders on the show this week, and none other than William Morgan, CEO of buoyant. What's your role in the link D project? >>So I was one of the original creators of link D but at this point I'm just the, the beautiful face of the project. >>Speaking of beautiful face of the project, linker D just graduated from as a CNCF project. >>Yeah, that's right. So last year we, we became the first service mesh to graduate in the CNCF. Very proud of that. And that's thanks, you know, largely to the incredible community around Linky that is just excited about the project and, you know, wants to talk about it and wants to be involved. >>So let's talk about the significance of that link D not the only service mesh project out there. Talk to me about the level effort to get it to the point that it's graduated. That's you don't see too many projects graduating CNCF in general. So let's talk about kind of the work needed to get Nier D to this point. >>Yeah. So, you know, the, the, the bar is high and it's mostly a measure, not necessarily of like the, the project being technically good or bad or anything, but it's really a measure of maturity of the community around it. So is it being adopted by organizations that are really relying on it in a critical way? Is it, you know, being adopted across industries, you know, is it having kind of a significant impact on the cloud native community? And so for us, you know, there was the, the work involved in that was really not any different from the work involved in, in kind of maintaining ity and growing the community in the first place, which is you try and make it really useful. You try and make it really easy to get started with you, try and be supportive and to, you know, have a, a friendly and welcoming community. And if you do those things and, you know, you kind of naturally get yourself to the point where it's a, it's a really strong community full of people who are excited about it. >>So from the of view of, you know, users adopting the, this technology, so we are talking about everybody, or do you see really, you know, large organization, large Kubernetes yeah. Clusters infrastructure adopting it. >>Yeah. So that's the answer to that is changed a little bit over time. But at this point we see Linky adoption across industries, across verticals, and we see it from very small companies to very large ones. So, you know, one of the talks I'm really excited about at this conference is from the folks at Xbox cloud gaming, who talked about, who are gonna talk about how they deployed Linky across, you know, 22,000 pods around the world to serve, you know, basically on demand video games, never a use case I would ever have imagined for Linky. And at the previous Kuan, you know, virtually Kuan EU, we had a whole keynote about how Linky was used to combat COVID 19. So all sorts of uses. And it really doesn't, you know, whether, whether it's a small cluster or large cluster it's equally applicable. >>Wow. So as we talk about link D service match, we obviously are gonna talk about security application control, etcetera. But in this climate Software supply chain is critical, right. And as we think about open source software supply chain, talk to us about the recent security audit of link dealer. >>Yeah. So one of the things that we do as part of a CNCF project, and also as part of, I, I think our relationship with our community is we have regular security audits, you know, where we, we engage security professionals who are very thorough and, you know, dig into all the details. Of course the source code is all out there, you know, so anyone can read through the code, but they'll build threat model analyses and things like that. And then we take their, their report and we publish it. We say, Hey, look, here's, you know, here's the situation. So we have earlier reports online, and this newest one was done by a company called trail of bits. And they built a whole threat model and looked through all the different ways that Linky could go wrong. And they always find issues. Of course, you know, it's, it would be very scary, I think, to get a report that was like, no, we didn't find yeah. Earth clean, you know? Yeah. Everything's fine. You know, should be okay. I don't know. Right. But they, you know, they did not find anything critical. They found some issues that we rapidly addressed and then, you know, everything gets written up in the report and, and then we publish it, you know, as part of an open source artifact >>Are, you let's say, you know, do they give you and add something? So if something happens so that you can act on the code before, you know, somebody else discovers the >>Yeah, yeah. They'll give you a preview of what they found. And then often, you know, it's not like you're going before the judge and the judge makes a judgment and then like off the jail, right. It's, it's a dialogue because they don't necessarily understand the project. Well, they definitely don't understand it as well as you do. So you are helping them, you know, understand which parts and, and your, you know, are, are interesting to look at from the security perspective, which parts are not that interesting. They do their own investigation of course, but it's a dialogue the entire time. So you do have an opportunity to say, oh, you told me that was a, a, a minor issue. I actually think that's larger or, or vice versa. You know, you, you think that's a big problem. Actually, we thought about that, and it's not a big problem because of whatever. So it's a collaborative process. >>So link D been around, like, when I first learned about service me link D was the project that I learned about. Yeah. It's been there for a long time, but just mentioned 22,000 clusters. That's just mind boggling pod, 22,000 pods, the pods. Okay. >>Clusters would be >>Great. Yeah. Yeah. Clusters would be great too, but filled 22 thousands pods, big deployment. That's the big deployment of link D but all the way down to the small, smallest set of pods as well. What are some of the recent project updates from of the learnings you bought back from the community and updated the, the project as a result? >>Yeah. So a big one for us, you know, on the topic of security link, a big driver of link adoption is security and, and less on the supply chain side and more on the traffic, like live traffic security. So things like mutual TLS. So you can encrypt the communication between pods and make sure it's authenticated. One of the recent feature additions is authorization policy. So you can lock down connections between services and you can say service a is only allowed to talk to service B. And I wanna do that. Not based on network identity, you know, and not based on like IP addresses, cuz those are spoof. And you know, we've kind of like as an industry moved, moved, we've gotten a little more advanced from that, but actually based on the workload identity, you know, as captured by the mutual TLS certificate exchange. So we give you the ability now to, to, to restrict the types of communication that are allowed to happen on your cluster. >>So, okay. This is what happened. What about the future? Can you give us, you know, into suggestion of what is going to happen in the medium and long term? >>I think we're done, you know, we graduated, so we're just gonna >>Stop there's >>What else is there to do? There's no grad school, you know? No, no. So for us, there's a clear roadmap ahead, continuing down the, the security realm, for sure. We've given you kind of the very first building block, which at the service level, but coming up in, in the two point 12 release, we'll have route based policy as well, as you can say, this service is only allowed to call these three, you know, routes on this end point and we'll be working later to do things like mesh expansion so we can run the data plane outside of Kubernetes. You know, so the control plane will stay in in Kubernetes, but the data plane will, you'll be able to run that on VMs and, and, and things like that. And then of course in the, you know, we're also starting to look at things like I like to make a fun of WAM a lot, but we are actually starting to look at WAM in, in the ways that that might actually be useful for Linky users. >>So we talk a lot about the flexibility of a project, like link D you can do amazing things with it from a security perspective, but we're talking still to a DevOps type cloud of, of, of developers who are spread thin across their skillset. How do you help balance the need for the flexibility, which usually becomes more nerd knobs and servicing a crowd that wants even higher levels of abstraction and simplicity. >>Yeah. Yeah. That's a great question. And this is, this is what makes Linky so unique in the service mesh spaces. We have a laser focus on simplicity and especially on operational simplicity. So our audience, you know, we can make it easy to install Linky, but what we really care about is when you're running it and you're on call for it and it's sitting in this critical, vulnerable part of your infrastructure, do you feel confident in that? Do you feel like you understand it? Do you feel like you can observe it? Do you feel like you can predict what it's gonna do? And so every aspect of Linky is designed to be as operationally simple as possible. So when we deliver features, you know, that's always our, our primary consideration is, you know, we have to reject the urge. You know, we have an urge as, as engineers to like want to build everything, you know, it's an ultimate platform to solve all problems and we have to really be disciplined and say, we're not gonna do that. >>We're gonna look at solving the minimum possible problem with a minimum set of features because we need to keep things simple. And, and then we need to look at the human aspect to that. And I think that's been a part of, of Link's success. And then on the buoyant side, of course, you know, I don't just work on link day. I also work on, on buoyant, which helps organizations adopt Linky and, and increasingly large organizations that are not service mesh experts don't wanna be service mesh experts that, you know, they wanna spend their time and energy developing their business, right. And, and building the business logic that powers their company. So for them, we have actually re recently introduced, fully managed. Linky where we can take on, even though Linky has to run on your cluster, right? The, the, the, the sidecar proxies has to be alongside your application. We can actually take on the operational burden of, of upgrades and trust, anchor rotation, and installation. And you can effectively treat it as a utility, right. And, and, and have a, a hosted, like, experience, even though the, the actual bits, at least most of them, not all of them, most of 'em have to live on your cluster. >>I love the focus of most CNCF projects, you know, it's, it's peanut butter or jelly, not peanut butter. Yeah. Trying to be become jelly. Right. What's the, what's the, what's the peanut butter to link D's jelly. Like where does link D stop and some of the things that customers should really consider yeah. When looking at service mesh. >>Yeah. No, that's a great way of looking at it. And I, I actually think that that philosophy comes from Kubernetes. I think Kubernetes itself, one of the reasons it was so successful is because it had some clearly delineated, it said, this is what we're gonna do. Right. And this is what we're not gonna do. So we're gonna do layer three, four networking. Right. But we're gonna stop there. We're not gonna do anything with layer seven. And that allowed the service mesh. So I guess if I were to go down the, the bread, the bread of the sandwich has Kubernetes, and then Linky is the, is the peanut butter, I guess, and then the jelly, you know, so I think the jelly is every other aspect of, of building a platform. Right. So if you are the, the audience for Linky, most of the time, it's a platform owners, right. They're building a platform, an internal platform for their developers to write code. And so, as part of that, of course, you've got Kubernetes, you've got Linky, but you've also got a C I CD system. You've also got a, you know, a code repository, if it's GitLab or, or GitHub or wherever you've got, you know, other kind of tools that are enforcing various other constraints. All of that is the jelly, you know, in the, this is, analogy's getting complicated now. And like the, the platform sandwich that, you know, that you're serving. >>So talk to us about trans and service mesh from the, from the, as we think of the macro. >>Yeah. Yeah. So, you know, it's been an interesting space because we were talking a little bit about, you know, about this before the show, but the, there was so much buzz, you know, and then what we, what we saw was basically it took two years for that buzz to become actual adoption, you know, and now a lot of the buzz is off on other exciting things. And the people who remain in the Linky space are, are very focused on, oh, I actually have a, a real problem that I need to solve and I need to solve it now. So that's been great. So in terms of broader trends, you know, I think one thing we've seen for sure is the service mesh space is kind of notorious for complexity, you know, and a lot of what we've been doing on the Linky side has been trying to, to reverse that, that, that idea, you know, because it doesn't actually have to be complex. There's interesting stuff you can do, especially when you get into the way we handle the sidecar model. It's actually really, it's a wonderful model operationally. It's really, it feels weird at first. And then you're like, oh, actually this makes my operations a lot easier. So a lot of the trends that I see at least for Linky is doubling down on the sidecar model, trying to make side cards as small and as thin as possible and try and make them, you know, kind of transparent to the rest of the application. So >>Well, William Morgan, one of the coolest Twitter handles I've seen at WM on Twitter, that's actually a really cool Twitter handle. Thank you, CEO of buoyant. Thank you for joining the cube again. Cube alum from Valencia Spain. I'm Keith towns, along with en Rico, and you're watching the cube, the leader in high tech coverage.