>>Hey, welcome to the live panel. My name is Brett. I am your host, and indeed we are live. In fact, if you're curious about that, if you don't believe us, um, let's just show a little bit of the browser real quick to see. Yup. There you go. We're live. So, all right. So how this is going to work is I'm going to bring in some guests and, uh, in one second, and we're going to basically take your questions on the topic designer of the day, that continuous integration testing. Uh, thank you so much to my guests welcoming into the panel. I've got Carlos, Nico and Mandy. Hello everyone. >>Hello? All right, >>Let's go. Let's go around the room and all pretend we don't know each other and that the internet didn't read below the video who we are. Uh, hi, my name is Brett. I am a Docker captain, which means I'm supposed to know something about Docker. I'm coming from Virginia Beach. I'm streaming here from Virginia Beach, Virginia, and, uh, I make videos on the internet and courses on you to me, Carlos. Hey, >>Hey, what's up? I'm Carlos Nunez. I am a solutions architect, VMware. I do solution things with computers. It's fun. I live in Dallas when I'm moving to Houston in a month, which is where I'm currently streaming. I've been all over the Northeast this whole week. So, um, it's been fun and I'm excited to meet with all of you and talk about CIA and Docker. Sure. >>Yeah. Hey everyone. Uh, Nico, Khobar here. I'm a solution engineer at HashiCorp. Uh, I am streaming to you from, uh, the beautiful Austin, Texas. Uh, ignore, ignore the golden gate bridge here. This is from my old apartment in San Francisco. Uh, just, uh, you know, keeping that, to remember all the good days, um, that that lived at. But, uh, anyway, I work at Patrick Corp and I work on all things, automation, um, and cloud and dev ops. Um, and I'm excited to be here and Mandy, >>Hi. Yeah, Mandy Hubbard. I am streaming from Austin, Texas. I am, uh, currently a DX engineer at ship engine. Um, I've worked in QA and that's kind of where I got my, uh, my Docker experience and, um, uh, moving into DX to try and help developers better understand and use our products and be an advocate for them. >>Nice. Well, thank you all for joining me. Uh, I really appreciate you taking the time out of your busy schedule to be here. And so for those of you in chat, the reason we're doing this live, because it's always harder to do things live. The reason we're here is to answer a question. So we didn't come with a bunch of slides and demos or anything like that. We're here to talk amongst ourselves about ideas and really here for you. So we've, we obviously, this is about easy CII, so we're, we're going to try to keep the conversation around testing and continuous integration and all the things that that entails with containers. But we may, we may go down rabbit holes. We may go veer off and start talking about other things, and that's totally fine if it's in the realm of dev ops and containers and developer and ops workflows, like, Hey, it's, it's kinda game. >>And, uh, these people have a wide variety of expertise. They haven't done just testing, right? We, we live in a world where you all kind of have to wear many hats. So feel free to, um, ask what you think is on the top of your mind. And we'll do our best to answer. It may, might not be the best answer or the correct answer, but we're going to do our best. Um, well, let's get it start off. Uh, let's, let's get a couple of topics to start off with. Uh, th the, the easy CGI was my, one of my three ideas. Cause he's the, one of the things that I'm most excited about is the innovation we're seeing around easier testing, faster testing, automated testing, uh, because as much as we've all been doing this stuff for, you know, 15 years, since 20 years since the sort of Jenkins early days, um, it it's, it seems like it's still really hard and it's still a lot of work. >>So, um, let's go around the room real quick, and everybody can just kind of talk for a minute about like your experience with testing and maybe some of your pain points, like what you don't like about our testing world. Um, and we can talk about some pains, cause I think that will lead us to kind of talk about what, what are the things we're seeing now that might be better, uh, ideas about how to do this. I know for me, uh, testing, obviously there's the code part, but just getting it automated, but mostly getting it in the hands of developers so that they can control their own testing. And don't have to go talk to a person to run that test again, or the mysterious Jenkins platform somewhere. I keep mentioning Jenkins cause it's, it is still the dominant player out there. Um, so for me, I'm, I'm, I, I don't like it when I'm walking into a room and there's, there's only one or two people that know how the testing works or know how to make the new tests go into the testing platform and stuff like that. So I'm always trying to free those things so that any of the developers are enabled and empowered to do that stuff. So someone else, Carlos, anybody, um, >>Oh, I have a lot of opinions on that. Having been a QA engineer for most of my career. Um, the shift that we're saying is everyone is dev ops and everyone is QA. Th the issue I see is no one asked developers if they wanted to be QA. Um, and so being the former QA on the team, when there's a problem, even though I'm a developer and we're all running QA, they always tend to come to the one of the former QA engineers. And they're not really owning that responsibility and, um, and digging in. So that's kind of what I'm saying is that we're all expected to test now. And some people, well, some people don't know how it's, uh, for me it was kind of an intuitive skill. It just kind of fit with my personality, but not knowing what to look for, not knowing what to automate, not even understanding how your API end points are used by your front end to know what to test when a change is made. It's really overwhelming for developers. And, um, we're going to need to streamline that and, and hold their hands a little bit until they get their feet wet with also being QA. >>Right. Right. So, um, uh, Carlos, >>Yeah, uh, testing is like, Tesla is one of my favorite subjects to talk about when I'm baring with developers. And a lot of it is because of what Mandy said, right? Like a lot of developers now who used to write a test and say, Hey, QA, go. Um, I wrote my unit tests. Now write the rest of the test. Essentially. Now developers are expected to be able to understand how testing, uh, testing methodologies work, um, in their local environments, right? Like they're supposed to understand how to write an integration tasks federate into and tasks, a component test. And of course, how to write unit tests that aren't just, you know, assert true is true, right? Like more comprehensive, more comprehensive, um, more high touch unit tests, which include things like mocking and stubbing and spine and all that stuff. And, you know, it's not so much getting those tests. Well, I've had a lot of challenges with developers getting those tests to run in Docker because of usually because of dependency hell, but, um, getting developers to understand how to write tests that matter and mean something. Um, it's, it's, it can be difficult, but it's also where I find a lot of the enjoyment of my work comes into play. So yeah. I mean, that's the difficulty I've seen around testing. Um, big subject though. Lots to talk about there. >>Yeah. We've got, we've already got so many questions coming in. You already got an hour's worth of stuff. So, uh, Nico 81st thoughts on that? >>Yeah, I think I definitely agree with, with other folks here on the panel, I think from a, um, the shift from a skillset perspective that's needed to adopt the new technologies, but I think from even from, uh, aside from the organizational, um, and kind of key responsibilities that, that the new developers have to kinda adapt to and, and kind of inherit now, um, there's also from a technical perspective as there's, you know, um, more developers are owning the full stack, including the infrastructure piece. So that adds a lot more to the plate in Tim's oaf, also testing that component that they were not even, uh, responsible for before. Um, and, um, also the second challenge that, you know, I'm seeing is that on, you know, the long list of added, um, uh, tooling and, you know, there's new tool every other day. Um, and, um, that kind of requires more customization to the testing, uh, that each individual team, um, any individual developer Y by extension has to learn. Uh, so the customization, uh, as well as the, kind of the scope that had, uh, you know, now in conferences, the infrastructure piece, um, uh, both of act to the, to the challenges that we're seeing right now for, um, for CGI and overall testing, um, uh, the developers are saying, uh, in, in the market today. >>Yeah. We've got a lot of questions, um, about all the, all the different parts of this. So, uh, let me just go straight to them. Cause that's why we're here is for the people, uh, a lot of people asking about your favorite tools and in one of this is one of the challenges with integration, right? Is, um, there is no, there are dominant players, but there, there is such a variety. I mean, every one of my customers seems like they're using a different workflow and a different set of tools. So, and Hey, we're all here to just talk about what we're, what we're using, uh, you know, whether your favorite tools. So like a lot of the repeated questions are, what are your favorite tools? Like if you could create it from scratch, uh, what would you use? Pierre's asking, you know, GitHub actions sounds like they're a fan of GitHub actions, uh, w you know, mentioning, pushing the ECR and Docker hub and, uh, using vs code pipeline, I guess there may be talking about Azure pipelines. Um, what, what's your preferred way? So, does anyone have any, uh, thoughts on that anyone want to throw out there? Their preferred pipeline of tooling? >>Well, I have to throw out mine. I might as Jenkins, um, like kind of a honorary cloud be at this point, having spoken a couple of times there, um, all of the plugins just make the functionality. I don't love the UI, but I love that it's been around so long. It has so much community support, and there are so many plugins so that if you want to do something, you don't have to write the code it's already been tested. Um, unfortunately I haven't been able to use Jenkins in, uh, since I joined ship engine, we, most of our, um, our, our monolithic core application is, is team city. It's a dotnet application and TeamCity plays really well with.net. Um, didn't love it, uh, Ms. Jenkins. And I'm just, we're just starting some new initiatives that are using GitHub actions, and I'm really excited to learn, to learn those. I think they have a lot of the same functionality that you're looking for, but, um, much more simplified in is right there and get hubs. So, um, the integration is a lot more seamless, but I do have to go on record that my favorite CICT tools Jenkins. >>All right. You heard it here first people. All right. Anyone else? You're muted? I'm muted. Carlin says muted. Oh, Carla says, guest has muted themselves to Carlos. You got to unmute. >>Yes. I did mute myself because I was typing a lot, trying to, you know, try to answer stuff in the chat. And there's a lot of really dark stuff in there. That's okay. Two more times today. So yeah, it's fine. Yeah, no problem. So totally. And it's the best way to start a play more. So I'm just going to go ahead and light it up. Um, for enterprise environments, I actually am a huge fan of Jenkins. Um, it's a tool that people really understand. Um, it has stood the test of time, right? I mean, people were using Hudson, but 15 years ago, maybe longer. And, you know, the way it works, hasn't really changed very much. I mean, Jenkins X is a little different, but, um, the UI and the way it works internally is pretty familiar to a lot of enterprise environments, which is great. >>And also in me, the plugin ecosystem is amazing. There's so many plugins for everything, and you can make your own if you know, Java groovy. I'm sure there's a perfect Kotlin in there, but I haven't tried myself, but it's really great. It's also really easy to write, um, CIS code, which is something I'm a big fan of. So Jenkins files have been, have worked really well for me. I, I know that I can get a little bit more complex as you start to build your own models and such, but, you know, for enterprise enterprise CIO CD, if you want, especially if you want to roll your own or own it yourself, um, Jenkins is the bellwether and for very good reason now for my personal projects. And I see a lot on the chat here, I think y'all, y'all been agreed with me get hub actions 100%, my favorite tool right now. >>Um, I love GitHub actions. It's, it's customizable, it's modular. There's a lot of plugins already. I started using getting that back maybe a week after when GA and there was no documentation or anything. And I still, it was still my favorite CIA tool even then. Um, and you know, the API is really great. There's a lot to love about GitHub actions and, um, and I, and I use it as much as I can from my personal project. So I still have a soft spot for Travis CAI. Um, you know, they got acquired and they're a little different now trying to see, I, I can't, I can't let it go. I just love it. But, um, yeah, I mean, when it comes to Seattle, those are my tools. So light me up in the comments I will respond. Yeah. >>I mean, I, I feel with you on the Travis, the, I think, cause I think that was my first time experiencing, you know, early days get hub open source and like a free CIA tool that I could describe. I think it was the ammo back then. I don't actually remember, but yeah, it was kind of an exciting time from my experience. There was like, oh, this is, this is just there as a service. And I could just use it. It doesn't, it's like get hub it's free from my open source stuff. And so it does have a soft spot in my heart too. So yeah. >>All right. We've got questions around, um, cam, so I'm going to ask some questions. We don't have to have these answers because sometimes they're going to be specific, but I want to call them out because people in chat may have missed that question. And there's probably, you know, that we have smart people in chat too. So there's probably someone that knows the answer to these things. If, if it's not us, um, they're asking about building Docker images in Kubernetes, which to me is always a sore spot because it's Kubernetes does not build images by default. It's not meant for that out of the gate. And, uh, what is the best way to do this without having to use privileged containers, which privileged containers just implying that yeah, you, you, it probably has more privileges than by default as a container in Kubernetes. And that is a hard thing because, uh, I don't, I think Docker doesn't lie to do that out of the gate. So I don't know if anyone has an immediate answer to that. That's a pretty technical one, but if you, if you know the answer to that in chat, call it out. >>Um, >>I had done this, uh, but I'm pretty sure I had to use a privileged, um, container and install the Docker Damon on the Kubernetes cluster. And I CA I can't give you a better solution. Um, I've done the same. So, >>Yeah, uh, Chavonne asks, um, back to the Jenkins thing, what's the easiest way to integrate Docker into a Jenkins CICB pipeline. And that's one of the challenges I find with Jenkins because I don't claim to be the expert on Jenkins. Is there are so many plugins because of this, of this such a huge ecosystem. Um, when you go searching for Docker, there's a lot that comes back, right. So I, I don't actually have a preferred way because every team I find uses it differently. Um, I don't know, is there a, do you know if there's a Jenkins preferred, a default plugin? I don't even know for Docker. Oh, go ahead. Yeah. Sorry for Docker. And jacon sorry, Docker plugins for Jenkins. Uh, as someone's asking like the preferred or easy way to do that. Um, and I don't, I don't know the back into Jenkins that well, so, >>Well, th the new, the new way that they're doing, uh, Docker builds with the pipeline, which is more declarative versus the groovy. It's really simple, and their documentation is really good. They, um, they make it really easy to say, run this in this image. So you can pull down, you know, public images and add your own layers. Um, so I don't know the name of that plugin, uh, but I can certainly take a minute after this session and going and get that. Um, but if you really are overwhelmed by the plugins, you can just write your, you know, your shell command in Jenkins. You could just by, you know, doing everything in bash, calling the Docker, um, Damon directly, and then getting it working just to see that end to end, and then start browsing for plugins to see if you even want to use those. >>The plugins will allow more integration from end to end. Some of the things that you input might be available later on in the process for having to manage that yourself. But, you know, you don't have to use any of the plugins. You can literally just, you know, do a block where you write your shell command and get it working, and then decide if, for plugins for you. Um, I think it's always under important to understand what is going on under the hood before you, before you adopt the magic of a plugin, because, um, once you have a problem, if you're, if it's all a lockbox to you, it's going to be more difficult to troubleshoot. It's kind of like learning, get command line versus like get cracking or something. Once, once you get in a bind, if you don't understand the underlying steps, it's really hard to get yourself out of a bind, versus if you understand what the plugin or the app is doing, then, um, you can get out of situations a lot easier. That's a good place. That's, that's where I'd start. >>Yeah. Thank you. Um, Camden asks better to build test environment images, every commit in CII. So this is like one of those opinions of we're all gonna have some different, uh, or build on build images on every commit, leveraging the cash, or build them once outside the test pile pipeline. Um, what say you people? >>Uh, well, I I've seen both and generally speaking, my preference is, um, I guess the ant, the it's a consultant answer, right? I think it depends on what you're trying to do, right. So if you have a lot of small changes that are being made and you're creating images for each of those commits, you're going to have a lot of images in your, in your registry, right? And on top of that, if you're building those images, uh, through CAI frequently, if you're using Docker hub or something like that, you might run into rate limiting issues because of Docker's new rate, limiting, uh, rate limits that they put in place. Um, but that might be beneficial if the, if being able to roll back between those small changes while you're testing is important to you. Uh, however, if all you care about is being able to use Docker images, um, or being able to correlate versions to your Docker images, or if you're the type of team that doesn't even use him, uh, does he even use, uh, virgins in your image tags? Then I would think that that might be a little, much you might want to just have in your CIO. You might want to have a stage that builds your Docker images and Docker image and pushes it into your registry, being done first particular branches instead of having to be done on every commit regardless of branch. But again, it really depends on the team. It really depends on what you're building. It really depends on your workflow. It can depend on a number of things like a curse sometimes too. Yeah. Yeah. >>Once had two points here, you know, I've seen, you know, the pattern has been at every, with every, uh, uh, commit, assuming that you have the right set of tests that would kind of, uh, you would benefit from actually seeing, um, the, the, the, the testing workflow go through and can detect any issue within, within the build or whatever you're trying to test against. But if you're just a building without the appropriate set of tests, then you're just basically consuming almond, adding time, as well as all the, the image, uh, stories associated with it without treaty reaping the benefit of, of, of this pattern. Uh, and the second point is, again, I think if you're, if you're going to end up doing a per commit, uh, definitely recommend having some type of, uh, uh, image purging, um, uh, and, and, and garbage collection process to ensure that you're not just wasting, um, all the stories needed and also, um, uh, optimizing your, your bill process, because that will end up being the most time-consuming, um, um, you know, within, within your pipeline. So this is my 2 cents on this. >>Yeah, that's good stuff. I mean, those are both of those are conversations that could lead us into the rabbit hole for the rest of the day on storage management, uh, you know, CP CPU minutes for, uh, you know, your build stuff. I mean, if you're in any size team, more than one or two people, you immediately run into headaches with cost of CIA, because we have now the problem of tools, right? We have so many tools. We can have the CIS system burning CPU cycles all day, every day, if we really wanted to. And so you re very quickly, I think, especially if you're on every commit on every branch, like that gets you into a world of cost mitigation, and you probably are going to have to settle somewhere in the middle on, uh, between the budget, people that are saying you're spending way too much money on the CII platform, uh, because of all these CPU cycles, and then the developers who would love to have everything now, you know, as fast as possible and the biggest, biggest CPU's, and the biggest servers, and have the bills, because the bills can never go fast enough, right. >>There's no end to optimizing your build workflow. Um, we have another question on that. This is another topic that we'll all probably have different takes on is, uh, basically, uh, version tags, right? So on images, we, we have a very established workflow in get for how we make commits. We have commit shots. We have, uh, you know, we know get tags and there's all these things there. And then we go into images and it's just this whole new world that's opened up. Like there's no real consensus. Um, so what, what are your thoughts on the strategy for teams in their image tag? Again, another, another culture thing. Um, commander, >>I mean, I'm a fan of silver when we have no other option. Um, it's just clean and I like the timestamp, you know, exactly when it was built. Um, I don't really see any reason to use another, uh, there's just normal, incremental, um, you know, numbering, but I love the fact that you can pull any tag and know exactly when it was created. So I'm a big fan of bar, if you can make that work for your organization. >>Yep. People are mentioned that in chat, >>So I like as well. Uh, I'm a big fan of it. I think it's easy to be able to just be as easy to be able to signify what a major changes versus a minor change versus just a hot fix or, you know, some or some kind of a bad fix. The problem that I've found with having teams adopt San Bernardo becomes answering these questions and being able to really define what is a major change, what is a minor change? What is a patch, right? And this becomes a bit of an overhead or not so much of an overhead, but, uh, uh, uh, a large concern for teams who have never done versioning before, or they never been responsible for their own versioning. Um, in fact, you know, I'm running into that right now, uh, with, with a client that I'm working with, where a lot, I'm working with a lot of teams, helping them move their applications from a legacy production environment into a new one. >>And in doing so, uh, versioning comes up because Docker images, uh, have tags and usually the tax correlate to versions, but some teams over there, some teams that I'm working with are only maintaining a script and others are maintaining a fully fledged JAK, three tier application, you know, with lots of dependencies. So telling the script, telling the team that maintains a script, Hey, you know, you should use somber and you should start thinking about, you know, what's major, what's my number what's patch. That might be a lot for them. And for someone or a team like that, I might just suggest using commit shots as your versions until you figure that out, or maybe using, um, dates as your version, but for the more for the team, with the larger application, they probably already know the answers to those questions. In which case they're either already using Sember or they, um, or they may be using some other version of the strategy and might be in December, might suit them better. So, um, you're going to hear me say, it depends a lot, and I'm just going to say here, it depends. Cause it really does. Carlos. >>I think you hit on something interesting beyond just how to version, but, um, when to consider it a major release and who makes those decisions, and if you leave it to engineers to version, you're kind of pushing business decisions down the pipe. Um, I think when it's a minor or a major should be a business decision and someone else needs to make that call someone closer to the business should be making that call as to when we want to call it major. >>That's a really good point. And I add some, I actually agree. Um, I absolutely agree with that. And again, it really depends on the team that on the team and the scope of it, it depends on the scope that they're maintaining, right? And so it's a business application. Of course, you're going to have a product manager and you're going to have, you're going to have a product manager who's going to want to make that call because that version is going to be out in marketing. People are going to use it. They're going to refer to and support calls. They're going to need to make those decisions. Sember again, works really, really well for that. Um, but for a team that's maintaining the scripts, you know, I don't know, having them say, okay, you must tell me what a major version is. It's >>A lot, but >>If they want it to use some birds great too, which is why I think going back to what you originally said, Sember in the absence of other options. I think that's a good strategy. >>Yeah. There's a, there's a, um, catching up on chat. I'm not sure if I'm ever going to catch up, but there's a lot of people commenting on their favorite CII systems and it's, and it, it just goes to show for the, the testing and deployment community. Like how many tools there are out there, how many tools there are to support the tools that you're using. Like, uh, it can be a crazy wilderness. And I think that's, that's part of the art of it, uh, is that these things are allowing us to build our workflows to the team's culture. Um, and, uh, but I do think that, you know, getting into like maybe what we hope to be at what's next is I do hope that we get to, to try to figure out some of these harder problems of consistency. Uh, one of the things that led me to Docker at the beginning to begin with was the fact that it wa it created a consistent packaging solution for me to get my code, you know, off of, off of my site of my local system, really, and into the server. >>And that whole workflow would at least the thing that I was making at each step was going to be the same thing used. Right. And that, that was huge. Uh, it was also, it also took us a long time to get there. Right. We all had to, like Docker was one of those ones that decade kind of ideas of let's solidify the, enter, get the consensus of the community around this idea. And we, and it's not perfect. Uh, you know, the Docker Docker file is not the most perfect way to describe how to make your app, but it is there and we're all using it. And now I'm looking for that next piece, right. Then hopefully the next step in that, um, that where we can all arrive at a consensus so that once you hop teams, you know, okay. We all knew Docker. We now, now we're all starting to get to know the manifests, but then there's this big gap in the middle where it's like, it might be one of a dozen things. Um, you know, so >>Yeah, yeah. To that, to that, Brett, um, you know, uh, just maybe more of a shameless plug here and wanting to kind of talk about one of the things that I'm on. So excited, but I work, I work at Tasha Corp. I don't know anyone, or I don't know if many people have heard of, um, you know, we tend to focus a lot on workflows versus technologies, right. Because, you know, as you can see, even just looking at the chat, there's, you know, ton of opinions on the different tooling, right. And, uh, imagine having, you know, I'm working with clients that have 10,000 developers. So imagine taking the folks in the chat and being partnered with one organization or one company and having to make decisions on how to build software. Um, but there's no way you can conversion one or, or one way or one tool, uh, and that's where we're facing in the industry. >>So one of the things that, uh, I'm pretty excited about, and I don't know if it's getting as much traction as you know, we've been focused on it. This is way point, which is a project, an open source project. I believe we got at least, uh, last year, um, which is, it's more of, uh, it's, it is aim to address that really, uh, uh, Brad set on, you know, to come to tool to, uh, make it extremely easy and simple. And, you know, to describe how you want to build, uh, deploy or release your application, uh, in, in a consistent way, regardless of the tools. So similar to how you can think of Terraform and having that pluggability to say Terraform apply or plan against any cloud infrastructure, uh, without really having to know exactly the details of how to do it, uh, this is what wave one is doing. Um, and it can be applied with, you know, for the CIA, uh, framework. So, you know, task plugability into, uh, you know, circle CEI tests to Docker helm, uh, Kubernetes. So that's the, you know, it's, it's a hard problem to solve, but, um, I'm hopeful that that's the path that we're, you know, we'll, we'll eventually get to. So, um, hope, you know, you can, you can, uh, see some of the, you know, information, data on it, on, on HashiCorp site, but I mean, I'm personally excited about it. >>Yeah. Uh I'm to gonna have to check that out. And, um, I told you on my live show, man, we'll talk about it, but talk about it for a whole hour. Uh, so there's another question here around, uh, this, this is actually a little bit more detailed, but it is one that I think a lot of people deal with and I deal with a lot too, is essentially the question is from Cameron, uh, D essentially, do you use compose in your CIO or not Docker compose? Uh, because yes I do. Yeah. Cause it, it, it, it solves so many problems am and not every CGI can, I don't know, there's some problems with a CIO is trying to do it for me. So there are pros and cons and I feel like I'm still on the fence about it because I use it all the time, but also it's not perfect. It's not always meant for CIA. And CIA sometimes tries to do things for you, like starting things up before you start other parts and having that whole order, uh, ordering problem of things anyway. W thoughts and when have thoughts. >>Yes. I love compose. It's one of my favorite tools of all time. Um, and the reason why it's, because what I often find I'm working with teams trying to actually let me walk that back, because Jack on the chat asked a really interesting question about what, what, what the hardest thing about CIS for a lot of teams. And in my experience, the hardest thing is getting teams to build an app that is the same app as what's built in production. A lot of CGI does things that are totally different than what you would do in your local, in your local dev. And as a result of that, you get, you got this application that either doesn't work locally, or it does work, but it's a completely different animal than what you would get in production. Right? So what I've found in trying to get teams to bridge that gap by basically taking their CGI, shifting the CII left, I hate the shift left turn, but I'll use it. >>I'm shifting the CIO left to your local development is trying to say, okay, how do we build an app? How do we, how do we build mot dependencies of that app so that we can build so that we can test our app? How do we run tests, right? How do we build, how do we get test data? And what I found is that trying to get teams to do all this in Docker, which is normally a first for a lot of teams that I'm working with, trying to get them all to do all of this. And Docker means you're running Docker, build a lot running Docker, run a lot. You're running Docker, RM a lot. You ran a lot of Docker, disparate Docker commands. And then on top of that, trying to bridge all of those containers together into a single network can be challenging without compose. >>So I like using a, to be able to really easily categorize and compartmentalize a lot of the things that are going to be done in CII, like building a Docker image, running tests, which is you're, you're going to do it in CII anyway. So running tests, building the image, pushing it to the registry. Well, I wouldn't say pushing it to the registry, but doing all the things that you would do in local dev, but in the same network that you might have a mock database or a mock S3 instance or some of something else. Um, so it's just easy to take all those Docker compose commands and move them into your Yammel file using the hub actions or your dankest Bob using Jenkins, or what have you. Right. It's really, it's really portable that way, but it doesn't work for every team. You know, for example, if you're just a team that, you know, going back to my script example, if it's a really simple script that does one thing on a somewhat routine basis, then that might be a lot of overhead. Um, in that case, you know, you can get away with just Docker commands. It's not a big deal, but the way I looked at it is if I'm, if I'm building, if I build something that's similar to a make bile or rate file, or what have you, then I'm probably gonna want to use Docker compose. If I'm working with Docker, that's, that's a philosophy of values, right? >>So I'm also a fan of Docker compose. And, um, you know, to your point, Carlos, the whole, I mean, I'm also a fan of shifting CEI lift and testing lift, but if you put all that logic in your CTI, um, it changes the L the local development experience from the CGI experience. Versus if you put everything in a compose file so that what you build locally is the same as what you build in CGI. Um, you're going to have a better experience because you're going to be testing something more, that's closer to what you're going to be releasing. And it's also very easy to look at a compose file and kind of, um, understand what the dependencies are and what's happening is very readable. And once you move that stuff to CGI, I think a lot of developers, you know, they're going to be intimidated by the CGI, um, whatever the scripting language is, it's going to be something they're going to have to wrap their head around. >>Um, but they're not gonna be able to use it locally. You're going to have to have another local solution. So I love the idea of a composed file use locally, um, especially if he can Mount the local workspace so that they can do real time development and see their changes in the exact same way as it's going to be built and tested in CGI. It gives developers a high level of confidence. And then, you know, you're less likely to have issues because of discrepancies between how it was built in your local test environment versus how it's built in NCI. And so Docker compose really lets you do all of that in a way that makes your solution more portable, portable between local dev and CGI and reduces the number of CGI cycles to get, you know, the test, the test data that you need. So that's why I like it for really, for local dev. >>It'll be interesting. Um, I don't know if you all were able to see the keynote, but there was a, there was a little bit, not a whole lot, but a little bit talk of the Docker, compose V two, which has now built into the Docker command line. And so now we're shifting from the Python built compose, which was a separate package. You could that one of the challenges was getting it into your CA solution because if you don't have PIP and you got down on the binary and the binary wasn't available for every platform and, uh, it was a PI installer. It gets a little nerdy into how that works, but, uh, and the team is now getting, be able to get unified with it. Now that it's in Golang and it's, and it's plugged right into the Docker command line, it hopefully will be easier to distribute, easier to, to use. >>And you won't have to necessarily have dependencies inside of where you're running it because there'll be a statically compiled binary. Um, so I've been playing with that, uh, this year. And so like training myself to do Docker going from Docker dash compose to Docker space, compose. It is a thing I I'm almost to the point of having to write a shell replacement. Yeah. Alias that thing. Um, but, um, I'm excited to see what that's going, cause there's already new features in it. And it, these built kit by default, like there's all these things. And I, I love build kit. We could make a whole session on build kit. Um, in fact there's actually, um, maybe going on right now, or right around this time, there is a session on, uh, from Solomon hikes, the seat, uh, co-founder of Docker, former CTO, uh, on build kit using, uh, using some other tool on top of build kit or whatever. >>So that, that would be interesting for those of you that are not watching that one. Cause you're here, uh, to do a check that one out later. Um, all right. So another good question was caching. So another one, another area where there is no wrong answers probably, and everyone has a different story. So the question is, what are your thoughts on CII build caching? There's often a debate between security. This is from Quentin. Thank you for this great question. There's often a debate between security reproducibility and build speeds. I haven't found a good answer so far. I will just throw my hat in the ring and say that the more times you want to build, like if you're trying to build every commit or every commit, if you're building many times a day, the more caching you need. So like the more times you're building, the more caching you're gonna likely want. And in most cases caching doesn't bite you in the butt, but that could be, yeah, we, can we get the bit about that? So, yeah. Yeah. >>I'm going to quote Carlos again and say, it depends on, on, you know, how you're talking, you know, what you're trying to build and I'm quoting your colors. Um, yeah, it's, it's got, it's gonna depend because, you know, there are some instances where you definitely want to use, you know, depends on the frequency that you're building and how you're building. Um, it's you would want to actually take advantage of cashing functionalities, um, for the build, uh, itself. Um, but if, um, you know, as you mentioned, there could be some instances where you would want to disable, um, any caching because you actually want to either pull a new packages or, um, you know, there could be some security, um, uh, disadvantages related to security aspects that would, you know, you know, using a cache version of, uh, image layer, for example, could be a problem. And you, you know, if you have a fleet of build, uh, engines, you don't have a good grasp of where they're being cashed. We would have to, um, disable caching in that, in that, um, in those instances. So it, it would depend. >>Yeah, it's, it's funny you have that problem on both sides of cashing. Like there are things that, especially in Docker world, they will cash automatically. And, and then, and then you maybe don't realize that some of that caching could be bad. It's, it's actually using old, uh, old assets, old artifacts, and then there's times where you would expect it to cash, that it doesn't cash. And then you have to do something extra to enable that caching, especially when you're dealing with that cluster of, of CIS servers. Right. And the cloud, the whole clustering problem with caching is even more complex, but yeah, >>But that's, that's when, >>Uh, you know, ever since I asked you to start using build kits and able to build kit, you know, between it's it's it's reader of Boston in, in detecting word, you know, where in, in the bill process needs to cash, as well as, uh, the, the, um, you know, the process. I don't think I've seen any other, uh, approach there that comes close to how efficient, uh, that process can become how much time it can actually save. Uh, but again, I think, I think that's, for me that had been my default approach, unless I actually need something that I would intentionally to disable caching for that purpose, but the benefits, at least for me, the benefits of, um, how bill kit actually been processing my bills, um, from the builds as well as, you know, using the cash up until, you know, how it detects the, the difference in, in, in the assets within the Docker file had been, um, you know, uh, pretty, you know, outweigh the disadvantages that it brings in. So it, you know, take it each case by case. And based on that, determine if you want to use it, but definitely recommend those enabling >>In the absence of a reason not to, um, I definitely think that it's a good approach in terms of speed. Um, yeah, I say you cash until you have a good reason not to personally >>Catch by default. There you go. I think you catch by default. Yeah. Yeah. And, uh, the trick is, well, one, it's not always enabled by default, especially when you're talking about cross server. So that's a, that's a complexity for your SIS admins, or if you're on the cloud, you know, it's usually just an option. Um, I think it also is this, this veers into a little bit of, uh, the more you cash the in a lot of cases with Docker, like the, from like, if you're from images and checked every single time, if you're not pinning every single thing, if you're not painting your app version, you're at your MPN versions to the exact lock file definition. Like there's a lot of these things where I'm I get, I get sort of, I get very grouchy with teams that sort of let it, just let it all be like, yeah, we'll just build two images and they're totally going to have different dependencies because someone happened to update that thing and after whatever or MPM or, or, and so I get grouchy about that, cause I want to lock it all down, but I also know that that's going to create administrative burden. >>Like the team is now going to have to manage versions in a very much more granular way. Like, do we need to version two? Do we need to care about curl? You know, all that stuff. Um, so that's, that's kind of tricky, but when you get to, when you get to certain version problems, uh, sorry, uh, cashing problems, you, you, you don't want those set those caches to happen because it, if you're from image changes and you're not constantly checking for a new image, and if you're not pinning that V that version, then now you, you don't know whether you're getting the latest version of Davion or whatever. Um, so I think that there's, there's an art form to the more you pen, the less you have, the less, you have to be worried about things changing, but the more you pen, the, uh, all your versions of everything all the way down the stack, the more administrative stuff, because you're gonna have to manually change every one of those. >>So I think it's a balancing act for teams. And as you mature, I to find teams, they tend to pin more until they get to a point of being more comfortable with their testing. So the other side of this argument is if you trust your testing, then you, and you have better testing to me, the less likely to the subtle little differences in versions have to be penned because you can get away with those minor or patch level version changes. If you're thoroughly testing your app, because you're trusting your testing. And this gets us into a whole nother rant, but, uh, yeah, but talking >>About penny versions, if you've got a lot of dependencies isn't that when you would want to use the cash the most and not have to rebuild all those layers. Yeah. >>But if you're not, but if you're not painting to the exact patch version and you are caching, then you're not technically getting the latest versions because it's not checking for all the time. It's a weird, there's a lot of this subtle nuance that people don't realize until it's a problem. And that's part of the, the tricky part of allow this stuff, is it, sometimes the Docker can be almost so much magic out of the box that you, you, you get this all and it all works. And then day two happens and you built it a second time and you've got a new version of open SSL in there and suddenly it doesn't work. Um, so anyway, uh, that was a great question. I've done the question on this, on, uh, from heavy. What do you put, where do you put testing in your pipeline? Like, so testing the code cause there's lots of types of testing, uh, because this pipeline gets longer and longer and Docker building images as part of it. And so he says, um, before staging or after staging, but before production, where do you put it? >>Oh man. Okay. So, um, my, my main thought on this is, and of course this is kind of religious flame bait, so sure. You know, people are going to go into the compensation wrong. Carlos, the boy is how I like to think about it. So pretty much in every stage or every environment that you're going to be deploying your app into, or that your application is going to touch. My idea is that there should be a build of a Docker image that has all your applications coded in, along with its dependencies, there's testing that tests your application, and then there's a deployment that happens into whatever infrastructure there is. Right. So the testing, they can get tricky though. And the type of testing you do, I think depends on the environment that you're in. So if you're, let's say for example, your team and you have, you have a main branch and then you have feature branches that merged into the main branch. >>You don't have like a pre-production branch or anything like that. So in those feature branches, whenever I'm doing CGI that way, I know when I freak, when I cut my poll request, that I'm going to merge into main and everything's going to work in my feature branches, I'm going to want to probably just run unit tests and maybe some component tests, which really, which are just, you know, testing that your app can talk to another component or another part, another dependency, like maybe a database doing tests like that, that don't take a lot of time that are fascinating and right. A lot of would be done at the beach branch level and in my opinion, but when you're going to merge that beach branch into main, as part of a release in that activity, you're going to want to be able to do an integration tasks, to make sure that your app can actually talk to all the other dependencies that it talked to. >>You're going to want to do an end to end test or a smoke test, just to make sure that, you know, someone that actually touches the application, if it's like a website can actually use the website as intended and it meets the business cases and all that, and you might even have testing like performance testing, low performance load testing, or security testing, compliance testing that would want to happen in my opinion, when you're about to go into production with a release, because those are gonna take a long time. Those are very expensive. You're going to have to cut new infrastructure, run those tests, and it can become quite arduous. And you're not going to want to run those all the time. You'll have the resources, uh, builds will be slower. Uh, release will be slower. It will just become a mess. So I would want to save those for when I'm about to go into production. Instead of doing those every time I make a commit or every time I'm merging a feature ranch into a non main branch, that's the way I look at it, but everything does a different, um, there's other philosophies around it. Yeah. >>Well, I don't disagree with your build test deploy. I think if you're going to deploy the code, it needs to be tested. Um, at some level, I mean less the same. You've got, I hate the term smoke tests, cause it gives a false sense of security, but you have some mental minimum minimal amount of tests. And I would expect the developer on the feature branch to add new tests that tested that feature. And that would be part of the PR why those tests would need to pass before you can merge it, merge it to master. So I agree that there are tests that you, you want to run at different stages, but the earlier you can run the test before going to production. Um, the fewer issues you have, the easier it is to troubleshoot it. And I kind of agree with what you said, Carlos, about the longer running tests like performance tests and things like that, waiting to the end. >>The only problem is when you wait until the end to run those performance tests, you kind of end up deploying with whatever performance you have. It's, it's almost just an information gathering. So if you don't run your performance test early on, um, and I don't want to go down a rabbit hole, but performance tests can be really useless if you don't have a goal where it's just information gap, uh, this is, this is the performance. Well, what did you expect it to be? Is it good? Is it bad? They can get really nebulous. So if performance is really important, um, you you're gonna need to come up with some expectations, preferably, you know, set up the business level, like what our SLA is, what our response times and have something to shoot for. And then before you're getting to production. If you have targets, you can test before staging and you can tweak the code before staging and move that performance initiative. Sorry, Carlos, a little to the left. Um, but if you don't have a performance targets, then it's just a check box. So those are my thoughts. I like to test before every deployment. Right? >>Yeah. And you know what, I'm glad that you, I'm glad that you brought, I'm glad that you brought up Escalades and performance because, and you know, the definition of performance says to me, because one of the things that I've seen when I work with teams is that oftentimes another team runs a P and L tests and they ended, and the development team doesn't really have too much insight into what's going on there. And usually when I go to the performance team and say, Hey, how do you run your performance test? It's usually just a generic solution for every single application that they support, which may or may not be applicable to the application team that I'm working with specifically. So I think it's a good, I'm not going to dig into it. I'm not going to dig into the rabbit hole SRE, but it is a good bridge into SRE when you start trying to define what does reliability mean, right? >>Because the reason why you test performance, it's test reliability to make sure that when you cut that release, that customers would go to your site or use your application. Aren't going to see regressions in performance and are not going to either go to another website or, you know, lodge in SLA violation or something like that. Um, it does, it does bridge really well with defining reliability and what SRE means. And when you have, when you start talking about that, that's when you started talking about how often do I run? How often do I test my reliability, the reliability of my application, right? Like, do I have nightly tasks in CGI that ensure that my main branch or, you know, some important branch I does not mean is meeting SLA is meeting SLR. So service level objectives, um, or, you know, do I run tasks that ensure that my SLA is being met in production? >>Like whenever, like do I use, do I do things like game days where I test, Hey, if I turn something off or, you know, if I deploy this small broken code to production and like what happens to my performance? What happens to my security and compliance? Um, you can, that you can go really deep into and take creating, um, into creating really robust tests that cover a lot of different domains. But I liked just using build test deploy is the overall answer to that because I find that you're going to have to build your application first. You're going to have to test it out there and build it, and then you're going to want to deploy it after you test it. And that order generally ensures that you're releasing software. That works. >>Right. Right. Um, I was going to ask one last question. Um, it's going to have to be like a sentence answer though, for each one of you. Uh, this is, uh, do you lint? And if you lint, do you lent all the things, if you do, do you fail the linters during your testing? Yes or no? I think it's going to depend on the culture. I really do. Sorry about it. If we >>Have a, you know, a hook, uh, you know, on the get commit, then theoretically the developer can't get code there without running Melinta anyway, >>So, right, right. True. Anyone else? Anyone thoughts on that? Linting >>Nice. I saw an additional question online thing. And in the chat, if you would introduce it in a multi-stage build, um, you know, I was wondering also what others think about that, like typically I've seen, you know, with multi-stage it's the most common use case is just to produce the final, like to minimize the, the, the, the, the, the image size and produce a final, you know, thin, uh, layout or thin, uh, image. Uh, so if it's not for that, like, I, I don't, I haven't seen a lot of, you know, um, teams or individuals who are actually within a multi-stage build. There's nothing really against that, but they think the number one purpose of doing multi-stage had been just producing the minimalist image. Um, so just wanted to kind of combine those two answers in one, uh, for sure. >>Yeah, yeah, sure. Um, and with that, um, thank you all for the great questions. We are going to have to wrap this up and we could go for another hour if we all had the time. And if Dr. Khan was a 24 hour long event and it didn't sadly, it's not. So we've got to make room for the next live panel, which will be Peter coming on and talking about security with some developer ex security experts. And I wanted to thank again, thank you all three of you for being here real quick, go around the room. Um, uh, where can people reach out to you? I am, uh, at Bret Fisher on Twitter. You can find me there. Carlos. >>I'm at dev Mandy with a Y D E N D Y that's me, um, >>Easiest name ever on Twitter, Carlos and DFW on LinkedIn. And I also have a LinkedIn learning course. So if you check me out on my LinkedIn learning, >>Yeah. I'm at Nicola Quebec. Um, one word, I'll put it in the chat as well on, on LinkedIn, as well as, uh, uh, as well as Twitter. Thanks for having us, Brett. Yeah. Thanks for being here. >>Um, and, and you all stay around. So if you're in the room with us chatting, you're gonna, you're gonna, if you want to go to see the next live panel, I've got to go back to the beginning and do that whole thing, uh, and find the next, because this one will end, but we'll still be in chat for a few minutes. I think the chat keeps going. I don't actually know. I haven't tried it yet. So we'll find out here in a minute. Um, but thanks you all for being here, I will be back a little bit later, but, uh, coming up next on the live stuff is Peter Wood security. Ciao. Bye.

>>Hello, everyone. My name is John Jane Shake. I work from Iran. Tous Andi. I am here this afternoon very gratefully with Anders Vulcan, who is VP of technology strategy for cloud bees, a Miranda's partner and a well known company in the space that we're going to be discussing. Anders is also a well known entity in this space, which is continuous integration and continuous delivery. Um, you've seen already today some sessions that focus on specific implementations of continuous integration and delivery, um, particularly around security. And, uh, we think this is a critically important topic for anyone in the cloud space, particularly in this increasingly complicated kubernetes space. To understand, um, Miranda's thanks, Uh, if I can recapitulate our own our own strategy and, uh, and language that with complexity on uncertainty consistently increasing with the depth of the technology stacks that we have to deal with consistently, um um elaborating themselves that navigating this requires, um first three implementation of automation to increase speed, which is what C and C d do. Um, and that this speed ba leveraged toe let us ship and iterate code faster. Since that's ultimately the business that all of us air in one way or another. I would like, I guess, toe open this conversation by asking Onders what does he think of that core strategy? >>You know, I think you know, hitting the security thing, right? Right off the bat. You know, security doesn't happen by accident. You know, security is not something that you know, Like a like a server in a restaurant. You know, Sprinkles a little bit of Parmesan cheese right before they serve you the the food. It's not something you Sprinkle on at the end. It's something that has to be baked in from the beginning, not just in the kitchen, but in the supply chain from from from the very beginning. So the you know it's a feature, and if you don't build it, if you're not going to get an outcome that you're not gonna be happy with and I think the you know it's increasingly it's obviously increasingly important and increasingly visible. You know, the you know, the kinds of security problems that we that we see these days can can be, you know, life altering, for for people that are subject to them and and can be, you know, life or death for a company that that's exposed to it. So it's it's it's very, very important. Thio pay attention to it and to work to achieve that as an explicit outcome of the software delivery process. And I think, you know, C i n c d as as process as tooling as culture plays a big part in that because ah, lot of it has to do with, you know, set things up, right? Um run them the same way over and over, you know, get the machine going. Turned the crane. Now, you wanna you wanna make improvements over over time. You know, it's not just, you know, set it and forget it. You know, we got that set up. We don't have to worry about it anymore, but it really is a question of, you know, get the human out of the loop a lot of the times because if if you're dealing with configuring complex systems, you wanna make sure that you get them set up configured, you know, documented Ideally, you know, as code, whether it's a domain specific language or or something like that. And then that's something that you contest against that you can verify against that you can that you can difficult. And then that becomes the basis for for your, you know, for yourself, for pipelines, for your automation around, you know, kind of the software factory floor. So I think automation is a key aspect of that because it, you know, it takes a lot of the drudgery out of it, for one thing, So now the humans have more time to spend on doing on the on the creative things on the things that we're good at a zoo. Humans and it also make sure that, you know, one of the things that computers are really good at is doing the same thing over and over and over and over. Eso that kind of puts that responsibility into the hands of the entity that that knows how to do that well, which is which is the machine eso I think it's, you know, it's a light. It's a deep, deep topic, obviously, but, you know, automation plays into it. Uh, you know, small batch sizes play into it, you know, being able to test very frequently whether that's testing in. You're kind of you're C I pipeline where you're sort of doing building mostly unit testing, maybe some integration testing, but also in layering in the mawr. Serious kinds of testing in terms of security scanning, penetration, testing, vulnerability, scanning. You know those sorts of things which, you know, maybe you do on every single see I Bill. But most people don't because those things tend toe take a little bit longer on. And you know you want your sea ice cycle to be as fast as possible because that's really in service of the developer who has committed code and wants toe kind of see the thumbs up from the system saying it. And, um, so most organizations most organizations are are are focusing on, you know, making sure that there's a follow on pipeline to follow on set of tests that happened after the C I passes successfully and and that's, you know, where a lot of the security scanning and those sorts of things happen. >>It's a It's an interesting problem. I mean, you mentioned, um, what almost sounds like a Lawrence Lessig Ian kind of idea that, you know, code is law in enterprises today, code particularly see, I code ends up being policy, but At the same time, there's, Ah, it seems to me there's a an alternative peril, which is, as you increase speed, particularly when you become more and more dependent on things like containers and layering technology to provide components and capabilities that you don't have to build yourself to your build pipeline, that there are new vulnerabilities, potentially that creep in and can creep in despite automation. Zor at least 1st. 1st order automation is attempts toe to prevent them from creeping in. You don't wanna you wanna freeze people on a six month old version of a key container image. But on the other hand, if the latest version has vulnerabilities, that could be a problem. >>Yeah, I mean, it's, you know, it's it's a it's a it's a double edged sword. It's two sides of the same coin. I think you know, when I talked to a lot of security people, um, you know, people to do it for a living is supposed to mean I just talk about it, um, that Z not completely true. But, um, the ah, lot of times the problem is old vulnerabilities. The thing that I think keeps a lot of people up at night isn't necessarily that the thing at the tip of the releases for particular, you know, well known open source, library or something like that. But that's gonna burn you all the vast majority of the time. And I want to say, like, 80 85% of the time. The vulnerability is that you that you get hosed by are ones that have been known about for years. And so I think the if I had to pick. So if you know, in that sort of two sides of that coin, if I had to pick, I would say Be aggressive in making sure that your third party dependencies are updated frequently and and continuously right, because that is the biggest, biggest cause of of of security vulnerabilities when it comes to third party code. Um, now you know the famous saying, You know, move fast and break things Well, there's certain things you don't want to break. You know you don't want to break a radiation machine that's going to deliver radio radiotherapy to someone because that will endanger their health. So So those sorts of systems, you know, naturally or subject a little bit more kind of caution and scrutiny and rigor and process those sorts of things. The micro service that I run that shows my little avatar when I log in, that one probably gets a little less group. You know, Andre rightfully so. So I think a lot of it has to do. And somebody once said in a I think it was, Ah, panel. I was on a PR say conference, which was, which was kind of a wise thing to say it was Don't spend a million dollars protecting a $5 assets. You know, you wanna be smart and you wanna you wanna figure out where your vulnerabilities they're going to come from and in my experience, and and you know, what I hear from a lot of the security professionals is pay attention to your supply chain. You're you want to make sure that you're up to date with the latest patches of, of all of your third party, you know, open source or close source. It doesn't really matter. I mean, if anything, you know, open source is is more open. Eso You could inspect things a little bit better than the close source, but with both kinds of streams of code that you consume and and use. You wanna make sure that you're you're more up to date as opposed to a less up to date? Um, that generally will be better. Now, can a new version of the library cause problems? You know, introduce bugs? You know, those sorts of things? Yes. That's why we have tests. That's what we have automated tests, regression, sweets, You know, those sorts of things. And so you wanna, you know, you wanna live in a in a world where you feel the confidence as a as a developer, that if I update this library from, you know, one debt owed at 3 to 1 debt owed at 10 to pick up a bunch of, you know, bug fixes and patches and those sorts of things. But that's not going to break some on demand in the test suites that that will run against that ought to cover that that sort of functionality. And I'd rather be in that world of Oh, yeah, we tried to update to that, but it But it broke the tests and then have to go spend time on that, then say, Oh, it broke the test. So let's not update. And then six months later, you do find out. Oh, geez. There was a problem in one that owed at three. And it was fixed in one. That about four. If only we had updated. Um, you know, you look at the, um you look at some of the highest profile security breaches that are out there that you sort of can trace toe third party libraries. It's almost always gonna be that it was out of date and hadn't been patched. That's so that's my you know, opinionated. Take on that. Sure. >>What are the parts of modern C I c D. As opposed to what one would encounter 56 years ago? Maybe if we can imagine that is being before the micro services and containers revolution really took off. >>You know, I think e think you're absolutely right that, you know, not the whole world is not doing. See, I Yeah, and certainly the whole world is not doing city yet. Um, you know, I think you know, as you say, we kind of live in a little bit of an ivory tower. You know, we live in an echo chamber in a little bit of a bubble Aziz vendors in this space. The truth is that I would say less than 50% of the software organizations out there do real. See, I do real CD. The number's probably less than that. Um, you know, I don't have anything to back that up other than just I talked to a lot of folks and work with, you know, with a lot of organizations and like, Yeah, that team does see I that team does Weekly builds You know, those sorts of things. It's it's really all over the place, Onda. Lot of times there's There's definitely, in my experience, a high correlation there with the amount of time that a team or a code base has been around, and the amount of sort of modern technologies and processes and and and so on that are that are brought to it on. And that sort of makes sense. I mean, if you if you're starting with the green field with a blank sheet of paper, you're gonna adopt, you know, the technologies and the processes and the cultures of today. A knot of 5, 10 15 15 years ago, Um but but most organizations air moving in that direction. Right? Andi, I think you know what? What? What? What's really changed in the last few years is the level of integration between the various tools between the various pieces and the amount of automation that you could bring to bear. I mean, I you know, I remember, you know, five or 10 years ago having all kinds of conversations with customers and prospects and and people of conferences and so on and they said, Oh, yeah, we'd like to automate our our software development life cycle, but, you know, we can't We have a manual thing here. We have a manual thing there. We do this kind of testing that we can automate it, and then we have this system, but it doesn't have any guy. So somebody has to sit and click on the screen. And, you know, and I used to say e used to say I don't accept No for an answer of can you automate this right? Everything. Anything can be automated. Even if you just get the little drinking bird. You know that just pokes the mouse. Everyone something. You can automate it, and I Actually, you know, I had one customer who was like, Okay, and we had a discussion and and and and they said, Well, we had this old Windows tool. We Its's an obscure tool. It's no longer updated, but it's it's it's used in a critical part of the life cycle and it can't be automated. And I said, Well, just install one of those Windows tools that allows you to peek and poke at the, you know, mass with my aunt I said so I don't accept your answer. And I said, Well, unfortunately, security won't allow us to install those tools, Eh? So I had to accept No, at that point, but But I think the big change were one of the biggest changes that's happened in the last few years is the systems now have all I'll have a p i s and they all talk to each other. So if you've gotta, you know, if you if you've got a scanning tool, if you've got a deployment tool, if you have a deployment, you know, infrastructure, you know, kubernetes based or, you know, kind of sitting in front of our around kubernetes thes things. I'll talk to each other and are all automated. So one of the things that's happened is we've taken out a lot of the weight states. A lot of the pauses, right? So if you you know, if you do something like a value stream mapping where you sit down and I'll date myself here and probably lose some of the audience with this analogy. But if you remember Schoolhouse Rock cartoons in in the late seventies, early eighties, there was one which was one of my favorites, and and the guy who did the music for this passed away last year, sadly, But, uh, the it was called How a bill Becomes a Law and they personified the bill. So the bill, you know, becomes a little person and, you know, first time passed by the house and then the Senate, and then the president either signs me or doesn't and or he vetoes, and it really sort of did this and what I always talk about with respect to sort of value stream mapping and talking about your processes, put a GoPro camera on your source codes head, and then follow that source code all the way through to your customer understand all of the stuff that happens to it, including nothing, right? Because a lot of times in that elapsed time, nothing keeps happening, right. If we build software the way we were sorry. If we build cars the way we build software, we would install the radio in a car, and then we would park it in a corner of the factory for three weeks. And then we might remember to test the radio before we ship the car out to the customer. Right, Because that's how a lot of us still develop some for. And I think one thing that's changed in the in the last few years is that we don't have these kind of, Well, we did the bill. So now we're waiting for somebody to create an environment and rack up some hardware and install an operating system and install. You know, this that and the other. You know, that that went from manual to we use Scheffer puppet to do it, which then went to we use containers to do it, which then went to we use containers and kubernetes to do it. So whole swaths of elapsed time in our software development life cycles basically went to nothing, right and went to the point where we can weaken, weaken, configure them way to the left and and and follow them all the way through. And that the artifact that we're delivering isn't necessarily and execute herbal. It could be a container, right? So now that starts to get interesting for us in terms of being able to test against that container scan against that container, def. Against that container, Um, you know, and it, you know, it does bring complexity to in terms of now you've got a layered file system in there. Well, what all is in there, you know, And so there's tools for scanning those kinds of things, But But I think that one of the biggest things that's happened is a lot of the natural pause. Points are no longer natural. Pause points their unnatural pause points, and they're now just delays in yourself for delivery. And so what? What a lot of organizations are working on is kind of getting to the point where those sorts of things get get automated and connected, and that's now possible. And it wasn't 55 or 10 years ago. >>So It sounds like a great deal of the speed benefit, which has been quantified many different ways. But is once you get one of these systems working, as we've all experienced enormous, um, is actually done by collapsing out what would have been unused time in a prior process or non paralyze herbal stuff has been made parallel. >>I remember doing a, uh, spent some time with a customer, and they did a value stream mapping, and they they found out at the end that of the 30 days of elapsed time they were spending three days on task. Everything else was waiting, waiting for a build waiting foran install, waiting for an environment, waiting for an approval, having meetings, you know, those sorts of things. And I thought to myself, Oh, my goodness, you know, 90% of the elapsed time is doing nothing. And I was talking to someone Gene Kim, actually, and I said, Oh my God, it was terrible that these you know, these people are screwed and he says, 0 90%. That's actually pretty good, you know? So So I think you know, if you if you think today, you know, if you If you if you look at the teams that are doing just really pure continuous delivery, you know, write some code committed, gets picked up by the sea ice system and passes through CIA goes through whatever coast, see I processing, you need to do security scanning and so on. It gets staged and it gets pushed into production. That stuff can happen in minutes, right? That's new. That's different. Now, if you do that without having the right automated gates in place around security and and and and those sorts of things you know, then you're living a little bit dangerously, although I would argue not necessarily any more dangerously, than just letting that insecure coat sit around for a week before your shipment, right? It's not like that problem is going to fix itself if you just let it sit there, Um, but But, you know, you definitely operated at a higher velocity. Now that's a lot of the benefit that you're tryingto trying to get out of it, right? You can get stuff out to the market faster, or if you take a little bit more time, you get more out to the market in, in in the same amount of time you could turn around and fix problems faster. Um, if you have a vulnerability, you can get it fixed and pushed out much more quickly. If you have a competitive threat that you need to address, you can you know, you could move that that much faster if you have a critical bug. You know, I mean, all security issues or bugs, sort of by definition. But, you know, if you have a functionality bug, you can you can get that pushed out faster. Eso So I think kind of all factors of the business benefit from from this increase in speed. And I think developers due to because anybody you know, any human that has a context switch and step away from something for for for, you know, duration of time longer than a few minutes, you know, you're gonna you're gonna you're gonna you're gonna have to load back up again. And so that's productivity loss. Now, that's a soft cost. But man, is it Is it expensive and is a painful So you see a lot of benefit there. Think >>if you have, you know, an organization that is just starting this journey What would you ask that organization to consider in orderto sort of move them down this path? >>It's by far the most frequent and almost always the first question I get at the end of the talk or or a presentation or something like that is where do we start? How do I know where to start? And and And there's a couple of answers to that. What one is Don't boil the ocean, right? Don't try to fix everything all at once. You know that because that's not agile, right? The be agile about your transformation Here, you know, pick, pick a set of problems that you have and and make a, you know, basically make a burn down list and and do them in order. So find find a pain point that you have right and, you know, just go address that and and try to make it small and actionable and especially early on when you're trying to affect change. And you're tryingto convinced teams that this is the way to go and you may have some naysayers, or you may have people who are skeptical or have been through these processes before that have been you know failures released, not the successes that they that they were supposed to be. You know, it's important to have some wind. So what I always say is look, you know, if you have a pebble in your shoe, you've got a pain point. You know how to address that. You know, you're not gonna address that by changing out your wardrobe or or by buying a new pair of shoes. You know, you're gonna address that by taking your shoe off, shaking it until the pebble falls out there putting the shoe back on. So look for those kinds of use cases, right? So if you're engineers are complaining that whenever I check in the build is broken and we're not doing see, I well, then let's look at doing C I Let's do see eye, right? If you're not doing that. And for most organizations, you know, setting up C I is a very manageable, very doable thing. There's lots of open source tooling out there. There's lots of commercial tooling out there. Thio do that to do it for small teams to do it for large teams and and everything in between. Um, if the problem is Gosh, Every time we push a change, we break something. You know where every time something works in staging it doesn't work in production. Then you gotta look at Well, how are these systems being configured? If you're If you're configuring them manually, stop automate the configuration of them. Um, you know, if you're if you're fixing system manually, don't you know, as a friend of mine says, don't fix, Repave? Um, you know, you don't wanna, you know, there's a story of, you know how how Google operates in their data centers. You know, they don't they don't go look for a broken disk drive and swap it out. You know, when it breaks, they just have a team of people that, like once a month or something, I don't know what the interval is. They just walked through the data center and they pull out all the dead stuff and they throw it out, and what they did was they assume that if the scale that they operate, things are always going to break physical things are always going to break. You have to build a software to assume that breakage and any system that assumes that we're going to step in when a disk drive is broken and fix it so that we can get back to running just isn't gonna work at scale. There's a similarity. There's sort of ah, parallel to that in in software, which is you know, any time you have these kinds of complex systems, you have to assume that they're gonna break and you have to put the things in place to catch those things. The automated testing, whether it's, you know, whether you have 10,000 tests that you that you've written already or whether you have no tests and you just need to go right, your first test that that journey, you've got to start somewhere. But my answer thio their questions generally always just start small, pick a very specific problem. Build a plan around it, you know, build a burned down list of things that you wanna address and just start working your way down that the same way that you would for any, you know, kind of agile project, your transformation of your own processes of your own internal systems. You should use agile processes for those as well, because if you if you go off for six months and and build something. By the time you come back, it's gonna be relevant. Probably thio the problems that you were facing six months ago. >>A Then let's consider the situation of, ah, company that's using C I and maybe sea ice and C d together. Um, and they want to reach what you might call the next level. Um, they've seen obvious benefits they're interested in, you know, in increasing their investment in, you know and cycles devoted to this technology. You don't have to sell them anymore, but they're looking for a next direction. What would you say that direction should be? I >>think oftentimes what organizations start to do is they start to look at feedback loops. So on DAT starts to go into the area of sort of metrics and analytics and those sorts of things. You know what we're we're always concerned about? You know, we're always affected by things like meantime to recovery. Meantime, the detection, what are our cycle times from, you know, ideation, toe codecommit. What's the cycle? Time from codecommit the production, those sorts of things. And you know you can't change what you don't measure eso so a lot of times the next step after kind of getting the rudimentary zoo of C I Orsini or some combination of both in places start to measure. Stop you, Um, and and then but But there. I think you know, you gotta be smart about it, because what you don't want to do is kind of just pull all the metrics out that exists. Barf them up on the dashboard. And the giant television screens say boom metrics, right. You know, Mike, drop go home. That's the wrong way to do it. You want to use metrics very specifically to achieve outcomes. So if you have an outcome that you want to achieve and you can tie it to a metric start looking at that metric and start working that problem once you saw that problem, you can take that metric. And you know, if that's the metric you're showing on the big you know, the big screen TV, you can pop that off and pick the next one and put it up there. I I always worry when you know a little different when you're in a knock or something like that. When when you're looking at the network stuff and so on. But I'm always leery of when I walk into to a software development organization. You know, just a Brazilian different metrics, this whole place because they're not all relevant. They're not all relevant at the same time. Some of them you wanna look at often, some of them you just want to kind of set an alarm on and make sure that, you know, I mean, you don't go down in your basement every day to check that the sump pump is working. What you do is you put a little water detector in there and you have an alarm go off if the water level ever rises above a certain amount. Well, you want to do the same thing with metrics, right? Once you've got in the water out of your basement, you don't have to go down there and look at it all the time. You put the little detector in, and then you move on and you worry about something else. And so organizations as they start to get a little bit more sophisticated and start to look at the analytics, the metrics, um, start to say, Hey, look, if our if our cycle time from from, you know, commit to deploy is this much. And we want it to be this much. What happens during that time, And where can we take slices out of that? You know, without without affecting the outcomes in terms of quality and so on, or or if it's, you know, from from ideation, toe codecommit. You know what? What can we do there? Um, you start to do that. And and then as you get those sort of virtuous cycles of feedback loops happening, you know, you get better and better and better, but you wanna be careful with metrics, you know, you don't wanna, you know, like I said, you don't wanna barf a bunch of metrics up just to say, Look, we got metrics. Metrics are there to serve a particular outcome. And once you've achieved that outcome, and you know that you can continue to achieve that outcome, you turn it into an alarm or a trigger, and you put it out of sight. And you know that. You know, you don't need to have, like, a code coverage metric prominently displayed you you pick a code coverage number that you're happy with you work to achieve that. Once you achieve it, you just worry about not going below that threshold again. So you can take that graph off and just put a trigger on this as if we ever get below this, you know, raising alarm or fail a build or fail a pipeline or something like that and then start to focus on improving another man. Uh, or another outcome using another matter >>makes enormous sense. So I'm afraid we are getting to be out of time. I want to thank you very much on this for joining us today. This has been certainly informative for me, and I hope for the audience, um, you know, thank you very, very much for sharing your insulin.

(soft upbeat music) (crowd chattering) >> Over the past 18 to 24 months, chief information security officers have dramatically changed their priorities. They had to, to support the remote work trend. So things like endpoint security, cloud security, and in particular identity and access management became top of mind. And a whole shift occurred. And we're going to talk about that today. Hi everybody, this is Dave Vellante and you're watching theCUBE. We're here at AWS re:Invent 2021. Katie Curtin-Mestre is here. She's the vice president of marketing at CyberArk and Bar Lavie senior product manager at Cloud Identity and Security. Bar, sorry for botching your name, but folks welcome to theCUBE, great to see you. >> Glad to be here. >> Great to hear. >> So Katie, upfront I talked about some of those trends. It's been a hugely dramatic shift away from this kind of traditional approaches to cyber. What are some of the trends that CyberArk has seen? >> Well, Bar is going to take the first part of this. >> Great, just go on. (Bar laughing) >> Yeah, so one trait that we are seeing is that cloud migration projects accelerate as organization turbocharged digital transformation. Is they're a looking to take advantage off the agility and operational efficiency of the cloud providers. Some of the concerns that I can think about one of those is the reducing the potential loss of data that is caused due to the excessive access to resources. And the other one is provision secure and scalable access to resources. And the third one would be implementing least privilege for all type of identity whether if it's a human identity or non-human identity. >> And on that end Dave, we recently commissioned a survey with the Cloud Security Alliance. We co-sponsored a survey and found that 94% of respondents said that securing human permissions was a top security challenge and machine identities weren't far behind at 77%. Another challenge that we're hearing from our customers is the need to secure the secrets used by applications. So we're really excited by today's news from AWS. They announced some new capabilities with a code guru called Secret Detector that helps to find unsecured secrets in applications. And the other concern that we're hearing from our customers is the need to monitor and audit the activity of all of their cloud identities. This is really important to help their security operation teams with their investigations and also to meet audit and compliance requirements. >> So the definition of identity is now more encompassing and includes like you say machines, right? It's not just people anymore. Of course we've seen, you know, phishing has always been problematic. It's escalated daily, right? We get phished. I mean, are we going to see the day where we finally get rid of passwords? Is that even possible? But maybe we could talk a little bit about sort of identity, how identity is evolving, this notion of zero trust. Zero trust used to be a Password. So, maybe Bar you could talk a little bit about what you're seeing in terms of identity access management. Maybe privileged access management are those things coming together? How does CyberArk think about those things? >> You going to take this one Katie >> Well, what CyberArk sees is we definitely see a trend where access management and privileged access management are coming together. Security teams are struggling too many security tools and they're really looking to standardize on a small handful of vendors and get more bank for their buck from their security investment. So we're definitely seeing that trends of unified platforms across access and privileged access management to secure any identity, whether human or machine from kind of like your standard workforce identity, to those who have highly privileged access. >> I don't know if you've ever, ever seen that chart. I think Optiv puts it out. It's consultancy. And it's this eye chart. It's a taxonomy of all the different security I have published at a number of times. it's mind boggling. So CSOs, SecOps teams they have to manage all this complexity, all these different tools and you ask CSOs what's your biggest challenge? They'll tell you lack of skills. We just can't find people. We can't train them fast enough. So what's CyberArk working on? What are some of the key initiatives that you guys are focused on that people should know about? >> Well, one of the things that we're working on is actually, and we see a greater adoption of it is something that was actually started as an initiative within our innovation lab. It's a CyberArk Clouding Titles Manager, which help to detect and remediate excessive permissions to cloud resources for any type of identity. I mentioned before the both human and non-human. Which are the something that you were looking to to secure. Another solution that we see a great adoption is our circuit ranger which helps organization to re remove the necessity of having a hard-coded credentials within application. It can be either traditional applications for their own premise or even cloud native applications. And peg this also into your CI CD pipeline. And we are actually innovating in these type of area with AWS as well. So this is one of the great things that we were doing. Also we're investing on a new solution for just-in-time access for cloud VMs and cloud consoles. And all of these solutions that I've mentioned and more to that are part of our identity security platform which came to provide you with the suite of solution to apply least privilege and secure access to any type of resource from any device for any type of identity. >> So is that best practice? I mean, if you had to, you know, advise a customer on best practice in identity, how should they think about that? Where should they start? >> Well, on the best practices front we recently published an ebook with AWS. And it's focused on the shared responsibility model and foundational best practices for securing cloud access. And it's all part of an initiative that CyberArk has, which is our identity security blueprint. Which guides customers on how best to move forward with their identity security initiatives. >> So where do they start? First of all how do they get that is it a security website or? >> It's available on our website and we detailed some of the steps that that customers can take. For example, one of the steps that we recommend to our customers is to limit the use of the root account and also to very much lock down the root account to use federated identities whenever possible. And Bar already alluded to some of the other best practices that we recommend. Such as removing hard-coded credentials from secrets. Another best practice that we really recommend to our customers is to have a consistent set of controls across their entire estate. Both from on-premises to the cloud. And this really helps to reduce complexity by having a unified and consistent set of security controls. And in fact one of our customers who is one of the world's largest convenience chains. They're using CyberArk to secure the credentials both for their on-premise servers and their AWS EC2 instances. And they're also using us as well to secure the credentials used by applications in the CI CD pipeline. So getting to those consistent controls is another best practice we highly recommend. >> So, consistent identity across your state, whether it's on-prem or in the cloud. And then also you've referenced CI CD a couple of times. So it's it's developer friendly? Are you're designing security in as opposed to a bolt on after the fact? And then you mentioned root accounts access. Is that where privilege access management comes in? Are we going to treat everybody as privileged access? Or how do you deal with machines? You mentioned hard-coded? Like some machines are hard-coded. Like I would imagine a lot of these internet cameras are exposures. How do you deal with all that? I mean, do you just have to cycle through and modernize your fleet of machines? Are there ways in which CyberArk can help sort of anticipate that or defend against that? >> Well, CyberArk can help on, on multiple fronts. Of course you need to secure the root account but that's just only one example of needing to secure a privilege access. And one thing that customers need to understand is that now going forward, any identity can have privilege access at any point in time, because at any point and time, you yourself could have access to a highly sensitive system or have access to highly sensitive data. So with CyberArk we help our customers understand which of their applications and infrastructure have the most sensitive data and then work with them to secure the access to that data whether that access be a human access or machine or programmatic access. >> So what are the customer implications of all this? I mean pre pandemic, you know, this whole zero trust thing with password. Now it's like fundamental premise. You don't trust to verify. What are the customer implications as we enter this new era ransomware through the roof, the adversaries are well funded highly capable. They're living off the land, they're island hopping. They're, doing self forming malware. It's a new world, right? So what are the customer implications? What should they be thinking about? You know, they don't have unlimited budget. So what's the advice? >> Well, eventually at the end of the day, there are all kinds of best practices of how to applies security. I think that both AWS have their own best practices and CyberArk has also our own best practices calling the blueprint which help organization to focus on to crown jewel on the most important stuff. And then going deeper and lower within each and every initiative. And on each and every level, try to investigate what you're trying to protect and what kind of security mechanisms can be applied in order to protect both access and maintaining that no one whether if it's internal or external attacker can gain access to it. >> Yup, I think the other implication for customers and you already alluded to it is really to continue to move forward with their zero trust initiatives. I think that that is a foundational going forward. Now that remote work is kind of the defacto norm and we can no longer rely on the traditional network perimeter. And so in this new environment securing your identities is the new perimeter. So that's an important implication for customers. And then another one that I would mention is that security teams need to work more closely with their dev and dev ops counterparts to bacon security earlier. It really can't be that security is brought in after the fact. Security very much needs to shift left and be included in the very early stages of application development before an application comes to production. >> I mean, I think it's that last point but all good points. The last point was a huge theme at CubeCon this year. That notion of shift left developers, you've mentioned the CI CD pipeline several times. I mean I think that is, you know, especially when you think about machines and the edge and IoT. I used to say all the time, you know that you used to put a moat around the castle, build a wall, protect the queen. Well, the queen has left the castle. But now with the pandemic, we've seen the effects of that. And as I say, the adversaries are seeing huge opportunities. Well-funded super sophisticated. It's like it makes Stuxnet look like a kindergarten. I know that was still >> That's scary. still pretty sophisticated. But I mean, look at what we saw with the government hack and solar winds, you know huge huge. But if we can talk to CSOs about that, they're like, you know, that's, we have to move fast. But they don't have unlimited budget, right? Cybersecurity is their number one initiative in terms of priorities. But then they have all these other things to fund. They have to fund a forced march to digital transformation, machine learning and AI, they're migrating to the cloud. They're driving automation. They're modernizing their application portfolio. So, security is still number one, isn't it? So it's a good business that you're in. >> Yes, and we really want to work with our CSOs so they can get the most investment out of what they're putting into CyberArk and the rest of their strategic security vendors. Because as you mentioned there's a talent shortage. So anything that we can do as vendors to make it easier for them to use our products and get more value from our solutions, is something that's really important. >> And automation is part of the answer but it's not the only answer, right? You got to follow the NIST framework and follow these best practices and keep fighting the fight. Guys. Thanks so much for coming on theCUBE. It was great to have you. I'd love to have you back. >> Thanks for having us. >> Thank you for having us. >> All right. Our pleasure. All right, this is Dave Vellante for theCUBE. You're watching our coverage of AWS re:Invent 2021. (gentle upbeat music)

>> Narrator: From around the globe, it's theCUBE with coverage of KubeCon and CloudNativeCon Europe 2021 virtual. Brought to you by Red Hat, The Cloud Native Computing Foundation, and ecosystem partners. >> Hey, welcome back to theCUBE's coverage of KubeCon 2021 CloudNativeCon Europe. Part of the CNCF and ongoing, could be in there from the beginning, love this community, theCUBE's proud to support and continue to cover it. We're virtual this year again because of the pandemic but it looks like we'll be right around the corner for a physical event, hopefully for the next one, fingers crossed. Got a great guest here from Red Hat. Siamak Sadeghianfar, a Senior Principal Product Manager. Welcome to theCUBE. Thanks for coming in. >> Thank you for having me. >> So, this topic's about GitOps, Pipelines, code. Obviously Infrastructure as Code has been the ethos since I can remember going back to 2008 and the original cloutaroti vision. And we were always talking about that. Now it's mainstream. Now it's DevSecOps. So, it's now, day two operations, shifting left with security. OpenShift is continuing to get, take ground. Congratulations on that. So my first question is you guys announced the general availability of OpenShift Pipelines and GitOps at KubeCon. What are, what's this about? And what's the benefits for the customer. Let's get into the news >> Thanks for, to begin with for the Congress and this, this is definitely a hot topic around the DevSecOps. And the different variations of that year about some versions that during in, in FinTech and other verticals as well. The idea is here really is that CI/CD has been around for a long time, continuous integration and continuous delivery, as one of the core practices of the DevOps movement. DevOps movement is quite widespread, now. You, you see reports of above 90% of organizations are in the process of adoption in their journey. And this is one of the main practices but something that has become quite apparent is that many of these organizations that are investing more and more in Cloud Native apps and adopting Cloud Native ways of building applications the tooling and technology that they use for CI/CD since CI/CD is nothing new is from 10 years old, five years old pre Kubernetes era which is not quite Cloud Native. So there is always a clash of how do I build Cloud Natives application using these technologies that are not really built for Cloud Native space and an OpenShift Pipelines OpenShift GitOps is really an opening in this direction and bring more Cloud Native ways of continuous integration and continuous delivery to customers on OpenShift. >> Got it, so I got to ask you, so a couple of questions on this topic, I really want to dig into. Can you describe the Cloud Native CI/CD process versus traditional CI/CD? >> Sure, so traditional when we think about CI/CD there is usually this monolithic solutions that are running on a virtual machine on a type of infrastructure that they use to deploy applications as well. 'Cause you, you need reliability and you have to be making an assumption about an infrastructure that you're running on. And when you come to Cloud Native infrastructure you have a much more dynamic infrastructure. We have a lot less assumptions. You might be running on a public cloud or on premise infrastructure or different types of public cloud. So these environments are often also containerized. So there are, there's a high chance you're running on a container platform, regardless if it's a public or on premises. And with the whole containers, you, you have different types of disciplines and principals to think in, about your infrastructure. So in the Cloud Native ways of CI/CD, you're running most likely in a container platform. You don't have dedicated infrastructure. You are running mostly on demand. You scale when there is a demand for running CI/CD, for example, rather than dedicated infrastructure to it. And also from the mode of operation from organization perspective, they are more adapted to this decentralized ways of ownership. As a part of the DevOps culture, this comes really with that movement, that more and more development teams are getting ownership of some portion of the delivery of their applications. And it's cognitive CS/CD solutions, they focus on supporting these models that you go away from that central model of control to decentralize and have more ownership, more capabilities within the development teams for delivering application. >> Okay, so I then have to ask you the next question. It's like you, like a resource, you'd say: Hey Siri, what is, what is GitOps? What is GitOps? 'Cause that's the topic that's been getting a lot of traction, everyone's talking about it. I mean we know DevOps. So what is the GitOps model? Can you define that? And is that what a, it that what comes after DevOps? Is it DevOps 2.0, what is the GitOps model? >> That's a very good question. GitOps is nothing really new. It's rather a more descriptive way of DevOps principles. DevOps talks about the cultural changes and mindset and ways of working. And when it comes to the, to the concrete work flow it is quite open for interpretation. So GitOps is one, a specific interpretation of how you, you do continuous integration and continuous delivery, how we implement DevOps. And the concept have been around for a couple of years. But just recently, it's got a lot of traction within the Cloud Native space. >> So how does GitOps fit into Kubernetes then? 'Cause that's going to be the next dot that we want to connect. What is that, what is, how, how. How does GitOps fit into Kubernetes? >> So GitOps is really the, the core principle of GitOps is that you, you, you think about everything in your infrastructure and application in a declarative manner. So everything needs to be declared in, in, in a number of gate repositories and you drive your operations through Git Workflows. Which if you think about it is quite similar to how Kubernetes operates. The, the reason Kubernetes became so popular is because of this declarative way of thinking about your infrastructure. You declare what you expect and Kubernetes actualizes that on, on some sort of infrastructure. So GitOps is, is, is exact same concept, but the, but applied not to the infrastructure itself, but to the operations of that infrastructure, operations of those applications. It becomes a really nice fit together. It's the same mindset really applied in different place. >> It's like Kubernetes is like the linchpin or the enabler for GitOps. Just a whole nother level of, I mean, I think GitOps essentially DevOps 2.0 in my opinion because it takes this whole nother level above that for the developer modern developer because it allows them to do more. So it's been around for a while. We've been talking about this, it's got a new name but GitOps is kind of concept has been around. Why is the increase adoption happening now in your opinion or do you have any data on or any facts or opinion on why it's such an increase in, in conversation and adoption? >> You had the, you had like very accurate point there that Kubernetes has been a great enabler for, for DevOps and later the same applies to GitOps as well because of that, that great fit. It has been, GitOps the concept has been there but implementation of that has been quite difficult before Kubernetes and also for non-containerized environments. Kubernetes is, is a very potent platform for this kind of operation because the the mindset and the ways of working is really native to how Kubernetes thinks. But there is also another driver that has been influential in, in the rise of GitOps in the last year or two. And this is an observation we see at a lot of our customers, that the number of clusters that organizations are deploying, Kubernetes clusters increasing. As their maturity increases they get more comfortable with Cloud Native way of working and transfer the workflows to become Cloud Native, they are, they are having, they move more and more of their infrastructure to Kubernetes clusters. So a new challenge rises with this. And now that I have a larger number of clusters how do I ensure consistency across all these, all these clusters? So before I had to deploy an application to production environment, perhaps, which meant two clusters across two geographical zones. Now I have to deploy to 20 clusters. And these 20 clusters also change over time. So this week is a different 20 clusters then three weeks from now. So this, this dynamic ways of working and the customers maturing in, in dealing with Kubernetes operating communities has increased really the pace of adoption of GitOps because it addresses a lot of those challenges that customers are dealing with in this space. >> Yeah, you bring up a really good challenge there. And I think that's worth calling out, this idea of expansion. And I won't say sprawl because it's not a sprawl of cluster. It's more a state provisioning and standing up clusters. And you said they they're changing because the environment has needs and the workloads might have requirements. This makes total sense in a DevOps kind of GitOps way. So I get that and I see that definitely happening. So this brings up the question, if I'm a customer, what I'm worried about is I don't want to have that Hadoop factor where I build a cluster and it takes too long to manage it, or I can't measure it, or understand the data, or have any observability. So I want to have an ease of provisioning and standing up and I want to have consistency that my apps who are using it, don't have to be, you know mangled with or coded with. So, you know, this combination of ease of deploying, ease of integrating, ease of consuming the clusters becomes a service model. Can you share your thoughts on how that gets solved? >> Yeah, absolutely. So that, that's a great point because as, as this is happening, there is also heterogenesis in this, this type of Kubernetes infrastructure window. Like, they're all Kubernetes but this problem also has multiple facets as customers running on multiple public clouds and, and combination of that with their on-premise Kubernetes clusters. And that is, they may as well be OpenShift across all this, all this infrastructure. But the, the problem that GitOps helps its customers advise that they can have the exact same operational model across all these apps and infrastructure, regardless of what kind of application it is. And regardless of where OpenShift is installed or if you're using that combined with a public cloud managed a Kubernetes stats, is the exact same process because you're relying on, on the Gits Workflows, right? And even beyond that, this standard workflow has the benefit of something that many organizations are already familiar with. So if you think about what GitOps operations mean it is essentially what developers have been always using for developing applications. So this standardizes the operations of both application and infrastructure as solvers. >> Listen to me, I got to ask you as the product manager on the whole pipelining in Kubernetes deployments. In your opinion, share your perspective on, real quick, on Kubernetes, where we're at? Because just the accelerated adoption has been phenomenal. We've seen it mature this year at KubeCon. And certainly when KubeCon North America happens, you're going to see more and more end user participation. You're going to see much more end-user use cases. You mentioned clusters are growing. What's the state of Kubernetes from your perspective, from a developer mindset? >> So Kubernetes, I think it has moved from a place that it was seen as only a, a type of infrastructure for Cloud Native applications because of the capability that it provides to a type of infrastructure for any type of application, any type of workload. I think what we have seen over the last two years is, is a shift to expansion of the use cases. And if, if you are, you talked about head open if you are a data scientist, or if you are an AIML type of developer or any type of workload really, see use cases that are coming to the Kubernetes platform as the targets type of infrastructure. So that's really where we see Kubernetes at right now is the really, the preferred infrastructure for any type of workload. And I believe this trend going to to keep continuing to address any of the challenge that exists that prevents maybe part of the, a particular type of workload to address that within the platform and opens that to add to, to developers. Which means for the developers now, once you learn the platform you are really proficient in a, you have this skills for any type of application or any type of infrastructure because they're all standardized, regardless of what type of application or workloads or technology you're specialized in. They're all going to the exact same platform. So it's very standardized type of skills across organizations, different type of teams that they have. >> Awesome, great, thanks for sharing that insight and definition. You're like a walking dictionary today for our CUBE audience. Thank you for all this good stuff. Appreciate it. Final question for you is, what does it mean for developers that are using Jenkins or other cloud-based CI solutions like GitHub Actions? What, what's the impact to them with all this from a working standpoint? 'Cause obviously you've got to make it workable. >> Right, so it's CI/CD also like it's, it's it's great to see like with DevOps adoption, there are many organizations that already have processes in place. They have, they're already using a CI tool or a CD tool. They might be using Jenkins. A lot of organizations really use, use Jenkins even though it comes with challenges and you might be using public cloud services or cloud-based CI tools, like you have Actions, you have pipelines and so on. So we are very well aware of the existing investment that many organizational teams have made. And we make sure that OpenShift as a platform works really well alongside all these different types of CI and CD technology that exists. We want to make sure that for developers starting on OpenShift, they, they have a really solid Cloud Native foundation for CI/CD. They have of strategies included but replaceable type of strategies. So they, they have a supportive platform that is Cloud Native, that gives them capability that matches the type of Cloud Native workloads that they have on the platform but also integrate well with existing tooling that exists around CI/CD. So that they can match and choose if they want to replace a piece of that with an existing investment that they have done, integrated with the rest of the platform. >> Awesome, well, great to have you on. Having the principal product manager is awesome, to talk about the two new announcements here. OpenShift pipe, Pipelines, and OpenShift GitOps. Final, final question, bumper sticker this for the audience. What's the bottom line with OpenShift Pipelines and GitOps? What's the, what's the bottom line benefit for customers? >> It's a, so OpenShift Pipeline and OpenShift GitOps makes it really simple for customers to create Cloud Native Pipelines and GitOps model for delivering application. And also making cluster changes across a large range of clusters that they have, make it really simple to grow from that point to many, many clusters and still manage the complexity of this complex infrastructure that it will be growing into. >> All right, Siamak Sadeghianfar, Senior Principal Product Manager at Red Hat. Here for the KubeCon + CloudNativeCon, Europe. CUBE conversation, thanks for coming on, appreciate it. >> Thanks John, thanks for having me. Okay, CUBE coverage continues. I'm John Farrow with theCUBE. Thanks for watching. (upbeat music)

>>mhm Yes, Welcome back to the cubes coverage of red hat summit 2021 virtual, which we were in person this year but we're still remote. We still got the Covid coming around the corner. Soon to be in post. Covid got a great guest here, Clayton Coleman architect that red hat cuba love and I've been on many times expanded role again this year. More cloud, more cloud action. Great, great to see you. Thanks for coming on. >>It's a pleasure >>to be here. So great to see you were just riffing before we came on camera about distributed computing uh and the future of the internet, how it's all evolving, how much fun it is, how it's all changing still. The game is still the same, all that good stuff. But here at Red had some and we're gonna get into that, but I want to just get into the hard news and the real big, big opportunities here you're announcing with red hat new managed cloud services portfolio, take us through that. >>Sure. We're continuing to evolve our open shift managed offerings which has grown now to include um the redhead open shift service on amazon to complement our as your redhead open shift service. Um that means that we have um along with our partnership on IBM cloud and open ship dedicated on both a W S and G C P. We now have um managed open shift on all of the major clouds. And along with that we are bringing in and introducing the first, I think really the first step what we see as uh huh growing and involving the hybrid cloud ecosystem on top of open shift and there's many different ways to slice that, but it's about bringing capabilities on top of open shift in multiple environments and multiple clouds in ways that make developers and operation teams more productive because at the heart of it, that's our goal for open shift. And the broader, open source ecosystem is do what makes all of us safer, more, uh, more productive and able to deliver business value? >>Yeah. And that's a great steak you guys put in the ground. Um, and that's great messaging, great marketing, great value proposition. I want to dig into a little bit with you. I mean, you guys have, I think the only native offering on all the clouds out there that I know of, is that true? I mean, you guys have, it's not just, you know, you support AWS as your and I B M and G C P, but native offerings. >>We do not have a native offering on GCPD offered the same service. And this is actually interesting as we've evolved our approach. You know, everyone, when we talk about hybrid, Hybrid is, um, you know, dealing with the realities of the computing world, We live in, um, working with each of the major clouds, trying to deliver the best immigration possible in a way that drives that consistency across those environments. And so actually are open shift dedicated on AWS service gave us the inspiration a lot of the basic foundations for what became the integrated Native service. And we've worked with amazon very closely to make sure that that does the right thing for customers who have chosen amazon. And likewise, we're trying to continue to deliver the best experience, the best operational reliability that we can so that the choice of where you run your cloud, um, where you run your applications, um, matches the decisions you've already made and where your future investments are gonna be. So we want to be where customers are, but we also want to give you that consistency. That has been a hallmark of um of open shift since the beginning. >>Yeah. And thanks for clarifying, I appreciate that because the manage serves on GCB rest or native. Um let me ask about the application services because Jeff Barr from AWS posted a few weeks ago amazon celebrated their 15th birthday. They're still teenagers uh relatively speaking. But one comment he made that he that was interesting to me. And this applies kind of this cloud native megatrend happening is he says the A. P. I. S are basically the same and this brings up the hybrid environment. You guys are always been into the api side of the management with the cloud services and supporting all that. As you guys look at this ecosystem in open source. How is the role of A PS and these integrations? Because without solid integration all these services could break down and certainly the open source, more and more people are coding. So take me through how you guys look at these applications services because many people are predicting more service is going to be on boarding faster than ever before. >>It's interesting. So um for us working across multiple cloud environments, there are many similarities in those mps, but for every similarity there is a difference and those differences are actually what dr costs and drive complexity when you're integrating. Um and I think a lot of the role of this is, you know, the irresponsible to talk about the role of an individual company in the computing ecosystem moving to cloud native because as many of these capabilities are unlocked by large cloud providers and transformations in the kinds of software that we run at scale. You know, everybody is a participant in that. But then you look at the broad swath of developer and operator ecosystem and it's the communities of people who paper over those differences, who write run books and build um you know, the policies and who build the experience and the automation. Um not just in individual products or an individual clouds, but across the open source ecosystem. Whether it's technologies like answerable or Terror form, whether it's best practices websites around running kubernetes, um every every part of the community is really involved in driving up uh driving consistency, um driving predictability and driving reliability and what we try to do is actually work within those constraints um to take the ecosystem and to push it a little bit further. So the A. P. I. S. May be similar, but over time those differences can trip you up. And a lot of what I think we talked about where the industry is going, where where we want to be is everyone ultimately is going to own some responsibility for keeping their services running and making sure that their applications and their businesses are successful. The best outcome would be that the A. P. R. S are the same and they're open and that both the cloud providers and the open source ecosystem and vendors and partners who drive many of these open source communities are actually all working together to have the most consistent environment to make portability a true strength. But when someone does differentiate and has a true best to bring service, we don't want to build artificial walls between those. I mean, I mean, that's hybrid cloud is you're going to make choices that make sense for you if we tell people that their choices don't work or they can't integrate or, you know, an open source project doesn't support this vendor, that vendor, we're actually leaving a lot of the complexity buried in those organizations. So I think this is a great time to, as we turn over for cloud. Native looking at how we, as much as possible try to drive those ap is closer together and the consistency underneath them is both a community and a vendor. And uh for red hat, it's part of what we do is a core mission is trying to make sure that that consistency is actually real. You don't have to worry about those details when you're ignoring them. >>That's a great point. Before I get into some architectural impact, I want to get your thoughts on um, the, this trends going on, Everyone jumps on the bandwagon. You know, you say, oh yeah, I gotta, I want a data cloud, you know, everything is like the new, you know, they saw Snowflake Apollo, I gotta have some, I got some of that data, You've got streaming data services, you've got data services and native into the, these platforms. But a lot of these companies think it's just, you're just gonna get a data cloud, just, it's so easy. Um, they might try something and then they get stuck with it or they have to re factor, >>how do you look >>at that as an architect when you have these new hot trends like say a data cloud, how should customers be thinking about kicking the tires on services like that And how should they think holistically around architect in that? >>There's a really interesting mindset is, uh, you know, we deal with this a lot. Everyone I talked to, you know, I've been with red hat for 10 years now in an open shift. All 10 years of that. We've gone through a bunch of transformations. Um, and every time I talked to, you know, I've talked to the same companies and organizations over the last 10 years, each point in their evolution, they're making decisions that are the right decision at the time. Um, they're choosing a new capability. So platform as a service is a great example of a capability that allowed a lot of really large organizations to standardize. Um, that ties into digital transformation. Ci CD is another big trend where it's an obvious wind. But depending on where you jumped on the bandwagon, depending on when you adopted, you're going to make a bunch of different trade offs. And that, that process is how do we improve the ability to keep all of the old stuff moving forward as well? And so open api is open standards are a big part of that, but equally it's understanding the trade offs that you're going to make and clearly communicating those so with data lakes. Um, there was kind of the 1st and 2nd iterations of data lakes, there was the uh, in the early days these capabilities were knew they were based around open source software. Um, a lot of the Hadoop and big data ecosystem, you know, started based on some of these key papers from amazon and google and others taking infrastructure ideas bringing them to scale. We went through a whole evolution of that and the input and the output of that basically let us into the next phase, which I think is the second phase of data leak, which is we have this data are tools are so much better because of that first phase that the investments we made the first time around, we're going to have to pay another investment to make that transformation. And so I've actually, I never want to caution someone not to jump early, but it has to be the right jump and it has to be something that really gives you a competitive advantage. A lot of infrastructure technology is you should make the choices that you make one or two big bets and sometimes people say this, you call it using their innovation tokens. You need to make the bets on big technologies that you operate more effectively at scale. It is somewhat hard to predict that. I certainly say that I've missed quite a few of the exciting transformations in the field just because, um, it wasn't always obvious that it was going to pay off to the degree that um, customers would need. >>So I gotta ask you on the real time applications side of it, that's been a big trend, certainly in cloud. But as you look at hybrid hybrid cloud environments, for instance, streaming data has been a big issue. Uh any updates there from you on your managed service? >>That's right. So one of we have to manage services um that are both closely aligned three managed services that are closely aligned with data in three different ways. And so um one of them is redhead open shift streams for Apache Kafka, which is managed cloud service that focuses on bringing that streaming data and letting you run it across multiple environments. And I think that, you know, we get to the heart of what's the purpose of uh managed services is to reduce operational overhead and to take responsibilities that allow users to focus on the things that actually matter for them. So for us, um managed open shift streams is really about the flow of data between applications in different environments, whether that's from the edge to an on premise data center, whether it's an on premise data center to the cloud. And increasingly these services which were running in the public cloud, increasingly these services have elements that run in the public cloud, but also key elements that run close to where your applications are. And I think that bridge is actually really important for us. That's a key component of hybrid is connecting the different locations and different footprints. So for us the focus is really how do we get data moving to the right place that complements our API management service, which is an add on for open ship dedicated, which means once you've brought the data and you need to expose it back out to other applications in the environment, you can build those applications on open shift, you can leverage the capabilities of open shift api management to expose them more easily, both to end customers or to other applications. And then our third services redhead open shift data science. Um and that is a, an integration that makes it easy for data scientists in a kubernetes environment. On open shift, they easily bring together the data to make, to analyze it and to help route it is appropriate. So those three facets for us are pretty important. They can be used in many different ways, but that focus on the flow of data across these different environments is really a key part of our longer term strategy. >>You know, all the customer checkboxes there you mentioned earlier. I mean I'll just summarize that that you said, you know, obviously value faster application velocity time to value. Those are like the checkboxes, Gardner told analysts check those lower complexity. Oh, we do the heavy lifting, all cloud benefits, so that's all cool. Everyone kind of gets that, everyone's been around cloud knows devops all those things come into play right now. The innovation focuses on operations and day to operations, becoming much more specific. When people say, hey, I've done some lift and shift, I've done some Greenfield born in the cloud now, it's like, whoa, this stuff, I haven't seen this before. As you start scaling. So this brings up that concept and then you add in multi cloud and hybrid cloud, you gotta have a unified experience. So these are the hot areas right this year, I would say, you know, that day to operate has been around for a while, but this idea of unification around environments to be fully distributed for developers is huge. >>How do you >>architect for that? This is the number one question I get. And I tease out when people are kind of talking about their environments that challenges their opportunities, they're really trying to architect, you know, the foundation that building to be um future proof, they don't want to get screwed over when they have, they realize they made a decision, they weren't thinking about day to operation or they didn't think about the unified experience across clouds across environments and services. This is huge. What's your take on this? >>So this is um, this is probably one of the hardest questions I think I could get asked, which is uh looking into the crystal ball, what are the aspects of today's environments that are accidental complexity? That's really just a result of the slow accretion of technologies and we all need to make bets when, when the time is right within the business, um and which parts of it are essential. What are the fundamental hard problems and so on. The accidental complexity side for red hat, it's really about um that consistent environment through open shift bringing capabilities, our connection to open source and making sure that there's an open ecosystem where um community members, users vendors can all work together to um find solutions that work for them because there's not, there's no way to solve for all of computing. It's just impossible. I think that is kind of our that's our development process and that's what helps make that accidental complexity of all that self away over time. But in the essential complexity data is tied the location, data has gravity data. Lakes are a great example of because data has gravity. The more data that you bring together, the bigger the scale the tools you can bring, you can invest in more specialized tools. I've almost do that as a specialization centralization. There's a ton of centralization going on right now at the same time that these new technologies are available to make it easier and easier. Whether that's large scale automation um with conflict management technologies, whether that's kubernetes and deploying it in multiple sites in multiple locations and open shift, bringing consistency so that you can run the apps the same way. But even further than that is concentrating, mhm. More of what would have typically been a specialist problem, something that you build a one off around in your organization to work through the problem. We're really getting to a point where pretty soon now there is a technology or a service for everyone. How do you get the data into that service out? How do you secure it? How do you glue it together? Um I think of, you know, some people might call this um you know, the ultimate integration problem, which is we're going to have all of this stuff and all of these places, what are the core concepts, location, security, placement, topology, latency, where data resides, who's accessing that data, We think of these as kind of the building blocks of where we're going next. So for us trying to make investments in, how do we make kubernetes work better across lots of environments. I have a coupon talk coming up this coupon, it's really exciting for me to talk about where we're going with, you know, the evolution of kubernetes, bringing the different pieces more closely together across multiple environments. But likewise, when we talk about our managed services, we've approached the strategy for managed services as it's not just the service in isolation, it's how it connects to the other pieces. What can we learn in the community, in our services, working with users that benefits that connectivity. So I mentioned the open shift streams connecting up environments, we'd really like to improve how applications connect across disparate environments. That's a fundamental property of if you're going to have data uh in one geographic region and you didn't move services closer to that well, those services I need to know and encode and have that behavior to get closer to where the data is, whether it's one data lake or 10. We gotta have that flexibility in place. And so those obstructions are really, and to >>your point about the building blocks where you've got to factor in those building blocks, because you're gonna need to understand the latency impact, that's going to impact how you're gonna handle the compute piece, that's gonna handle all these things are coming into play. So, again, if you're mindful of the building blocks, just as a cloud concept, um, then you're okay. >>We hear this a lot. Actually, there's real challenges in the, the ecosystem of uh, we see a lot of the problems of I want to help someone automate and improved, but the more balkanize, the more spread out, the more individual solutions are in play, it's harder for someone to bring their technology to bear to help solve the problem. So looking for ways that we can um, you know, grease the skids to build the glue. I think open source works best when it's defining de facto solutions that everybody agrees on that openness and the easy access is a key property that makes de facto standards emerged from open source. What can we do to grow defacto standards around multi cloud and application movement and application interconnect I think is a very, it's already happening and what can we do to accelerate it? That's it. >>Well, I think you bring up a really good point. This is probably a follow up, maybe a clubhouse talk or you guys will do a separate session on this. But I've been riffing on this idea of uh, today's silos, tomorrow's component, right, or module. If most people don't realize that these silos can be problematic if not thought through. So you have to kill the silos to bring in kind of an open police. So if you're open, not closed, you can leverage a monolith. Today's monolithic app or full stack could be tomorrow's building block unless you don't open up. So this is where interesting design question comes in, which is, it's okay to have pre existing stuff if you're open about it. But if you stay siloed, you're gonna get really stuck >>and there's going to be more and more pre existing stuff I think, you know, uh even the data lake for every day to lake, there is a huge problem of how to get data into the data lake or taking existing applications that came from the previous data link. And so there's a, there's a natural evolutionary process where let's focus on the mechanisms that actually move that day to get that data flowing. Um, I think we're still in the early phases of thinking about huge amounts of applications. Microservices or you know, 10 years old in the sense of it being a fairly common industry talking point before that we have service oriented architecture. But the difference now is that we're encouraging and building one developer, one team might run several services. They might use three or four different sas vendors. They might depend on five or 10 or 15 cloud services. Those integration points make them easier. But it's a new opportunity for us to say, well, what are the differences to go back to? The point is you can keep your silos, we just want to have great integration in and out of >>those. Exactly, they don't have to you have to break down the silos. So again, it's a tried and true formula integration, interoperability and abstracting away the complexity with some sort of new software abstraction layer. You bring that to play as long as you can paddle with that, you apply the new building blocks, you're classified. >>It sounds so that's so simple, doesn't it? It does. And you know, of course it'll take us 10 years to get there. And uh, you know, after cloud native will be will be galactic native or something like that. You know, there's always going to be a new uh concept that we need to work in. I think the key concepts we're really going after our everyone is trying to run resilient and reliable services and the clouds give us in the clouds take it away. They give us those opportunities to have some of those building blocks like location of geographic hardware resources, but they will always be data that spread. And again, you still have to apply those principles to the cloud to get the service guarantees that you need. I think there's a completely untapped area for helping software developers and software teams understand the actual availability and guarantees of the underlying environment. It's a property of the services you run with. If you're using a disk in a particular availability zone, that's a property of your application. I think there's a rich area that hasn't been mined yet. Of helping you understand what your effective service level goals which of those can be met. Which cannot, it doesn't make a lot of sense in a single cluster or single machine or a single location world the moment you start to talk about, Well I have my data lake. Well what are the ways my data leg can fail? How do we look at your complex web of interdependencies and say, well clearly if you lose this cloud provider, you're going to lose not just the things that you have running there, but these other dependencies, there's a lot of, there's a lot of next steps that we're just learning what happens when a major cloud goes down for a day or a region of a cloud goes down for a day. You still have to design and work around those >>cases. It's distributed computing. And again, I love the space where galactic cloud, you got SpaceX? Where's Cloud X? I mean, you know, space is the next frontier. You know, you've got all kinds of action happening in space. Great space reference there. Clayton, Great insight. Thanks for coming on. Uh, Clayton Coleman architect at red Hat. Clayton, Thanks for coming on. >>Pretty pleasure. >>Always. Great chat. I'm talking under the hood. What's going on in red hats? New managed cloud service portfolio? Again, the world's getting complex, abstract away. The complexities with software Inter operate integrate. That's the key formula with the cloud building blocks. I'm john ferry with the cube. Thanks for watching. Yeah.