>>Hey, welcome to the live panel. My name is Brett. I am your host, and indeed we are live. In fact, if you're curious about that, if you don't believe us, um, let's just show a little bit of the browser real quick to see. Yup. There you go. We're live. So, all right. So how this is going to work is I'm going to bring in some guests and, uh, in one second, and we're going to basically take your questions on the topic designer of the day, that continuous integration testing. Uh, thank you so much to my guests welcoming into the panel. I've got Carlos, Nico and Mandy. Hello everyone. >>Hello? All right, >>Let's go. Let's go around the room and all pretend we don't know each other and that the internet didn't read below the video who we are. Uh, hi, my name is Brett. I am a Docker captain, which means I'm supposed to know something about Docker. I'm coming from Virginia Beach. I'm streaming here from Virginia Beach, Virginia, and, uh, I make videos on the internet and courses on you to me, Carlos. Hey, >>Hey, what's up? I'm Carlos Nunez. I am a solutions architect, VMware. I do solution things with computers. It's fun. I live in Dallas when I'm moving to Houston in a month, which is where I'm currently streaming. I've been all over the Northeast this whole week. So, um, it's been fun and I'm excited to meet with all of you and talk about CIA and Docker. Sure. >>Yeah. Hey everyone. Uh, Nico, Khobar here. I'm a solution engineer at HashiCorp. Uh, I am streaming to you from, uh, the beautiful Austin, Texas. Uh, ignore, ignore the golden gate bridge here. This is from my old apartment in San Francisco. Uh, just, uh, you know, keeping that, to remember all the good days, um, that that lived at. But, uh, anyway, I work at Patrick Corp and I work on all things, automation, um, and cloud and dev ops. Um, and I'm excited to be here and Mandy, >>Hi. Yeah, Mandy Hubbard. I am streaming from Austin, Texas. I am, uh, currently a DX engineer at ship engine. Um, I've worked in QA and that's kind of where I got my, uh, my Docker experience and, um, uh, moving into DX to try and help developers better understand and use our products and be an advocate for them. >>Nice. Well, thank you all for joining me. Uh, I really appreciate you taking the time out of your busy schedule to be here. And so for those of you in chat, the reason we're doing this live, because it's always harder to do things live. The reason we're here is to answer a question. So we didn't come with a bunch of slides and demos or anything like that. We're here to talk amongst ourselves about ideas and really here for you. So we've, we obviously, this is about easy CII, so we're, we're going to try to keep the conversation around testing and continuous integration and all the things that that entails with containers. But we may, we may go down rabbit holes. We may go veer off and start talking about other things, and that's totally fine if it's in the realm of dev ops and containers and developer and ops workflows, like, Hey, it's, it's kinda game. >>And, uh, these people have a wide variety of expertise. They haven't done just testing, right? We, we live in a world where you all kind of have to wear many hats. So feel free to, um, ask what you think is on the top of your mind. And we'll do our best to answer. It may, might not be the best answer or the correct answer, but we're going to do our best. Um, well, let's get it start off. Uh, let's, let's get a couple of topics to start off with. Uh, th the, the easy CGI was my, one of my three ideas. Cause he's the, one of the things that I'm most excited about is the innovation we're seeing around easier testing, faster testing, automated testing, uh, because as much as we've all been doing this stuff for, you know, 15 years, since 20 years since the sort of Jenkins early days, um, it it's, it seems like it's still really hard and it's still a lot of work. >>So, um, let's go around the room real quick, and everybody can just kind of talk for a minute about like your experience with testing and maybe some of your pain points, like what you don't like about our testing world. Um, and we can talk about some pains, cause I think that will lead us to kind of talk about what, what are the things we're seeing now that might be better, uh, ideas about how to do this. I know for me, uh, testing, obviously there's the code part, but just getting it automated, but mostly getting it in the hands of developers so that they can control their own testing. And don't have to go talk to a person to run that test again, or the mysterious Jenkins platform somewhere. I keep mentioning Jenkins cause it's, it is still the dominant player out there. Um, so for me, I'm, I'm, I, I don't like it when I'm walking into a room and there's, there's only one or two people that know how the testing works or know how to make the new tests go into the testing platform and stuff like that. So I'm always trying to free those things so that any of the developers are enabled and empowered to do that stuff. So someone else, Carlos, anybody, um, >>Oh, I have a lot of opinions on that. Having been a QA engineer for most of my career. Um, the shift that we're saying is everyone is dev ops and everyone is QA. Th the issue I see is no one asked developers if they wanted to be QA. Um, and so being the former QA on the team, when there's a problem, even though I'm a developer and we're all running QA, they always tend to come to the one of the former QA engineers. And they're not really owning that responsibility and, um, and digging in. So that's kind of what I'm saying is that we're all expected to test now. And some people, well, some people don't know how it's, uh, for me it was kind of an intuitive skill. It just kind of fit with my personality, but not knowing what to look for, not knowing what to automate, not even understanding how your API end points are used by your front end to know what to test when a change is made. It's really overwhelming for developers. And, um, we're going to need to streamline that and, and hold their hands a little bit until they get their feet wet with also being QA. >>Right. Right. So, um, uh, Carlos, >>Yeah, uh, testing is like, Tesla is one of my favorite subjects to talk about when I'm baring with developers. And a lot of it is because of what Mandy said, right? Like a lot of developers now who used to write a test and say, Hey, QA, go. Um, I wrote my unit tests. Now write the rest of the test. Essentially. Now developers are expected to be able to understand how testing, uh, testing methodologies work, um, in their local environments, right? Like they're supposed to understand how to write an integration tasks federate into and tasks, a component test. And of course, how to write unit tests that aren't just, you know, assert true is true, right? Like more comprehensive, more comprehensive, um, more high touch unit tests, which include things like mocking and stubbing and spine and all that stuff. And, you know, it's not so much getting those tests. Well, I've had a lot of challenges with developers getting those tests to run in Docker because of usually because of dependency hell, but, um, getting developers to understand how to write tests that matter and mean something. Um, it's, it's, it can be difficult, but it's also where I find a lot of the enjoyment of my work comes into play. So yeah. I mean, that's the difficulty I've seen around testing. Um, big subject though. Lots to talk about there. >>Yeah. We've got, we've already got so many questions coming in. You already got an hour's worth of stuff. So, uh, Nico 81st thoughts on that? >>Yeah, I think I definitely agree with, with other folks here on the panel, I think from a, um, the shift from a skillset perspective that's needed to adopt the new technologies, but I think from even from, uh, aside from the organizational, um, and kind of key responsibilities that, that the new developers have to kinda adapt to and, and kind of inherit now, um, there's also from a technical perspective as there's, you know, um, more developers are owning the full stack, including the infrastructure piece. So that adds a lot more to the plate in Tim's oaf, also testing that component that they were not even, uh, responsible for before. Um, and, um, also the second challenge that, you know, I'm seeing is that on, you know, the long list of added, um, uh, tooling and, you know, there's new tool every other day. Um, and, um, that kind of requires more customization to the testing, uh, that each individual team, um, any individual developer Y by extension has to learn. Uh, so the customization, uh, as well as the, kind of the scope that had, uh, you know, now in conferences, the infrastructure piece, um, uh, both of act to the, to the challenges that we're seeing right now for, um, for CGI and overall testing, um, uh, the developers are saying, uh, in, in the market today. >>Yeah. We've got a lot of questions, um, about all the, all the different parts of this. So, uh, let me just go straight to them. Cause that's why we're here is for the people, uh, a lot of people asking about your favorite tools and in one of this is one of the challenges with integration, right? Is, um, there is no, there are dominant players, but there, there is such a variety. I mean, every one of my customers seems like they're using a different workflow and a different set of tools. So, and Hey, we're all here to just talk about what we're, what we're using, uh, you know, whether your favorite tools. So like a lot of the repeated questions are, what are your favorite tools? Like if you could create it from scratch, uh, what would you use? Pierre's asking, you know, GitHub actions sounds like they're a fan of GitHub actions, uh, w you know, mentioning, pushing the ECR and Docker hub and, uh, using vs code pipeline, I guess there may be talking about Azure pipelines. Um, what, what's your preferred way? So, does anyone have any, uh, thoughts on that anyone want to throw out there? Their preferred pipeline of tooling? >>Well, I have to throw out mine. I might as Jenkins, um, like kind of a honorary cloud be at this point, having spoken a couple of times there, um, all of the plugins just make the functionality. I don't love the UI, but I love that it's been around so long. It has so much community support, and there are so many plugins so that if you want to do something, you don't have to write the code it's already been tested. Um, unfortunately I haven't been able to use Jenkins in, uh, since I joined ship engine, we, most of our, um, our, our monolithic core application is, is team city. It's a dotnet application and TeamCity plays really well with.net. Um, didn't love it, uh, Ms. Jenkins. And I'm just, we're just starting some new initiatives that are using GitHub actions, and I'm really excited to learn, to learn those. I think they have a lot of the same functionality that you're looking for, but, um, much more simplified in is right there and get hubs. So, um, the integration is a lot more seamless, but I do have to go on record that my favorite CICT tools Jenkins. >>All right. You heard it here first people. All right. Anyone else? You're muted? I'm muted. Carlin says muted. Oh, Carla says, guest has muted themselves to Carlos. You got to unmute. >>Yes. I did mute myself because I was typing a lot, trying to, you know, try to answer stuff in the chat. And there's a lot of really dark stuff in there. That's okay. Two more times today. So yeah, it's fine. Yeah, no problem. So totally. And it's the best way to start a play more. So I'm just going to go ahead and light it up. Um, for enterprise environments, I actually am a huge fan of Jenkins. Um, it's a tool that people really understand. Um, it has stood the test of time, right? I mean, people were using Hudson, but 15 years ago, maybe longer. And, you know, the way it works, hasn't really changed very much. I mean, Jenkins X is a little different, but, um, the UI and the way it works internally is pretty familiar to a lot of enterprise environments, which is great. >>And also in me, the plugin ecosystem is amazing. There's so many plugins for everything, and you can make your own if you know, Java groovy. I'm sure there's a perfect Kotlin in there, but I haven't tried myself, but it's really great. It's also really easy to write, um, CIS code, which is something I'm a big fan of. So Jenkins files have been, have worked really well for me. I, I know that I can get a little bit more complex as you start to build your own models and such, but, you know, for enterprise enterprise CIO CD, if you want, especially if you want to roll your own or own it yourself, um, Jenkins is the bellwether and for very good reason now for my personal projects. And I see a lot on the chat here, I think y'all, y'all been agreed with me get hub actions 100%, my favorite tool right now. >>Um, I love GitHub actions. It's, it's customizable, it's modular. There's a lot of plugins already. I started using getting that back maybe a week after when GA and there was no documentation or anything. And I still, it was still my favorite CIA tool even then. Um, and you know, the API is really great. There's a lot to love about GitHub actions and, um, and I, and I use it as much as I can from my personal project. So I still have a soft spot for Travis CAI. Um, you know, they got acquired and they're a little different now trying to see, I, I can't, I can't let it go. I just love it. But, um, yeah, I mean, when it comes to Seattle, those are my tools. So light me up in the comments I will respond. Yeah. >>I mean, I, I feel with you on the Travis, the, I think, cause I think that was my first time experiencing, you know, early days get hub open source and like a free CIA tool that I could describe. I think it was the ammo back then. I don't actually remember, but yeah, it was kind of an exciting time from my experience. There was like, oh, this is, this is just there as a service. And I could just use it. It doesn't, it's like get hub it's free from my open source stuff. And so it does have a soft spot in my heart too. So yeah. >>All right. We've got questions around, um, cam, so I'm going to ask some questions. We don't have to have these answers because sometimes they're going to be specific, but I want to call them out because people in chat may have missed that question. And there's probably, you know, that we have smart people in chat too. So there's probably someone that knows the answer to these things. If, if it's not us, um, they're asking about building Docker images in Kubernetes, which to me is always a sore spot because it's Kubernetes does not build images by default. It's not meant for that out of the gate. And, uh, what is the best way to do this without having to use privileged containers, which privileged containers just implying that yeah, you, you, it probably has more privileges than by default as a container in Kubernetes. And that is a hard thing because, uh, I don't, I think Docker doesn't lie to do that out of the gate. So I don't know if anyone has an immediate answer to that. That's a pretty technical one, but if you, if you know the answer to that in chat, call it out. >>Um, >>I had done this, uh, but I'm pretty sure I had to use a privileged, um, container and install the Docker Damon on the Kubernetes cluster. And I CA I can't give you a better solution. Um, I've done the same. So, >>Yeah, uh, Chavonne asks, um, back to the Jenkins thing, what's the easiest way to integrate Docker into a Jenkins CICB pipeline. And that's one of the challenges I find with Jenkins because I don't claim to be the expert on Jenkins. Is there are so many plugins because of this, of this such a huge ecosystem. Um, when you go searching for Docker, there's a lot that comes back, right. So I, I don't actually have a preferred way because every team I find uses it differently. Um, I don't know, is there a, do you know if there's a Jenkins preferred, a default plugin? I don't even know for Docker. Oh, go ahead. Yeah. Sorry for Docker. And jacon sorry, Docker plugins for Jenkins. Uh, as someone's asking like the preferred or easy way to do that. Um, and I don't, I don't know the back into Jenkins that well, so, >>Well, th the new, the new way that they're doing, uh, Docker builds with the pipeline, which is more declarative versus the groovy. It's really simple, and their documentation is really good. They, um, they make it really easy to say, run this in this image. So you can pull down, you know, public images and add your own layers. Um, so I don't know the name of that plugin, uh, but I can certainly take a minute after this session and going and get that. Um, but if you really are overwhelmed by the plugins, you can just write your, you know, your shell command in Jenkins. You could just by, you know, doing everything in bash, calling the Docker, um, Damon directly, and then getting it working just to see that end to end, and then start browsing for plugins to see if you even want to use those. >>The plugins will allow more integration from end to end. Some of the things that you input might be available later on in the process for having to manage that yourself. But, you know, you don't have to use any of the plugins. You can literally just, you know, do a block where you write your shell command and get it working, and then decide if, for plugins for you. Um, I think it's always under important to understand what is going on under the hood before you, before you adopt the magic of a plugin, because, um, once you have a problem, if you're, if it's all a lockbox to you, it's going to be more difficult to troubleshoot. It's kind of like learning, get command line versus like get cracking or something. Once, once you get in a bind, if you don't understand the underlying steps, it's really hard to get yourself out of a bind, versus if you understand what the plugin or the app is doing, then, um, you can get out of situations a lot easier. That's a good place. That's, that's where I'd start. >>Yeah. Thank you. Um, Camden asks better to build test environment images, every commit in CII. So this is like one of those opinions of we're all gonna have some different, uh, or build on build images on every commit, leveraging the cash, or build them once outside the test pile pipeline. Um, what say you people? >>Uh, well, I I've seen both and generally speaking, my preference is, um, I guess the ant, the it's a consultant answer, right? I think it depends on what you're trying to do, right. So if you have a lot of small changes that are being made and you're creating images for each of those commits, you're going to have a lot of images in your, in your registry, right? And on top of that, if you're building those images, uh, through CAI frequently, if you're using Docker hub or something like that, you might run into rate limiting issues because of Docker's new rate, limiting, uh, rate limits that they put in place. Um, but that might be beneficial if the, if being able to roll back between those small changes while you're testing is important to you. Uh, however, if all you care about is being able to use Docker images, um, or being able to correlate versions to your Docker images, or if you're the type of team that doesn't even use him, uh, does he even use, uh, virgins in your image tags? Then I would think that that might be a little, much you might want to just have in your CIO. You might want to have a stage that builds your Docker images and Docker image and pushes it into your registry, being done first particular branches instead of having to be done on every commit regardless of branch. But again, it really depends on the team. It really depends on what you're building. It really depends on your workflow. It can depend on a number of things like a curse sometimes too. Yeah. Yeah. >>Once had two points here, you know, I've seen, you know, the pattern has been at every, with every, uh, uh, commit, assuming that you have the right set of tests that would kind of, uh, you would benefit from actually seeing, um, the, the, the, the testing workflow go through and can detect any issue within, within the build or whatever you're trying to test against. But if you're just a building without the appropriate set of tests, then you're just basically consuming almond, adding time, as well as all the, the image, uh, stories associated with it without treaty reaping the benefit of, of, of this pattern. Uh, and the second point is, again, I think if you're, if you're going to end up doing a per commit, uh, definitely recommend having some type of, uh, uh, image purging, um, uh, and, and, and garbage collection process to ensure that you're not just wasting, um, all the stories needed and also, um, uh, optimizing your, your bill process, because that will end up being the most time-consuming, um, um, you know, within, within your pipeline. So this is my 2 cents on this. >>Yeah, that's good stuff. I mean, those are both of those are conversations that could lead us into the rabbit hole for the rest of the day on storage management, uh, you know, CP CPU minutes for, uh, you know, your build stuff. I mean, if you're in any size team, more than one or two people, you immediately run into headaches with cost of CIA, because we have now the problem of tools, right? We have so many tools. We can have the CIS system burning CPU cycles all day, every day, if we really wanted to. And so you re very quickly, I think, especially if you're on every commit on every branch, like that gets you into a world of cost mitigation, and you probably are going to have to settle somewhere in the middle on, uh, between the budget, people that are saying you're spending way too much money on the CII platform, uh, because of all these CPU cycles, and then the developers who would love to have everything now, you know, as fast as possible and the biggest, biggest CPU's, and the biggest servers, and have the bills, because the bills can never go fast enough, right. >>There's no end to optimizing your build workflow. Um, we have another question on that. This is another topic that we'll all probably have different takes on is, uh, basically, uh, version tags, right? So on images, we, we have a very established workflow in get for how we make commits. We have commit shots. We have, uh, you know, we know get tags and there's all these things there. And then we go into images and it's just this whole new world that's opened up. Like there's no real consensus. Um, so what, what are your thoughts on the strategy for teams in their image tag? Again, another, another culture thing. Um, commander, >>I mean, I'm a fan of silver when we have no other option. Um, it's just clean and I like the timestamp, you know, exactly when it was built. Um, I don't really see any reason to use another, uh, there's just normal, incremental, um, you know, numbering, but I love the fact that you can pull any tag and know exactly when it was created. So I'm a big fan of bar, if you can make that work for your organization. >>Yep. People are mentioned that in chat, >>So I like as well. Uh, I'm a big fan of it. I think it's easy to be able to just be as easy to be able to signify what a major changes versus a minor change versus just a hot fix or, you know, some or some kind of a bad fix. The problem that I've found with having teams adopt San Bernardo becomes answering these questions and being able to really define what is a major change, what is a minor change? What is a patch, right? And this becomes a bit of an overhead or not so much of an overhead, but, uh, uh, uh, a large concern for teams who have never done versioning before, or they never been responsible for their own versioning. Um, in fact, you know, I'm running into that right now, uh, with, with a client that I'm working with, where a lot, I'm working with a lot of teams, helping them move their applications from a legacy production environment into a new one. >>And in doing so, uh, versioning comes up because Docker images, uh, have tags and usually the tax correlate to versions, but some teams over there, some teams that I'm working with are only maintaining a script and others are maintaining a fully fledged JAK, three tier application, you know, with lots of dependencies. So telling the script, telling the team that maintains a script, Hey, you know, you should use somber and you should start thinking about, you know, what's major, what's my number what's patch. That might be a lot for them. And for someone or a team like that, I might just suggest using commit shots as your versions until you figure that out, or maybe using, um, dates as your version, but for the more for the team, with the larger application, they probably already know the answers to those questions. In which case they're either already using Sember or they, um, or they may be using some other version of the strategy and might be in December, might suit them better. So, um, you're going to hear me say, it depends a lot, and I'm just going to say here, it depends. Cause it really does. Carlos. >>I think you hit on something interesting beyond just how to version, but, um, when to consider it a major release and who makes those decisions, and if you leave it to engineers to version, you're kind of pushing business decisions down the pipe. Um, I think when it's a minor or a major should be a business decision and someone else needs to make that call someone closer to the business should be making that call as to when we want to call it major. >>That's a really good point. And I add some, I actually agree. Um, I absolutely agree with that. And again, it really depends on the team that on the team and the scope of it, it depends on the scope that they're maintaining, right? And so it's a business application. Of course, you're going to have a product manager and you're going to have, you're going to have a product manager who's going to want to make that call because that version is going to be out in marketing. People are going to use it. They're going to refer to and support calls. They're going to need to make those decisions. Sember again, works really, really well for that. Um, but for a team that's maintaining the scripts, you know, I don't know, having them say, okay, you must tell me what a major version is. It's >>A lot, but >>If they want it to use some birds great too, which is why I think going back to what you originally said, Sember in the absence of other options. I think that's a good strategy. >>Yeah. There's a, there's a, um, catching up on chat. I'm not sure if I'm ever going to catch up, but there's a lot of people commenting on their favorite CII systems and it's, and it, it just goes to show for the, the testing and deployment community. Like how many tools there are out there, how many tools there are to support the tools that you're using. Like, uh, it can be a crazy wilderness. And I think that's, that's part of the art of it, uh, is that these things are allowing us to build our workflows to the team's culture. Um, and, uh, but I do think that, you know, getting into like maybe what we hope to be at what's next is I do hope that we get to, to try to figure out some of these harder problems of consistency. Uh, one of the things that led me to Docker at the beginning to begin with was the fact that it wa it created a consistent packaging solution for me to get my code, you know, off of, off of my site of my local system, really, and into the server. >>And that whole workflow would at least the thing that I was making at each step was going to be the same thing used. Right. And that, that was huge. Uh, it was also, it also took us a long time to get there. Right. We all had to, like Docker was one of those ones that decade kind of ideas of let's solidify the, enter, get the consensus of the community around this idea. And we, and it's not perfect. Uh, you know, the Docker Docker file is not the most perfect way to describe how to make your app, but it is there and we're all using it. And now I'm looking for that next piece, right. Then hopefully the next step in that, um, that where we can all arrive at a consensus so that once you hop teams, you know, okay. We all knew Docker. We now, now we're all starting to get to know the manifests, but then there's this big gap in the middle where it's like, it might be one of a dozen things. Um, you know, so >>Yeah, yeah. To that, to that, Brett, um, you know, uh, just maybe more of a shameless plug here and wanting to kind of talk about one of the things that I'm on. So excited, but I work, I work at Tasha Corp. I don't know anyone, or I don't know if many people have heard of, um, you know, we tend to focus a lot on workflows versus technologies, right. Because, you know, as you can see, even just looking at the chat, there's, you know, ton of opinions on the different tooling, right. And, uh, imagine having, you know, I'm working with clients that have 10,000 developers. So imagine taking the folks in the chat and being partnered with one organization or one company and having to make decisions on how to build software. Um, but there's no way you can conversion one or, or one way or one tool, uh, and that's where we're facing in the industry. >>So one of the things that, uh, I'm pretty excited about, and I don't know if it's getting as much traction as you know, we've been focused on it. This is way point, which is a project, an open source project. I believe we got at least, uh, last year, um, which is, it's more of, uh, it's, it is aim to address that really, uh, uh, Brad set on, you know, to come to tool to, uh, make it extremely easy and simple. And, you know, to describe how you want to build, uh, deploy or release your application, uh, in, in a consistent way, regardless of the tools. So similar to how you can think of Terraform and having that pluggability to say Terraform apply or plan against any cloud infrastructure, uh, without really having to know exactly the details of how to do it, uh, this is what wave one is doing. Um, and it can be applied with, you know, for the CIA, uh, framework. So, you know, task plugability into, uh, you know, circle CEI tests to Docker helm, uh, Kubernetes. So that's the, you know, it's, it's a hard problem to solve, but, um, I'm hopeful that that's the path that we're, you know, we'll, we'll eventually get to. So, um, hope, you know, you can, you can, uh, see some of the, you know, information, data on it, on, on HashiCorp site, but I mean, I'm personally excited about it. >>Yeah. Uh I'm to gonna have to check that out. And, um, I told you on my live show, man, we'll talk about it, but talk about it for a whole hour. Uh, so there's another question here around, uh, this, this is actually a little bit more detailed, but it is one that I think a lot of people deal with and I deal with a lot too, is essentially the question is from Cameron, uh, D essentially, do you use compose in your CIO or not Docker compose? Uh, because yes I do. Yeah. Cause it, it, it, it solves so many problems am and not every CGI can, I don't know, there's some problems with a CIO is trying to do it for me. So there are pros and cons and I feel like I'm still on the fence about it because I use it all the time, but also it's not perfect. It's not always meant for CIA. And CIA sometimes tries to do things for you, like starting things up before you start other parts and having that whole order, uh, ordering problem of things anyway. W thoughts and when have thoughts. >>Yes. I love compose. It's one of my favorite tools of all time. Um, and the reason why it's, because what I often find I'm working with teams trying to actually let me walk that back, because Jack on the chat asked a really interesting question about what, what, what the hardest thing about CIS for a lot of teams. And in my experience, the hardest thing is getting teams to build an app that is the same app as what's built in production. A lot of CGI does things that are totally different than what you would do in your local, in your local dev. And as a result of that, you get, you got this application that either doesn't work locally, or it does work, but it's a completely different animal than what you would get in production. Right? So what I've found in trying to get teams to bridge that gap by basically taking their CGI, shifting the CII left, I hate the shift left turn, but I'll use it. >>I'm shifting the CIO left to your local development is trying to say, okay, how do we build an app? How do we, how do we build mot dependencies of that app so that we can build so that we can test our app? How do we run tests, right? How do we build, how do we get test data? And what I found is that trying to get teams to do all this in Docker, which is normally a first for a lot of teams that I'm working with, trying to get them all to do all of this. And Docker means you're running Docker, build a lot running Docker, run a lot. You're running Docker, RM a lot. You ran a lot of Docker, disparate Docker commands. And then on top of that, trying to bridge all of those containers together into a single network can be challenging without compose. >>So I like using a, to be able to really easily categorize and compartmentalize a lot of the things that are going to be done in CII, like building a Docker image, running tests, which is you're, you're going to do it in CII anyway. So running tests, building the image, pushing it to the registry. Well, I wouldn't say pushing it to the registry, but doing all the things that you would do in local dev, but in the same network that you might have a mock database or a mock S3 instance or some of something else. Um, so it's just easy to take all those Docker compose commands and move them into your Yammel file using the hub actions or your dankest Bob using Jenkins, or what have you. Right. It's really, it's really portable that way, but it doesn't work for every team. You know, for example, if you're just a team that, you know, going back to my script example, if it's a really simple script that does one thing on a somewhat routine basis, then that might be a lot of overhead. Um, in that case, you know, you can get away with just Docker commands. It's not a big deal, but the way I looked at it is if I'm, if I'm building, if I build something that's similar to a make bile or rate file, or what have you, then I'm probably gonna want to use Docker compose. If I'm working with Docker, that's, that's a philosophy of values, right? >>So I'm also a fan of Docker compose. And, um, you know, to your point, Carlos, the whole, I mean, I'm also a fan of shifting CEI lift and testing lift, but if you put all that logic in your CTI, um, it changes the L the local development experience from the CGI experience. Versus if you put everything in a compose file so that what you build locally is the same as what you build in CGI. Um, you're going to have a better experience because you're going to be testing something more, that's closer to what you're going to be releasing. And it's also very easy to look at a compose file and kind of, um, understand what the dependencies are and what's happening is very readable. And once you move that stuff to CGI, I think a lot of developers, you know, they're going to be intimidated by the CGI, um, whatever the scripting language is, it's going to be something they're going to have to wrap their head around. >>Um, but they're not gonna be able to use it locally. You're going to have to have another local solution. So I love the idea of a composed file use locally, um, especially if he can Mount the local workspace so that they can do real time development and see their changes in the exact same way as it's going to be built and tested in CGI. It gives developers a high level of confidence. And then, you know, you're less likely to have issues because of discrepancies between how it was built in your local test environment versus how it's built in NCI. And so Docker compose really lets you do all of that in a way that makes your solution more portable, portable between local dev and CGI and reduces the number of CGI cycles to get, you know, the test, the test data that you need. So that's why I like it for really, for local dev. >>It'll be interesting. Um, I don't know if you all were able to see the keynote, but there was a, there was a little bit, not a whole lot, but a little bit talk of the Docker, compose V two, which has now built into the Docker command line. And so now we're shifting from the Python built compose, which was a separate package. You could that one of the challenges was getting it into your CA solution because if you don't have PIP and you got down on the binary and the binary wasn't available for every platform and, uh, it was a PI installer. It gets a little nerdy into how that works, but, uh, and the team is now getting, be able to get unified with it. Now that it's in Golang and it's, and it's plugged right into the Docker command line, it hopefully will be easier to distribute, easier to, to use. >>And you won't have to necessarily have dependencies inside of where you're running it because there'll be a statically compiled binary. Um, so I've been playing with that, uh, this year. And so like training myself to do Docker going from Docker dash compose to Docker space, compose. It is a thing I I'm almost to the point of having to write a shell replacement. Yeah. Alias that thing. Um, but, um, I'm excited to see what that's going, cause there's already new features in it. And it, these built kit by default, like there's all these things. And I, I love build kit. We could make a whole session on build kit. Um, in fact there's actually, um, maybe going on right now, or right around this time, there is a session on, uh, from Solomon hikes, the seat, uh, co-founder of Docker, former CTO, uh, on build kit using, uh, using some other tool on top of build kit or whatever. >>So that, that would be interesting for those of you that are not watching that one. Cause you're here, uh, to do a check that one out later. Um, all right. So another good question was caching. So another one, another area where there is no wrong answers probably, and everyone has a different story. So the question is, what are your thoughts on CII build caching? There's often a debate between security. This is from Quentin. Thank you for this great question. There's often a debate between security reproducibility and build speeds. I haven't found a good answer so far. I will just throw my hat in the ring and say that the more times you want to build, like if you're trying to build every commit or every commit, if you're building many times a day, the more caching you need. So like the more times you're building, the more caching you're gonna likely want. And in most cases caching doesn't bite you in the butt, but that could be, yeah, we, can we get the bit about that? So, yeah. Yeah. >>I'm going to quote Carlos again and say, it depends on, on, you know, how you're talking, you know, what you're trying to build and I'm quoting your colors. Um, yeah, it's, it's got, it's gonna depend because, you know, there are some instances where you definitely want to use, you know, depends on the frequency that you're building and how you're building. Um, it's you would want to actually take advantage of cashing functionalities, um, for the build, uh, itself. Um, but if, um, you know, as you mentioned, there could be some instances where you would want to disable, um, any caching because you actually want to either pull a new packages or, um, you know, there could be some security, um, uh, disadvantages related to security aspects that would, you know, you know, using a cache version of, uh, image layer, for example, could be a problem. And you, you know, if you have a fleet of build, uh, engines, you don't have a good grasp of where they're being cashed. We would have to, um, disable caching in that, in that, um, in those instances. So it, it would depend. >>Yeah, it's, it's funny you have that problem on both sides of cashing. Like there are things that, especially in Docker world, they will cash automatically. And, and then, and then you maybe don't realize that some of that caching could be bad. It's, it's actually using old, uh, old assets, old artifacts, and then there's times where you would expect it to cash, that it doesn't cash. And then you have to do something extra to enable that caching, especially when you're dealing with that cluster of, of CIS servers. Right. And the cloud, the whole clustering problem with caching is even more complex, but yeah, >>But that's, that's when, >>Uh, you know, ever since I asked you to start using build kits and able to build kit, you know, between it's it's it's reader of Boston in, in detecting word, you know, where in, in the bill process needs to cash, as well as, uh, the, the, um, you know, the process. I don't think I've seen any other, uh, approach there that comes close to how efficient, uh, that process can become how much time it can actually save. Uh, but again, I think, I think that's, for me that had been my default approach, unless I actually need something that I would intentionally to disable caching for that purpose, but the benefits, at least for me, the benefits of, um, how bill kit actually been processing my bills, um, from the builds as well as, you know, using the cash up until, you know, how it detects the, the difference in, in, in the assets within the Docker file had been, um, you know, uh, pretty, you know, outweigh the disadvantages that it brings in. So it, you know, take it each case by case. And based on that, determine if you want to use it, but definitely recommend those enabling >>In the absence of a reason not to, um, I definitely think that it's a good approach in terms of speed. Um, yeah, I say you cash until you have a good reason not to personally >>Catch by default. There you go. I think you catch by default. Yeah. Yeah. And, uh, the trick is, well, one, it's not always enabled by default, especially when you're talking about cross server. So that's a, that's a complexity for your SIS admins, or if you're on the cloud, you know, it's usually just an option. Um, I think it also is this, this veers into a little bit of, uh, the more you cash the in a lot of cases with Docker, like the, from like, if you're from images and checked every single time, if you're not pinning every single thing, if you're not painting your app version, you're at your MPN versions to the exact lock file definition. Like there's a lot of these things where I'm I get, I get sort of, I get very grouchy with teams that sort of let it, just let it all be like, yeah, we'll just build two images and they're totally going to have different dependencies because someone happened to update that thing and after whatever or MPM or, or, and so I get grouchy about that, cause I want to lock it all down, but I also know that that's going to create administrative burden. >>Like the team is now going to have to manage versions in a very much more granular way. Like, do we need to version two? Do we need to care about curl? You know, all that stuff. Um, so that's, that's kind of tricky, but when you get to, when you get to certain version problems, uh, sorry, uh, cashing problems, you, you, you don't want those set those caches to happen because it, if you're from image changes and you're not constantly checking for a new image, and if you're not pinning that V that version, then now you, you don't know whether you're getting the latest version of Davion or whatever. Um, so I think that there's, there's an art form to the more you pen, the less you have, the less, you have to be worried about things changing, but the more you pen, the, uh, all your versions of everything all the way down the stack, the more administrative stuff, because you're gonna have to manually change every one of those. >>So I think it's a balancing act for teams. And as you mature, I to find teams, they tend to pin more until they get to a point of being more comfortable with their testing. So the other side of this argument is if you trust your testing, then you, and you have better testing to me, the less likely to the subtle little differences in versions have to be penned because you can get away with those minor or patch level version changes. If you're thoroughly testing your app, because you're trusting your testing. And this gets us into a whole nother rant, but, uh, yeah, but talking >>About penny versions, if you've got a lot of dependencies isn't that when you would want to use the cash the most and not have to rebuild all those layers. Yeah. >>But if you're not, but if you're not painting to the exact patch version and you are caching, then you're not technically getting the latest versions because it's not checking for all the time. It's a weird, there's a lot of this subtle nuance that people don't realize until it's a problem. And that's part of the, the tricky part of allow this stuff, is it, sometimes the Docker can be almost so much magic out of the box that you, you, you get this all and it all works. And then day two happens and you built it a second time and you've got a new version of open SSL in there and suddenly it doesn't work. Um, so anyway, uh, that was a great question. I've done the question on this, on, uh, from heavy. What do you put, where do you put testing in your pipeline? Like, so testing the code cause there's lots of types of testing, uh, because this pipeline gets longer and longer and Docker building images as part of it. And so he says, um, before staging or after staging, but before production, where do you put it? >>Oh man. Okay. So, um, my, my main thought on this is, and of course this is kind of religious flame bait, so sure. You know, people are going to go into the compensation wrong. Carlos, the boy is how I like to think about it. So pretty much in every stage or every environment that you're going to be deploying your app into, or that your application is going to touch. My idea is that there should be a build of a Docker image that has all your applications coded in, along with its dependencies, there's testing that tests your application, and then there's a deployment that happens into whatever infrastructure there is. Right. So the testing, they can get tricky though. And the type of testing you do, I think depends on the environment that you're in. So if you're, let's say for example, your team and you have, you have a main branch and then you have feature branches that merged into the main branch. >>You don't have like a pre-production branch or anything like that. So in those feature branches, whenever I'm doing CGI that way, I know when I freak, when I cut my poll request, that I'm going to merge into main and everything's going to work in my feature branches, I'm going to want to probably just run unit tests and maybe some component tests, which really, which are just, you know, testing that your app can talk to another component or another part, another dependency, like maybe a database doing tests like that, that don't take a lot of time that are fascinating and right. A lot of would be done at the beach branch level and in my opinion, but when you're going to merge that beach branch into main, as part of a release in that activity, you're going to want to be able to do an integration tasks, to make sure that your app can actually talk to all the other dependencies that it talked to. >>You're going to want to do an end to end test or a smoke test, just to make sure that, you know, someone that actually touches the application, if it's like a website can actually use the website as intended and it meets the business cases and all that, and you might even have testing like performance testing, low performance load testing, or security testing, compliance testing that would want to happen in my opinion, when you're about to go into production with a release, because those are gonna take a long time. Those are very expensive. You're going to have to cut new infrastructure, run those tests, and it can become quite arduous. And you're not going to want to run those all the time. You'll have the resources, uh, builds will be slower. Uh, release will be slower. It will just become a mess. So I would want to save those for when I'm about to go into production. Instead of doing those every time I make a commit or every time I'm merging a feature ranch into a non main branch, that's the way I look at it, but everything does a different, um, there's other philosophies around it. Yeah. >>Well, I don't disagree with your build test deploy. I think if you're going to deploy the code, it needs to be tested. Um, at some level, I mean less the same. You've got, I hate the term smoke tests, cause it gives a false sense of security, but you have some mental minimum minimal amount of tests. And I would expect the developer on the feature branch to add new tests that tested that feature. And that would be part of the PR why those tests would need to pass before you can merge it, merge it to master. So I agree that there are tests that you, you want to run at different stages, but the earlier you can run the test before going to production. Um, the fewer issues you have, the easier it is to troubleshoot it. And I kind of agree with what you said, Carlos, about the longer running tests like performance tests and things like that, waiting to the end. >>The only problem is when you wait until the end to run those performance tests, you kind of end up deploying with whatever performance you have. It's, it's almost just an information gathering. So if you don't run your performance test early on, um, and I don't want to go down a rabbit hole, but performance tests can be really useless if you don't have a goal where it's just information gap, uh, this is, this is the performance. Well, what did you expect it to be? Is it good? Is it bad? They can get really nebulous. So if performance is really important, um, you you're gonna need to come up with some expectations, preferably, you know, set up the business level, like what our SLA is, what our response times and have something to shoot for. And then before you're getting to production. If you have targets, you can test before staging and you can tweak the code before staging and move that performance initiative. Sorry, Carlos, a little to the left. Um, but if you don't have a performance targets, then it's just a check box. So those are my thoughts. I like to test before every deployment. Right? >>Yeah. And you know what, I'm glad that you, I'm glad that you brought, I'm glad that you brought up Escalades and performance because, and you know, the definition of performance says to me, because one of the things that I've seen when I work with teams is that oftentimes another team runs a P and L tests and they ended, and the development team doesn't really have too much insight into what's going on there. And usually when I go to the performance team and say, Hey, how do you run your performance test? It's usually just a generic solution for every single application that they support, which may or may not be applicable to the application team that I'm working with specifically. So I think it's a good, I'm not going to dig into it. I'm not going to dig into the rabbit hole SRE, but it is a good bridge into SRE when you start trying to define what does reliability mean, right? >>Because the reason why you test performance, it's test reliability to make sure that when you cut that release, that customers would go to your site or use your application. Aren't going to see regressions in performance and are not going to either go to another website or, you know, lodge in SLA violation or something like that. Um, it does, it does bridge really well with defining reliability and what SRE means. And when you have, when you start talking about that, that's when you started talking about how often do I run? How often do I test my reliability, the reliability of my application, right? Like, do I have nightly tasks in CGI that ensure that my main branch or, you know, some important branch I does not mean is meeting SLA is meeting SLR. So service level objectives, um, or, you know, do I run tasks that ensure that my SLA is being met in production? >>Like whenever, like do I use, do I do things like game days where I test, Hey, if I turn something off or, you know, if I deploy this small broken code to production and like what happens to my performance? What happens to my security and compliance? Um, you can, that you can go really deep into and take creating, um, into creating really robust tests that cover a lot of different domains. But I liked just using build test deploy is the overall answer to that because I find that you're going to have to build your application first. You're going to have to test it out there and build it, and then you're going to want to deploy it after you test it. And that order generally ensures that you're releasing software. That works. >>Right. Right. Um, I was going to ask one last question. Um, it's going to have to be like a sentence answer though, for each one of you. Uh, this is, uh, do you lint? And if you lint, do you lent all the things, if you do, do you fail the linters during your testing? Yes or no? I think it's going to depend on the culture. I really do. Sorry about it. If we >>Have a, you know, a hook, uh, you know, on the get commit, then theoretically the developer can't get code there without running Melinta anyway, >>So, right, right. True. Anyone else? Anyone thoughts on that? Linting >>Nice. I saw an additional question online thing. And in the chat, if you would introduce it in a multi-stage build, um, you know, I was wondering also what others think about that, like typically I've seen, you know, with multi-stage it's the most common use case is just to produce the final, like to minimize the, the, the, the, the, the image size and produce a final, you know, thin, uh, layout or thin, uh, image. Uh, so if it's not for that, like, I, I don't, I haven't seen a lot of, you know, um, teams or individuals who are actually within a multi-stage build. There's nothing really against that, but they think the number one purpose of doing multi-stage had been just producing the minimalist image. Um, so just wanted to kind of combine those two answers in one, uh, for sure. >>Yeah, yeah, sure. Um, and with that, um, thank you all for the great questions. We are going to have to wrap this up and we could go for another hour if we all had the time. And if Dr. Khan was a 24 hour long event and it didn't sadly, it's not. So we've got to make room for the next live panel, which will be Peter coming on and talking about security with some developer ex security experts. And I wanted to thank again, thank you all three of you for being here real quick, go around the room. Um, uh, where can people reach out to you? I am, uh, at Bret Fisher on Twitter. You can find me there. Carlos. >>I'm at dev Mandy with a Y D E N D Y that's me, um, >>Easiest name ever on Twitter, Carlos and DFW on LinkedIn. And I also have a LinkedIn learning course. So if you check me out on my LinkedIn learning, >>Yeah. I'm at Nicola Quebec. Um, one word, I'll put it in the chat as well on, on LinkedIn, as well as, uh, uh, as well as Twitter. Thanks for having us, Brett. Yeah. Thanks for being here. >>Um, and, and you all stay around. So if you're in the room with us chatting, you're gonna, you're gonna, if you want to go to see the next live panel, I've got to go back to the beginning and do that whole thing, uh, and find the next, because this one will end, but we'll still be in chat for a few minutes. I think the chat keeps going. I don't actually know. I haven't tried it yet. So we'll find out here in a minute. Um, but thanks you all for being here, I will be back a little bit later, but, uh, coming up next on the live stuff is Peter Wood security. Ciao. Bye.