>>and hello, we're back. I've got my panel and we are doing things real time here. So sorry for the delay a few minutes late. So the way let's talk about things, the reason we're here and we're going around the room and introduce everybody. Got three special guests here. I got my evil or my john and the normal And we're going to talk about get ops I called it future office just because I want to think about what's the next thing for that at the end, we're gonna talk about what our ideas for what's next for getups, right? Um, because we're all starting to just get into get ups now. But of course a lot of us are always thinking about what's next? What's better? How can we make this thing better? So we're going to take your questions. That's the reason we're here, is to take your questions and answer them. Or at least the best we can for the next hour. And all right, so let's go around the room and introduce yourself. My name is Brett. I am streaming from Brett from that. From Brett. From Virginia Beach in Virginia beach, Virginia, United States. Um, and I talk about things on the internet, I sell courses on you, to me that talk about Docker and kubernetes Ive or introduce yourself. >>How's it going? Everyone, I'm a software engineer at axel Springer, currently based in Berlin and I happen to be Brett Brett's teaching assistant. >>All right, that's right. We're in, we're in our courses together almost every day. Mm john >>hey everyone, my name is john Harris, I used to work at Dhaka um, I now work at VM ware is a star field engineer. Um, so yeah, >>and normal >>awesome by the way, you are streaming from Brett Brett, >>I answered from breath to breath. >>Um I'm normal method. I'm a distinguished engineer with booz allen and I'm also a doctor captain and it's good to see either in person and it's good to see you again john it's been a little while. >>It has the pre covid times, right? You're up here in Seattle. >>Yeah. It feels, it feels like an eternity ago. >>Yeah, john shirt looks red and reminds me of the Austin T shirt. So I was like, yeah, so we all, we all have like this old limited edition doctor on E. >>T. That's a, that's a classic. >>Yeah, I scored that one last year. Sometimes with these old conference church, you have to like go into people's closets. I'm not saying I did that. Um, but you know, you have to go steal stuff, you to find ways to get the swag >>post post covid. If you ever come to my place, I'm going to have to lock the closets. That >>that's right, That's right. >>So the second I think it was the second floor of the doctor HQ in SAn Francisco was where they kept all the T shirts, just boxes and boxes and boxes floor to ceiling. So every time I went to HQ you just you just as many as you can fit in your luggage. I think I have about 10 of these. You >>bring an extra piece of luggage just for your your shirt shirt grab. Um All right, so I'm going to start scanning questions uh so that you don't have to you can you help you all are welcome to do that. And I'm going to start us off with the topic. Um So let's just define the parameters. Like we can talk about anything devops and here we can go down and plenty of rabbit holes. But the kind of, the goal here is to talk about get ups and get ups if you haven't heard about it is essentially uh using versioning systems like get like we've all been getting used to as developers to track your infrastructure changes, not just your code changes and then automate that with a bunch of tooling so that the robots take over. And essentially you have get as a central source of truth and then get log as a central source of history and then there's a bunch of magic little bits in the middle and then supposedly everything is wonderful. It's all automatic. The reality is is what it's often quite messy, quite tricky to get everything working. And uh the edges of this are not perfect. Um so it is a relatively new thing. It's probably three, maybe four years old as an official thing from. We've uh so we're gonna get into it and I'll let's go around the room and the same word we did before and um not to push on that, put you on the spot or anything. But what is, what is one of the things you either like or either hate about getups um that you've enjoyed either using it or you know, whatever for me. I really, I really love that I can point people to a repo that basically is hopefully if they look at the log a tracking, simplistic tracking of what might have changed in that part of the world or the environment. I remember many years past where, you know, I've had executive or some mid level manager wants to see what the changes were or someone outside my team went to see what we just changed. It was okay, they need access to this system into that dashboard and that spreadsheet and then this thing and it was always so complicated and now in a world where if we're using get up orbit bucket or whatever where you can just say, hey go look at that repo if there was three commits today, probably three changes happened. That's I love that particular part about it. Of course it's always more complicated than that. But um Ive or I know you've been getting into this stuff recently. So um any thoughts? Yeah, I think >>my favorite part about get ops is >>reproducibility. Um >>you know the ability to just test something and get it up and running >>and then just tear it down. >>Uh not >>being worried that how did I configure it the first time? I think that's my favorite part about >>it. I'm changing your background as we do this. >>I was going to say, did you just do it get ups pushed to like change his >>background, just a dialogue that different for that green screen equals false? Uh Change the background. Yeah, I mean, um and I mean I think last year was really my first year of actually using it on anything significant, like a real project. Um so I'm still, I still feel like I'm very new to john you anything. >>Yeah, it's weird getups is that thing which kind of crystallizes maybe better than anything else, the grizzled veteran life cycle of emotions with the technology because I think it's easy to get super excited about something new. And when I first looked into get up, so I think this is even before it was probably called getups, we were looking at like how to use guest source of truth, like everything sounds great, right? You're like, wait, get everyone knows, get gets the source of truth, There's a load of robust tooling. This just makes a sense. If everything dies, we can just apply the get again, that would be great. Um and then you go through like the trough of despair, right? We're like, oh no, none of this works. The application is super stateless if this doesn't work and what do we do with secrets and how do we do this? Like how do we get people access in the right place and then you realize everything is terrible again and then everything it equalizes and you're kind of, I think, you know, it sounds great on paper and they were absolutely fantastic things about it, but I think just having that measured approach to it, like it's, you know, I think when you put it best in the beginning where you do a and then there's a magic and then you get C. Right, like it's the magic, which is >>the magic is the mystery, >>right? >>Magic can be good and bad and in text so >>very much so yeah, so um concurrence with with john and ever uh in terms of what I like about it is the potential to apply it to moving security to left and getting closer to a more stable infrastructures code with respect to the whole entire environment. Um And uh and that reconciliation loop, it reminds me of what, what is old is new again? Right? Well, quote unquote old um in terms of like chef and puppet and that the reconciliation loop applied in a in a more uh in a cleaner interface and and into the infrastructure that we're kind of used to already, once you start really digging into kubernetes what I don't like and just this is in concurrence with the other Panelist is it's relatively new. It has um, so it has a learning curve and it's still being, you know, it's a very active um environment and community and that means that things are changing and constantly and there's like new ways and new patterns as people are exploring how to use it. And I think that trough of despair is typically figuring out incrementally what it actually is doing for you and what it's not going to solve for you, right, john, so like that's that trough of despair for a bit and then you realize, okay, this is where it fits potentially in my architecture and like anything, you have to make that trade off and you have to make that decision and accept the trade offs for that. But I think it has a lot of promise for, for compliance and security and all that good stuff. >>Yeah. It's like it's like the potentials, there's still a lot more potential than there is uh reality right now. I think it's like I feel like we're very early days and the idea of especially when you start getting into tooling that doesn't appreciate getups like you're using to get up to and use something else and that tool has no awareness of the concept so it doesn't flow well with all of the things you're trying to do and get um uh things that aren't state based and all that. So this is going to lead me to our first question from Camden asking dumb questions by the way. No dumb questions here. Um How is get apps? Not just another name for C. D. Anybody want to take that as an answer as a question. How is get up is not just another name for C. D. I have things but we can talk about it. I >>feel like we need victor foster kids. Yeah, sure you would have opinions. Yeah, >>I think it's a very yeah. One person replied said it's a very specific it's an opinionated version of cd. That's a great that's a great answer like that. Yeah. >>It's like an implement. Its it's an implementation of deployment if you want it if you want to use it for that. All right. I realize now it's kind of hard in terms of a physical panel and a virtual panel to figure out who on the panel is gonna, you know, ready to jump in to answer a question. But I'll take it. So um I'll um I'll do my best inner victor and say, you know, it's it's an implementation of C. D. And it's it's a choice right? It's one can just still do docker build and darker pushes and doctor pulls and that's fine. Or use other technologies to deploy containers and pods and change your, your kubernetes infrastructure. But get apps is a different implementation, a different method of doing that same thing at the end of the day. Yeah, >>I like it. I like >>it and I think that goes back to your point about, you know, it's kind of early days still, I think to me what I like about getups in that respect is it's nice to see kubernetes become a platform where people are experimenting with different ways of doing things, right? And so I think that encourages like lots of different patterns and overall that's going to be a good thing for the community because then more, you know, and not everything needs to settle in terms of only one way of doing things, but a lot of different ways of doing things helps people fit, you know, the tooling to their needs, or helps fit kubernetes to their needs, etcetera. Yeah, >>um I agree with that, the, so I'm gonna, since we're getting a load of good questions, so um one of the, one of the, one of the, I want to add to that real quick that one of the uh from the, we've people themselves, because I've had some on the show and one of things that I look at it is distinguishing is with continuous deployment tools, I sort of think that it's almost like previous generation and uh continuous deployment tools can be anything like we would consider Jenkins cd, right, if you if you had an association to a server and do a doctor pull and you know, dr up or dr composed up rather, or if it did a cube control apply uh from you know inside an ssh tunnel or something like that was considered considered C. D. Well get ops is much more rigid I think in terms of um you you need to apply, you have a specific repo that's all about your deployments and because of what tool you're using and that one your commit to a specific repo or in a specific branch that repo depends on how you're setting it up. That is what kicks off a workflow. And then secondly there's an understanding of state. So a lot of these tools now I have uh reconciliation where they they look at the cluster and if things are changing they will actually go back and to get and the robots will take over and will commit that. Hey this thing has changed um and you maybe you human didn't change it, something else might have changed it. So I think that's where getups is approaching it, is that ah we we need to we need to consider more than just a couple of commands that be runnin in a script. Like there needs to be more than that for a getups repo to happen anyway, that's just kind of the the take back to take away I took from a previous conversation with some people um >>we've I don't think that lost, its the last piece is really important, right? I think like for me, C d like Ci cd, they're more philosophical ideas, write a set of principles, right? Like getting an idea or a code change to environments promoting it. It's very kind of pipeline driven um and it's very imperative driven, right? Like our existing CD tools are a lot of the ways that people think about Cd, it would be triggered by an event, maybe a code push and then these other things are happening in sequence until they either fail or pass, right? And then we're done. Getups is very much sitting on the, you know, the reconciliation side, it's changing to a pull based model of reconciliation, right? Like it's very declarative, it's just looking at the state and it's automatically pulling changes when they happen, rather than this imperative trigger driven model. That's not to say that there aren't city tools which we're doing pull based or you can do pull based or get ups is doing anything creatively revolutionary here, but I think that's one of the main things that the ideas that are being introduced into those, like existing C kind of tools and pipelines, um certainly the pull based model and the reconciliation model, which, you know, has a lot in common with kubernetes and how those kind of controllers work, but I think that's the key idea. Yeah. >>Um This is a pretty specific one Tory asks, does anyone have opinions about get ops in a mono repo this is like this is getting into religion a little bit. How many repos are too many repose? How um any thoughts on that? Anyone before I rant, >>go >>for it, go for it? >>Yeah. How I'm using it right now in a monitor repo uh So I'm using GIT hub. Right, so you have what? The workflow and then inside a workflow? Yeah, mo file, I'll >>track the >>actual changes to the workflow itself, as well as a folder, which is basically some sort of service in Amman Arepa, so if any of those things changes, it'll trigger the actual pipeline to run. So that's like the simplest thing that I could figure out how to, you know, get it set up using um get hubs, uh workflow path future. Yeah. And it's worked for me for writing, you know? That's Yeah. >>Yeah, the a lot of these things too, like the mono repo discussion will, it's very tool specific. Each tool has various levels of support for branch branching and different repos and subdirectories are are looking at the defense and to see if there's changes in that specific directory. Yeah. Sorry, um john you're going to say something, >>I was just going to say, I've never really done it, but I imagine the same kind of downsides of mono repo to multiple report would exist there. I mean, you've got the blast radius issues, you've got, you know, how big is the mono repo? Do we have to pull does the tool have to pull that or cashier every time it needs to determine def so what is the support for being able to just look at directories versus you know, I think we can get way down into a deeper conversation. Maybe we'll save it for later on in the conversation about what we're doing. Get up, how do we structure our get reposed? We have super granular repo per environment, Perper out reaper, per cluster repo per whatever or do we have directories per environment or branches per environment? How how is everything organized? I think it's you know, it's going to be one of those, there's never one size fits all. I'll give the class of consultant like it depends answer. Right? >>Yeah, for sure. It's very similar to the code struggle because it depends. >>Right? >>Uh Yeah, it's similar to the to the code problem of teams trying to figure out how many repose for their code. Should they micro service, should they? Semi micro service, macro service. Like I mean, you know because too many repose means you're doing a bunch of repo management, a bunch of changes on your local system, you're constantly get pulling all these different things and uh but if you have one big repo then it's it's a it's a huge monolithic thing that you usually have to deal with. Path based issues of tools that only need to look at a specific directory and um yeah, it's a it's a culture, I feel like yeah, like I keep going back to this, it's a culture thing. Does your what is your team prefer? What do you like? What um what's painful for everyone and who's what's the loudest pain that you need to deal with? Is it is it repo management? That's the pain um or is it uh you know, is that that everyone's in one place and it's really hard to keep too many cooks out of the kitchen, which is a mono repo problem, you know? Um How do we handle security? So this is a great one from Tory again. Another great question back to back. And that's the first time we've done that um security as it pertains to get up to anyone who can commit can change the infrastructure. Yes. >>Yes. So the tooling that you have for your GIT repo and the authentication, authorization and permissions that you apply to the GIT repo using a get server like GIT hub or get lab or whatever your flavor of the day is is going to be how security is handled with respect to changes in your get ups configuration repository. So um that is completely specific to your implementation of that or ones implementation of of how they're handling that. Get repositories that the get ups tooling is looking at. To reconcile changes with respect to the permissions of the for lack of better term robot itself. Right? They get up tooling like flux or Argosy. D Um one kid would would create a user or a service account or uh other kind of authentication measures to limit the permissions for that service account that the Gaddafi's tooling needs to be able to read the repose and and send commits etcetera. So that is well within the realm of what you have already for your for your get your get um repo. Yeah. >>Yeah. A related question is from a g what they like about get apps if done nicely for a newbie it's you can get stuff done easily if you what they dislike about it is when you have too many get repose it becomes just too complicated and I agree. Um was making a joke with a team the other week that you know the developer used to just make one commit and they would pass pass it on to a QA team that would then eventually emerging in the master. But they made the commits to these feature branches or whatever. But now they make a commit, they make a pR there for their code then they go make a PR in the helm chart to update the thing to do that and then they go make a PR in the get ups repeal for Argo. And so we talked about that they're probably like four or five P. R. Is just to get their code in the production. But we were talking about the negative of that but the reality was It's just five or 4 or five prs like it wasn't five different systems that had five different methodologies and tooling and that. So I looked at it I was like well yeah that's kind of a pain in the get sense but you're also dealing with one type. It's a repetitive action but it's it's the one thing I don't have to go to five different systems with five different ways of doing it. And once in the web and one's on the client wants a command line that I don't remember. Um Yeah so it's got pros and cons I think when you >>I think when you get to the scale where those kind of issues are a problem then you're probably at the scale where you can afford to invest some time into automation into that. Right? Like what I've when I've seen this in larger customers or larger organizations if there ever at that stage where okay apps are coming up all the time. You know, there's a 10 X 100 X developer to operations folks who may be creating get repose setting up permissions then that stuff gets automated, right? Like, you know, maybe ticket based systems or whatever. Developers say I need a new app. It templates things or more often using the same model, right of reconciliation and operators and the horrific abuse of cogs that we're seeing in the communities community right now. Um You know, developers can create a crd which just says, hey, I'm creating a new app is called app A and then a controller will pick up that app a definition. It will go create a get a repo Programmatically it will add the right definitely will look up and held up the developers and the permissions that need to be able to get to that repo it will create and template automatically some name space and the clusters that it needs in the environments that it needs, depending on, you know, some metadata it might read. So I think, you know, those are definite problems and they're definitely like a teething, growing pain thing. But once you get to that scale, you kind of need to step back and say, well look, we just need to invest in time into the operational aspect of this and automating this pain away, I think. Yeah, >>yeah. And that ultimately ends in Yeah. Custom tooling, which it's hard to avoid it at scale. I mean, there's there's two, there's almost two conversations here, right. There is what I call the Solo admin Solo devops, I bought that domain Solo devops dot com because, you know, whenever I'm talking to dr khan in the real world, it's like I asked people to raise hands, I don't know how we can raise hands here, but I would ask people to raise hands and see how many of you here are. The sole person responsible for deploying the app that your team makes and like a quarter of the room would raise their hand. So I call that solo devops like those, that person can't make all the custom tooling in the world. So they really need dr like solutions where it's opinionated, the workflow is sort of built in and they don't have to wrangle things together with a bunch of glue, you know, in other words bash. Um and so this kind of comes to a conversation uh starting this question from lee he's asking how do you combine get ops with ci cd, especially the continuous bit. How do you avoid having a human uh sort of the complaint the team I was working with has, how do you avoid a human editing and get committing for every single deploy? They've settled on customized templates and a script for routine updates. So as a seed for this conference, this question I'm gonna ask you all uh instead of that specific question cause it's a little open ended. Um Tell me whether you agree with this. I I kind of look at the image, the image artifact because the doctor image or container image in general is an artifact that I I view it that way and that thing going into the registry with the right label or right part of the label. Um That tag rather not the label but the tag that to me is like one of the great demarche points of, we're kind of done with Ci and we're now into the deployment phase and it doesn't necessarily mean the tooling is a clear cut there, but that artifact being shipped in a specific way or promoted as we sometimes say. Um what do you think? Does anyone have opinions on that? I don't even know if that's the right opinion to have so mhm. >>So um I think what you're, what you're getting at is that get ups, models can trigger off of different events um to trigger the reconciliation loop. And one way to do that is if the image, if it notices a image change in the registry, the other is if there's a commit event on a specific rebo and branch and it's up to, you are up to the person that's implementing their get ups model, what event to trigger there, that reconciliation loop off of, You can do both, you can do one or the other. It also depends on the Templeton engine that you're using on top of um on top of kubernetes, such as helm or um you know, the other ones that are out there or if you're not even doing that, then, you know straight. Yeah, mo um so it kind of just depends, but those are the typically the two options one has and a combination of of those to trigger that event. You can also just trigger it manually, right? You can go into the command line and force a a, you know, a really like a scan or a new reconciliation loop to occur. So it kind of just, I don't want to say this, but it depends on what you're trying to do and what makes sense in your pipeline. Right? So if you're if you're set up where you are tag, if you're doing it based off of image tags, then you probably want to use get ups in a way that you're using the image tags. Right. And the pattern that you've established there, if you're not really doing that and you're more around, like, different branches are mapped to different environments, then triggered off of the correct branch. And that's where the permissions also come into play. Where if you don't want someone to touch production and you've got your getups for your production cluster based off of like uh you know, a main branch, then whoever can push a change to that main branch has the authority to push that change to production. Right? So that's your authentication and permissions um system same for the registry itself. Right. So >>Yeah. Yeah. Sorry, anyone else have any thoughts on that? I was about to go to the next topic, >>I was going to say. I think certain tools dictate the approach, like, if you're using Argosy d it's I think I'm correct me if I'm wrong, but I think the only way to use it right now is just through image modification. Like, the manifest changes, it looks at a specific directory and anything changes then it will do its thing. And uh Synchronize the cost there with whatever's and get >>Yeah, flux has both. Yeah, and flux has both. So it it kind of depends. I think you can make our go do that too, but uh this is back to what we were saying in the beginning, uh you know, these things are changing, right? So that might be what it is right now in terms of triggering the reconciliation loops and get ups, tooling, but there might be other events in the future that might trigger it, and it's not completely stand alone because you still need you're tooling to do any kind of testing or whatever you have in terms of like the specific pipeline. So oftentimes you're bolting in getups into some other part of broader Cfd solution. That makes sense. Yeah, >>we've got a lot of questions about secrets or people that are asking about secrets. >>So my my tongue and cheek answered the secrets question was, what's the best practices for kubernetes? Secrets? That's the same thing for secrets with good apps? Uh getups is not last time I checked and last time I was running this stuff get ups is not has nothing to do with secrets in that sense. It's just there to get your stuff running on communities. So, um there's probably a really good session on secrets at dr concept. I >>would agree with you, I agree with you. Yeah, I mean, get off stools, I mean every every project of mine handles secrets differently. Uh huh. And I think I'm not sure if it was even when I was talking to but talking to someone recently that I'm very bullish on get up actions, I love get up actions, it's not great for deployments yet, but we do have this new thing and get hub environments, I think it's called. So it allows me at least the store secrets per environment, which it didn't have the concept of that before, which you know, if you if any of you running kubernetes out there, you typically end up when you start running kubernetes, you end up with more than one kubernetes, like you're going to end up with a lot of clusters at some point, at least many multiple, more than two. Um and so if you're trying to store secret somewhere, you do have and there's a discussion happening in chat right now where people are talking about um sealed secrets which if you haven't heard of that, go look that up and just be versed on what sealed secrets is because it's a it's a fantastic concept for how to store secrets in the public. Um I love it because I'm a big P. K. I nerd but um it's not the only way and it doesn't fit all models. So I have clients that use A W. S. Secrets because they're in A W. S. And then they just have to use the kubernetes external secret. But again like like like normal sand, you know, it's that doesn't really affect get ops, get ops is just applying whatever helm charts or jahmal or images that you're, you're you're deploying, get off. It was more about the approach of when the changes happen and whether it's a push or pull model like we're talking about and you know, >>I would say there's a bunch of prerequisites to get ups secrets being one of them because the risk of you putting a secret into your git repo if you haven't figured out your community secrets architecture and start diving into getups is high and removing secrets from get repose is you know, could be its own industry, right. It's >>a thing, >>how do >>I hide this? How do I obscure this commit that's already now on a dozen machines. >>So there are some prerequisites in terms of when you're ready to adopt get up. So I think is the right way of saying the answer to that secrets being one of them. >>I think the secrets was the thing that made me, you know, like two or three years ago made me kind of see the ah ha moment when it came to get ups which, which was that the premier thing that everyone used to say about get up about why it was great. Was its the single source of truth. There's no state anywhere else. You just need to look at git. Um and then secrets may be realized along with a bunch of other things down the line that is not true and will never be true. So as soon as you can lose the dogmatism about everything is going to be and get it's fantastic. As long as you've understood everything is not going to get. There are things which will absolutely never be and get some tools just don't deal with that. They need to earn their own state, especially in communities, some controls on their own state. You know, cuz sealed secrets and and other projects like SOps and I think there are two or three others. That's a great way of dealing with secrets if you want to keep them in get. But you know, projects like vault more kind of like what I would say, production grade secret strategies. Right? And if you're in AWS or a cloud, you're more likely to be using their secrets. Your secret policy is maybe not dictated by you in large organizations might be dictated by CSO or security or Great. Like I think once if you, if you're trying to adopt getups or you're thinking about it, get the dogmatism of get as a single point of truth out of your mind and think about getups more as a philosophy and a set of best practice principles, then you will be in much better stead, >>right? Yeah. >>People are asking more questions in chat like infrastructure as code plus C d essentially get ups or C I rather, um, these are all great questions and a part of the debate, I'm actually just going to throw up on screen. I'm gonna put this in chat, but this is, this is to me the source, Right? So we worked with when they coined the term. We, a lot of us have been trying to get, if we talk about the history for a minute and then tell me if I'm getting this right. Um, a lot of us were trying to automate all these different parts of the puzzle, but a lot of them, they, some things might have been infrastructure as code. Some things weren't, some things were sort of like settings is coded, like you're going to Jenkins and type in secrets and settings or type in a certain thing in the settings of Jenkins and then that it wasn't really in get and so what we was trying to go for was a way to have almost like eventually a two way state understanding where get might change your infrastructure but then your infrastructure might also change and needs to be reflected in the get if the get is trying to be the single source of truth. Um and like you're saying the reality is that you're never gonna have one repo that has all of your infrastructure in it, like you would have to have, you have to have all your terra form, anything else you're spinning up. Right. Um but anyway, I'm gonna put this link in chat. So this guide actually, uh one of things they talk about is what it's not, so it's, it's kind of great to read through the different requirements and like what I was saying well ago um mhm. Having having ci having infrastructure as code and then trying a little bit of continuous deployment out, it's probably a prerequisite. Forget ops so it's hard to just jump into that when you don't already have infrastructure as code because a machine doing stuff on your behalf, it means that you have to have things documented and somewhere and get repo but let me put this in the in the >>chitty chat, I would like to know if the other panelists agree, but I think get apps is a okay. I would say it's a moderate level, it's not a beginner level communities thing, it's like a moderate level advanced, a little bit more advanced level. Um One can start off using it but you definitely have to have some pre recs in place or some understanding of like a pattern in place. Um So what do the other folks think about that opinion? >>I think if you're if you're trying to use get out before, you know what problem you have, you're probably gonna be in trouble. Right. It's like having a solution to it probably don't have yet. Mhm. Right. I mean if if you're just evil or and you're just typing, keep control apply, you're one person right, Get off. It doesn't seem like a big a big jump, like, I mean it doesn't like why would I do that? I'm just, I'm just gonna inside, it's the type of get commit right, I'm typing Q control apply. But I think one of the rules from we've is none of your developers and none of your admins can have cute control access to the cluster because if you can't, if you do have access and you can just apply something, then that's just infrastructure as code. That's just continuous deployment, that's, that's not really get ops um, getups implies that the only way things get into the cluster is through the get up, get automation that you're using with, you know, flux Argo, we haven't talked about, what's the other one that Victor Farsi talks about, by the way people are asking about victor, because victor would love to talk about this stuff, but he's in my next life, so come back in an hour and a half or whatever and victor is going to be talking about sys, admin list with me. Um >>you gotta ask him nothing but get up questions in the next, >>confuse them, confuse them. But anyway, that, that, that's um, it's hard, it's hard to understand and without having tried it, I think conceptually it's a little challenging >>one thing with getups, especially based off the we've works blog post that you just put up on there. It's an opinionated way of doing something. Uh you know, it's an opinionated way of of delivering changes to an environment to your kubernetes environment. So it's opinionated were often not used to seeing things that are very opinionated in this sense, in the in the ecosystem, but get apps is a opinionated thing. It's it's one way of doing it. Um there are ways to change it and like there are options um like what we were talking about in terms of the events that trigger, but the way that it's structured is an opinion opinionated way both from like a tooling perspective, like using get etcetera, but also from a devops cultural perspective, right? Like you were talking about not having anyone access cube control and changing the cluster directly. That's a philosophical opinion that get ups forces you to adopt otherwise. It kind of breaks the model and um I just I want everyone to just understand that. That is very opinion, anything in that sense. Yeah, >>polygamy is another thing. Infrastructure as code. Um someone's mentioning plummy and chat, I just had actually my life show self plug bread that live go there. I'm on Youtube every week. I did the same thing. These these are my friends um and had palami on two weeks ago uh last week, remember uh and it was in the last couple of weeks and we talked about their infrastructure as code solution. Were actually writing code instead of um oh that's an interesting take on uh developer team sort of owning coding the infrastructure through code rather than Yamil as a data language. I don't really have an opinion on it yet because I haven't used it in production or anything in the real real world, but um, I'm not sure how much they are applying trying to go towards the get up stuff. I will do a plug for Solomon hikes. Who has a, the beginning of the day, it's already happened so you can go back and watch it. It's a, it's a, what's it called? Q. Rethinking application delivery with Q. And build kit. So go look this up. This is the found co founder of Dr and former CTO Solomon hikes at the beginning of the day. He has a tool called dagger. I'm not sure why the title of the talk is delivering with Q. And built it, but the tool is showing off in there for an hour is called dagger. And it's, it's an interesting idea on how to apply a lot of this opinionated automated stuff to uh, to deployment and it's get off space and you use Q language. It's a graph language. I watched most of it and it was a really interesting take. I'm excited to see if that takes off and if they try that because it's another way that you can get a little bit more advanced with your you're get deployments and without having to just stick everything in Yemen, which is kind of what we're in today with helm charts and what not. All right. More questions about secrets, I think. I think we're not going to have a whole lot of more, a lot more about secrets basically. Uh put secrets in your cluster to start with and kubernetes in encrypted, you know, thing. And then, you know, as it gets harder, then you have to find another solution when you have five clusters, you don't wanna have to do it five times. That's when you have to go for Walton A W. S secrets and all >>that. Right? I'm gonna post it note. Yeah. Crm into the cluster. Just kidding. >>Yes, there are recordings of this. Yes, they will be later. Uh, because we're that these are all gonna be on youtube later. Um, yeah, detects secrets cushion saying detect secrets or get Guardian are absolute requirements. I think it's in reference to your secrets comment earlier. Um, Camels asking about Cuban is dropping support for Docker that this is not the place to ask for that, but it, it is uh, basically it's a Nonevent Marantz has actually just created that same plug in available in a different repos. So if you want to keep using Docker and kubernetes, you know, you can do it like it's no big deal. Most of us aren't using doctor in our communities anyway, so we're using like container D or whatever is provided to us by our provider. Um yeah, thank you so much for all these comments. These are great people helping each other and chat. I feel like we're just here to make sure the chats available so people can help each other. >>I feel like I want to pick up on something when you mentioned pollux me, I think there's a um we're talking about getups but I think in the original like the origination of that I guess was deploying applications to clusters right, picking up deployment manifest. But I think with the gloomy and I obviously terra form and things have been around a long time, folks are starting to apply this I think I found one earlier which was like um kub stack the Terror Forms get ups framework. Um but also with the advent of things like cluster A. P. I. Um in the Cuban at the space where you can declare actively build the infrastructure for your clusters and build the cluster right? We're not just talking about deploying applications, the cluster A. P. I will talk to a W. S. Spin up, VPc spin up machines, you know, we'll do the same kind of things that terra form does and and those other tools do I think applying getups principles to the infrastructure spin up right, the proper infrastructure as code stuff, constantly applying Terror form um you know, plans and whatever, constantly applying cluster Api resources spinning up stuff in those clouds. That's a super interesting. Um you know, extension of this area, I'd be curious to see if what the folks think about that. >>Yeah, that's why I picked this topic is one of my three. Uh I got I got to pick the topics. I was like the three things that there like the most bleeding edge exciting. Most people haven't, we haven't basically we haven't figured all this out yet. We as an industry, so um it's I think we're gonna see more ideas on it. Um what's the one with the popsicle as the as the icon victor talks about all the time? It's not it's another getups like tool, but it's um it's getups for you use this kubernetes limit and then we have to look it up, >>You're talking about cross plane. >>So >>my >>wife is over here with the sound effects and the first sound effect of the day that she chooses to use is one. >>All right, can we pick it? Let's let's find another question bret >>I'm searching >>so many of them. All right, so uh I think one really quick one is getups only for kubernetes, I think the main to tooling to tools that we're talking about, our Argosy D and flux and they're mostly geared toward kubernetes deployments but there's a, it seems like they're organized in a way that there's a clean abstraction in with respect to the agent that's doing the deployment and the tooling that that can interact with. So I would imagine that in the future and this might be true already right now that get ups could be applied to other types of deployments at some point in the future. But right now it's mostly focused and treats kubernetes as a first class citizen or the tooling on top of kubernetes, let's say something like how as a first class citizen? Yeah, to Brett, >>to me the field, back to you bret the thing I was looking for is cross plane. So that's another tool. Um Victor has been uh sharing a lot about it in Youtube cross plane and that is basically runs inside a kubernetes, but it handles your other infrastructure besides your app. It allows you to like get ops, you're a W. S stuff by using the kubernetes state engine as a, as a way to manage that. And I have not used it yet, but he does some really great demos on Youtube. So people are liking this idea of get off, so they're trying to figure out how do we, how do we manage state? How do we uh because the probably terra form is that, well, there's many problems, but it's always a lot of problems, but in the get outs world it's not quite the right fit yet, It might be, but you still, it's still largely as expected for people to, you know, like type the command, um, and it keeps state locally the ss, clouds and all that. And but the other thing is I'm I'm now realizing that when I saw the demo from Solomon, I'm going back to the Solomon hikes thing. He was using the demo and he was showing it apply deploying something on S three buckets, employing internet wifi and deploying it on google other things beyond kubernetes and saying that it's all getups approach. So I think we're just at the very beginning of seeing because it all started with kubernetes and now there's a swarm one, you can look up swarm, get office and there's a swarm, I can't take the name of it. Swarm sink I think is what's called swarm sink on git hub, which allows you to do swarm based getups like things. And now we're seeing these other tools coming out. They're saying we're going to try to do the get ups concepts, but not for kubernetes specifically and that's I think, you know, infrastructure as code started with certain areas of the world and then now then now we all just assume that you're going to have an infrastructure as code way of doing whatever that is and I think get off is going to have that same approach where pretty soon, you know, we'll have get apps for all the clouds stuff and it won't just be flexor Argo. And then that's the weird thing is will flex and Argo support all those things or will it just be focused on kubernetes apps? You know, community stuff? >>There's also, I think this is what you're alluding to. There is a trend of using um kubernetes and see rDS to provision and control things that are outside of communities like the cloud service providers services as if they were first class entities within kubernetes so that you can use the kubernetes um focus tooling for things that are not communities through the kubernetes interface communities. Yeah, >>yeah, even criticism. >>Yeah, yeah, I'm just going to say that sounds like cross plane. >>Yeah, yeah, I mean, I think that's that's uh there were, you know, for the last couple of years, it's been flux and are going back and forth. Um they're like frenemies, you know, and they've been going back and forth with iterating on these ideas of how do we manage this complicated thing? That is many kubernetes clusters? Um because like Argo, I don't know if the flux V two can do this, but Argo can manage multiple clusters now from one cluster, so your, you can manage other clusters, technically external things from a single entity. Um Originally flux couldn't do that, but I'm going to say that V two can, I don't actually >>know. Um I think all that is gonna, I think that's going to consolidate in the future. All right. In terms of like the common feature set, what Iver and john what do you think? >>I mean, I think it's already begun, right, I think haven't, didn't they collaborate on a common engine? I don't know whether it's finished yet, but I think they're working towards a common getups engine and then they're just going to layer on features on top. But I think, I mean, I think that's interesting, right, because where it runs and where it interacts with, if we're talking about a pull based model, it shouldn't, it's decentralized to a certain extent, right? We need get and we need the agent which is pulling if we're saying there's something else which is orchestrating something that we start to like fuzzy the model even right. Like is this state living somewhere else, then I think that's just interesting as well. I thought flux was completely decentralized, but I know you install our go somewhere like the cargo has a server as well, but it's been a while since I've looked in depth at them. But I think the, you know, does that muddy the agent only pull model? >>I'm reading a >>Yeah, I would say that there's like a process of natural selection going on as as the C. N. C. F. Landscape evolves and grows bigger and a lot of divide and conquer right now. But I think as certain things kind of get more prominent >>and popular, I think >>it starts to trend and it inspires other things and then it starts to aggregate and you know, kind of get back into like a unified kind of like core. Maybe like for instance, cross plane, I feel like it shouldn't even really exist. It should be, it like it's a communities add on, but it should be built in, it should be built into kubernetes, like why doesn't this exist already >>for like controlling a cloud? >>Yeah, like just, you know, having this interface with the cloud provider and be able to Yeah, >>exactly. Yeah, and it kinda, you're right. That kinda happens because you do, I mean when you start talking about storage providers and networking providers was very specific implementations of operators or just individual controllers that do operate and control other resources in the cloud, but certainly not universally right. Not every feature of AWS is available to kubernetes out of the box. Um and you know, it, one of the challenges across plane is you gotta have kubernetes before you can deploy kubernetes. Like there's a chicken and egg issue there where if you're going to use, if you're going to use our cross plane for your other infrastructure, but it's gotta, but it has to run on kubernetes who creates that first kubernetes in order for you to put that on there. And victor talks about one of his videos, the same problem with flux and Argo where like Argo, you can't deploy Argo itself with getups. There has to be that initial, I did a thing with, I'm a human and I typed in some commands on a server and things happened but they don't really have an easy deployment method for getting our go up and running using simply nothing but a get push to an existing system. There's something like that. So it's a it's an interesting problem of day one infrastructure which is again only day one, I think data is way more interesting and hard, but um how can we spend these things up if they're all depending on each other and who is the first one to get started? >>I mean it's true of everything though, I mean at the end of that you need some kind of big bang kind of function too, you know, I started running start everything I >>think without going over that, sorry, without going off on a tangent. I was, I was gonna say there's a, if folks have heard of kind which is kubernetes and Docker, which is a mini kubernetes cluster, you can run in a Docker container or each container will run as a as a node. Um you know, that's been a really good way to spin up things like clusters. KPI because they boot strap a local kind, install the manifests, it will go and spin up a fully sized cluster, it will transfer its resources over there and then it will die itself. Right? So that, that's kind of bootstrapping itself. And I think a couple of folks in the community, Jason to Tiberius, I think he works for Quinyx metal um has, has experimented with like an even more minimal just Api server, so we're really just leveraging the kubernetes ideas of like a reconciliation loop and a controller. We just need something to bootstrap with those C R D s and get something going and then go away again. So I think that's gonna be a pattern that comes up kind of more and more >>Yeah, for sure. Um, and uh, the next, next quick answer to the question, Angel asked what your thoughts on getups being a niche to get or versus others vcs tools? Well, if I knew anyone who is using anything other than get, I would say no, you know, get ops is a horrible name. It should just be CVS office, but that doesn't or vcs ops or whatever like that, but that doesn't roll off the tongue. So someone had to come up with the get ups phrase. Um but absolutely, it's all about version control solutions used for infrastructure, not code. Um might get doctor asks a great question, we're not gonna have time for it, but maybe people can reply and chat with what they think but about infrastructure and code, the lines being blurred and that do develop, how much of infrastructure does developer do developers need to know? Essentially, they're having to know all the things. Um so unfortunately we've had way more questions like every panel here today with all the great community, we've got way more questions we can handle in this time. So we're gonna have to wrap it up and say goodbye. Go to the next live panel. I believe the next one is um on developer, developer specific setups that's gonna be peter running that panel. Something about development in containers and I'm sure it's gonna be great. Just like this one. So let's go around the room where can people find you on the internet? I'm at Brett fisher on twitter. That's where you can usually find me most days you are? >>Yeah, I'm on twitter to um, I'll put it in the chat. It's kind of confusing because the TSR seven. >>Okay. Yeah, that's right. You can't just say it. You can also look at the blow of the video and like our faces are there and if you click on them, it tells you our twitter in Arlington and stuff, john >>John Harris 85, pretty much everywhere. Get hub Twitter slack, etc. >>Yeah >>and normal, normal faults or just, you know, living on Youtube live with Brett. >>Yeah, we're all on the twitter so go check us out there and thank you so much for joining. Uh thank you so much to you all for being here. I really appreciate you taking time in your busy schedule to join me for a little chit chat. Um Yes, all the, all the cheers, yes. >>And I think this kid apps loop has been declarative lee reconciled. >>Yeah, there we go. And with that ladies and gentlemen, uh bid you would do, we will see you in the next, next round coming up next with Peter >>bye.

>>Hey, welcome to the live panel. My name is Brett. I am your host, and indeed we are live. In fact, if you're curious about that, if you don't believe us, um, let's just show a little bit of the browser real quick to see. Yup. There you go. We're live. So, all right. So how this is going to work is I'm going to bring in some guests and, uh, in one second, and we're going to basically take your questions on the topic designer of the day, that continuous integration testing. Uh, thank you so much to my guests welcoming into the panel. I've got Carlos, Nico and Mandy. Hello everyone. >>Hello? All right, >>Let's go. Let's go around the room and all pretend we don't know each other and that the internet didn't read below the video who we are. Uh, hi, my name is Brett. I am a Docker captain, which means I'm supposed to know something about Docker. I'm coming from Virginia Beach. I'm streaming here from Virginia Beach, Virginia, and, uh, I make videos on the internet and courses on you to me, Carlos. Hey, >>Hey, what's up? I'm Carlos Nunez. I am a solutions architect, VMware. I do solution things with computers. It's fun. I live in Dallas when I'm moving to Houston in a month, which is where I'm currently streaming. I've been all over the Northeast this whole week. So, um, it's been fun and I'm excited to meet with all of you and talk about CIA and Docker. Sure. >>Yeah. Hey everyone. Uh, Nico, Khobar here. I'm a solution engineer at HashiCorp. Uh, I am streaming to you from, uh, the beautiful Austin, Texas. Uh, ignore, ignore the golden gate bridge here. This is from my old apartment in San Francisco. Uh, just, uh, you know, keeping that, to remember all the good days, um, that that lived at. But, uh, anyway, I work at Patrick Corp and I work on all things, automation, um, and cloud and dev ops. Um, and I'm excited to be here and Mandy, >>Hi. Yeah, Mandy Hubbard. I am streaming from Austin, Texas. I am, uh, currently a DX engineer at ship engine. Um, I've worked in QA and that's kind of where I got my, uh, my Docker experience and, um, uh, moving into DX to try and help developers better understand and use our products and be an advocate for them. >>Nice. Well, thank you all for joining me. Uh, I really appreciate you taking the time out of your busy schedule to be here. And so for those of you in chat, the reason we're doing this live, because it's always harder to do things live. The reason we're here is to answer a question. So we didn't come with a bunch of slides and demos or anything like that. We're here to talk amongst ourselves about ideas and really here for you. So we've, we obviously, this is about easy CII, so we're, we're going to try to keep the conversation around testing and continuous integration and all the things that that entails with containers. But we may, we may go down rabbit holes. We may go veer off and start talking about other things, and that's totally fine if it's in the realm of dev ops and containers and developer and ops workflows, like, Hey, it's, it's kinda game. >>And, uh, these people have a wide variety of expertise. They haven't done just testing, right? We, we live in a world where you all kind of have to wear many hats. So feel free to, um, ask what you think is on the top of your mind. And we'll do our best to answer. It may, might not be the best answer or the correct answer, but we're going to do our best. Um, well, let's get it start off. Uh, let's, let's get a couple of topics to start off with. Uh, th the, the easy CGI was my, one of my three ideas. Cause he's the, one of the things that I'm most excited about is the innovation we're seeing around easier testing, faster testing, automated testing, uh, because as much as we've all been doing this stuff for, you know, 15 years, since 20 years since the sort of Jenkins early days, um, it it's, it seems like it's still really hard and it's still a lot of work. >>So, um, let's go around the room real quick, and everybody can just kind of talk for a minute about like your experience with testing and maybe some of your pain points, like what you don't like about our testing world. Um, and we can talk about some pains, cause I think that will lead us to kind of talk about what, what are the things we're seeing now that might be better, uh, ideas about how to do this. I know for me, uh, testing, obviously there's the code part, but just getting it automated, but mostly getting it in the hands of developers so that they can control their own testing. And don't have to go talk to a person to run that test again, or the mysterious Jenkins platform somewhere. I keep mentioning Jenkins cause it's, it is still the dominant player out there. Um, so for me, I'm, I'm, I, I don't like it when I'm walking into a room and there's, there's only one or two people that know how the testing works or know how to make the new tests go into the testing platform and stuff like that. So I'm always trying to free those things so that any of the developers are enabled and empowered to do that stuff. So someone else, Carlos, anybody, um, >>Oh, I have a lot of opinions on that. Having been a QA engineer for most of my career. Um, the shift that we're saying is everyone is dev ops and everyone is QA. Th the issue I see is no one asked developers if they wanted to be QA. Um, and so being the former QA on the team, when there's a problem, even though I'm a developer and we're all running QA, they always tend to come to the one of the former QA engineers. And they're not really owning that responsibility and, um, and digging in. So that's kind of what I'm saying is that we're all expected to test now. And some people, well, some people don't know how it's, uh, for me it was kind of an intuitive skill. It just kind of fit with my personality, but not knowing what to look for, not knowing what to automate, not even understanding how your API end points are used by your front end to know what to test when a change is made. It's really overwhelming for developers. And, um, we're going to need to streamline that and, and hold their hands a little bit until they get their feet wet with also being QA. >>Right. Right. So, um, uh, Carlos, >>Yeah, uh, testing is like, Tesla is one of my favorite subjects to talk about when I'm baring with developers. And a lot of it is because of what Mandy said, right? Like a lot of developers now who used to write a test and say, Hey, QA, go. Um, I wrote my unit tests. Now write the rest of the test. Essentially. Now developers are expected to be able to understand how testing, uh, testing methodologies work, um, in their local environments, right? Like they're supposed to understand how to write an integration tasks federate into and tasks, a component test. And of course, how to write unit tests that aren't just, you know, assert true is true, right? Like more comprehensive, more comprehensive, um, more high touch unit tests, which include things like mocking and stubbing and spine and all that stuff. And, you know, it's not so much getting those tests. Well, I've had a lot of challenges with developers getting those tests to run in Docker because of usually because of dependency hell, but, um, getting developers to understand how to write tests that matter and mean something. Um, it's, it's, it can be difficult, but it's also where I find a lot of the enjoyment of my work comes into play. So yeah. I mean, that's the difficulty I've seen around testing. Um, big subject though. Lots to talk about there. >>Yeah. We've got, we've already got so many questions coming in. You already got an hour's worth of stuff. So, uh, Nico 81st thoughts on that? >>Yeah, I think I definitely agree with, with other folks here on the panel, I think from a, um, the shift from a skillset perspective that's needed to adopt the new technologies, but I think from even from, uh, aside from the organizational, um, and kind of key responsibilities that, that the new developers have to kinda adapt to and, and kind of inherit now, um, there's also from a technical perspective as there's, you know, um, more developers are owning the full stack, including the infrastructure piece. So that adds a lot more to the plate in Tim's oaf, also testing that component that they were not even, uh, responsible for before. Um, and, um, also the second challenge that, you know, I'm seeing is that on, you know, the long list of added, um, uh, tooling and, you know, there's new tool every other day. Um, and, um, that kind of requires more customization to the testing, uh, that each individual team, um, any individual developer Y by extension has to learn. Uh, so the customization, uh, as well as the, kind of the scope that had, uh, you know, now in conferences, the infrastructure piece, um, uh, both of act to the, to the challenges that we're seeing right now for, um, for CGI and overall testing, um, uh, the developers are saying, uh, in, in the market today. >>Yeah. We've got a lot of questions, um, about all the, all the different parts of this. So, uh, let me just go straight to them. Cause that's why we're here is for the people, uh, a lot of people asking about your favorite tools and in one of this is one of the challenges with integration, right? Is, um, there is no, there are dominant players, but there, there is such a variety. I mean, every one of my customers seems like they're using a different workflow and a different set of tools. So, and Hey, we're all here to just talk about what we're, what we're using, uh, you know, whether your favorite tools. So like a lot of the repeated questions are, what are your favorite tools? Like if you could create it from scratch, uh, what would you use? Pierre's asking, you know, GitHub actions sounds like they're a fan of GitHub actions, uh, w you know, mentioning, pushing the ECR and Docker hub and, uh, using vs code pipeline, I guess there may be talking about Azure pipelines. Um, what, what's your preferred way? So, does anyone have any, uh, thoughts on that anyone want to throw out there? Their preferred pipeline of tooling? >>Well, I have to throw out mine. I might as Jenkins, um, like kind of a honorary cloud be at this point, having spoken a couple of times there, um, all of the plugins just make the functionality. I don't love the UI, but I love that it's been around so long. It has so much community support, and there are so many plugins so that if you want to do something, you don't have to write the code it's already been tested. Um, unfortunately I haven't been able to use Jenkins in, uh, since I joined ship engine, we, most of our, um, our, our monolithic core application is, is team city. It's a dotnet application and TeamCity plays really well with.net. Um, didn't love it, uh, Ms. Jenkins. And I'm just, we're just starting some new initiatives that are using GitHub actions, and I'm really excited to learn, to learn those. I think they have a lot of the same functionality that you're looking for, but, um, much more simplified in is right there and get hubs. So, um, the integration is a lot more seamless, but I do have to go on record that my favorite CICT tools Jenkins. >>All right. You heard it here first people. All right. Anyone else? You're muted? I'm muted. Carlin says muted. Oh, Carla says, guest has muted themselves to Carlos. You got to unmute. >>Yes. I did mute myself because I was typing a lot, trying to, you know, try to answer stuff in the chat. And there's a lot of really dark stuff in there. That's okay. Two more times today. So yeah, it's fine. Yeah, no problem. So totally. And it's the best way to start a play more. So I'm just going to go ahead and light it up. Um, for enterprise environments, I actually am a huge fan of Jenkins. Um, it's a tool that people really understand. Um, it has stood the test of time, right? I mean, people were using Hudson, but 15 years ago, maybe longer. And, you know, the way it works, hasn't really changed very much. I mean, Jenkins X is a little different, but, um, the UI and the way it works internally is pretty familiar to a lot of enterprise environments, which is great. >>And also in me, the plugin ecosystem is amazing. There's so many plugins for everything, and you can make your own if you know, Java groovy. I'm sure there's a perfect Kotlin in there, but I haven't tried myself, but it's really great. It's also really easy to write, um, CIS code, which is something I'm a big fan of. So Jenkins files have been, have worked really well for me. I, I know that I can get a little bit more complex as you start to build your own models and such, but, you know, for enterprise enterprise CIO CD, if you want, especially if you want to roll your own or own it yourself, um, Jenkins is the bellwether and for very good reason now for my personal projects. And I see a lot on the chat here, I think y'all, y'all been agreed with me get hub actions 100%, my favorite tool right now. >>Um, I love GitHub actions. It's, it's customizable, it's modular. There's a lot of plugins already. I started using getting that back maybe a week after when GA and there was no documentation or anything. And I still, it was still my favorite CIA tool even then. Um, and you know, the API is really great. There's a lot to love about GitHub actions and, um, and I, and I use it as much as I can from my personal project. So I still have a soft spot for Travis CAI. Um, you know, they got acquired and they're a little different now trying to see, I, I can't, I can't let it go. I just love it. But, um, yeah, I mean, when it comes to Seattle, those are my tools. So light me up in the comments I will respond. Yeah. >>I mean, I, I feel with you on the Travis, the, I think, cause I think that was my first time experiencing, you know, early days get hub open source and like a free CIA tool that I could describe. I think it was the ammo back then. I don't actually remember, but yeah, it was kind of an exciting time from my experience. There was like, oh, this is, this is just there as a service. And I could just use it. It doesn't, it's like get hub it's free from my open source stuff. And so it does have a soft spot in my heart too. So yeah. >>All right. We've got questions around, um, cam, so I'm going to ask some questions. We don't have to have these answers because sometimes they're going to be specific, but I want to call them out because people in chat may have missed that question. And there's probably, you know, that we have smart people in chat too. So there's probably someone that knows the answer to these things. If, if it's not us, um, they're asking about building Docker images in Kubernetes, which to me is always a sore spot because it's Kubernetes does not build images by default. It's not meant for that out of the gate. And, uh, what is the best way to do this without having to use privileged containers, which privileged containers just implying that yeah, you, you, it probably has more privileges than by default as a container in Kubernetes. And that is a hard thing because, uh, I don't, I think Docker doesn't lie to do that out of the gate. So I don't know if anyone has an immediate answer to that. That's a pretty technical one, but if you, if you know the answer to that in chat, call it out. >>Um, >>I had done this, uh, but I'm pretty sure I had to use a privileged, um, container and install the Docker Damon on the Kubernetes cluster. And I CA I can't give you a better solution. Um, I've done the same. So, >>Yeah, uh, Chavonne asks, um, back to the Jenkins thing, what's the easiest way to integrate Docker into a Jenkins CICB pipeline. And that's one of the challenges I find with Jenkins because I don't claim to be the expert on Jenkins. Is there are so many plugins because of this, of this such a huge ecosystem. Um, when you go searching for Docker, there's a lot that comes back, right. So I, I don't actually have a preferred way because every team I find uses it differently. Um, I don't know, is there a, do you know if there's a Jenkins preferred, a default plugin? I don't even know for Docker. Oh, go ahead. Yeah. Sorry for Docker. And jacon sorry, Docker plugins for Jenkins. Uh, as someone's asking like the preferred or easy way to do that. Um, and I don't, I don't know the back into Jenkins that well, so, >>Well, th the new, the new way that they're doing, uh, Docker builds with the pipeline, which is more declarative versus the groovy. It's really simple, and their documentation is really good. They, um, they make it really easy to say, run this in this image. So you can pull down, you know, public images and add your own layers. Um, so I don't know the name of that plugin, uh, but I can certainly take a minute after this session and going and get that. Um, but if you really are overwhelmed by the plugins, you can just write your, you know, your shell command in Jenkins. You could just by, you know, doing everything in bash, calling the Docker, um, Damon directly, and then getting it working just to see that end to end, and then start browsing for plugins to see if you even want to use those. >>The plugins will allow more integration from end to end. Some of the things that you input might be available later on in the process for having to manage that yourself. But, you know, you don't have to use any of the plugins. You can literally just, you know, do a block where you write your shell command and get it working, and then decide if, for plugins for you. Um, I think it's always under important to understand what is going on under the hood before you, before you adopt the magic of a plugin, because, um, once you have a problem, if you're, if it's all a lockbox to you, it's going to be more difficult to troubleshoot. It's kind of like learning, get command line versus like get cracking or something. Once, once you get in a bind, if you don't understand the underlying steps, it's really hard to get yourself out of a bind, versus if you understand what the plugin or the app is doing, then, um, you can get out of situations a lot easier. That's a good place. That's, that's where I'd start. >>Yeah. Thank you. Um, Camden asks better to build test environment images, every commit in CII. So this is like one of those opinions of we're all gonna have some different, uh, or build on build images on every commit, leveraging the cash, or build them once outside the test pile pipeline. Um, what say you people? >>Uh, well, I I've seen both and generally speaking, my preference is, um, I guess the ant, the it's a consultant answer, right? I think it depends on what you're trying to do, right. So if you have a lot of small changes that are being made and you're creating images for each of those commits, you're going to have a lot of images in your, in your registry, right? And on top of that, if you're building those images, uh, through CAI frequently, if you're using Docker hub or something like that, you might run into rate limiting issues because of Docker's new rate, limiting, uh, rate limits that they put in place. Um, but that might be beneficial if the, if being able to roll back between those small changes while you're testing is important to you. Uh, however, if all you care about is being able to use Docker images, um, or being able to correlate versions to your Docker images, or if you're the type of team that doesn't even use him, uh, does he even use, uh, virgins in your image tags? Then I would think that that might be a little, much you might want to just have in your CIO. You might want to have a stage that builds your Docker images and Docker image and pushes it into your registry, being done first particular branches instead of having to be done on every commit regardless of branch. But again, it really depends on the team. It really depends on what you're building. It really depends on your workflow. It can depend on a number of things like a curse sometimes too. Yeah. Yeah. >>Once had two points here, you know, I've seen, you know, the pattern has been at every, with every, uh, uh, commit, assuming that you have the right set of tests that would kind of, uh, you would benefit from actually seeing, um, the, the, the, the testing workflow go through and can detect any issue within, within the build or whatever you're trying to test against. But if you're just a building without the appropriate set of tests, then you're just basically consuming almond, adding time, as well as all the, the image, uh, stories associated with it without treaty reaping the benefit of, of, of this pattern. Uh, and the second point is, again, I think if you're, if you're going to end up doing a per commit, uh, definitely recommend having some type of, uh, uh, image purging, um, uh, and, and, and garbage collection process to ensure that you're not just wasting, um, all the stories needed and also, um, uh, optimizing your, your bill process, because that will end up being the most time-consuming, um, um, you know, within, within your pipeline. So this is my 2 cents on this. >>Yeah, that's good stuff. I mean, those are both of those are conversations that could lead us into the rabbit hole for the rest of the day on storage management, uh, you know, CP CPU minutes for, uh, you know, your build stuff. I mean, if you're in any size team, more than one or two people, you immediately run into headaches with cost of CIA, because we have now the problem of tools, right? We have so many tools. We can have the CIS system burning CPU cycles all day, every day, if we really wanted to. And so you re very quickly, I think, especially if you're on every commit on every branch, like that gets you into a world of cost mitigation, and you probably are going to have to settle somewhere in the middle on, uh, between the budget, people that are saying you're spending way too much money on the CII platform, uh, because of all these CPU cycles, and then the developers who would love to have everything now, you know, as fast as possible and the biggest, biggest CPU's, and the biggest servers, and have the bills, because the bills can never go fast enough, right. >>There's no end to optimizing your build workflow. Um, we have another question on that. This is another topic that we'll all probably have different takes on is, uh, basically, uh, version tags, right? So on images, we, we have a very established workflow in get for how we make commits. We have commit shots. We have, uh, you know, we know get tags and there's all these things there. And then we go into images and it's just this whole new world that's opened up. Like there's no real consensus. Um, so what, what are your thoughts on the strategy for teams in their image tag? Again, another, another culture thing. Um, commander, >>I mean, I'm a fan of silver when we have no other option. Um, it's just clean and I like the timestamp, you know, exactly when it was built. Um, I don't really see any reason to use another, uh, there's just normal, incremental, um, you know, numbering, but I love the fact that you can pull any tag and know exactly when it was created. So I'm a big fan of bar, if you can make that work for your organization. >>Yep. People are mentioned that in chat, >>So I like as well. Uh, I'm a big fan of it. I think it's easy to be able to just be as easy to be able to signify what a major changes versus a minor change versus just a hot fix or, you know, some or some kind of a bad fix. The problem that I've found with having teams adopt San Bernardo becomes answering these questions and being able to really define what is a major change, what is a minor change? What is a patch, right? And this becomes a bit of an overhead or not so much of an overhead, but, uh, uh, uh, a large concern for teams who have never done versioning before, or they never been responsible for their own versioning. Um, in fact, you know, I'm running into that right now, uh, with, with a client that I'm working with, where a lot, I'm working with a lot of teams, helping them move their applications from a legacy production environment into a new one. >>And in doing so, uh, versioning comes up because Docker images, uh, have tags and usually the tax correlate to versions, but some teams over there, some teams that I'm working with are only maintaining a script and others are maintaining a fully fledged JAK, three tier application, you know, with lots of dependencies. So telling the script, telling the team that maintains a script, Hey, you know, you should use somber and you should start thinking about, you know, what's major, what's my number what's patch. That might be a lot for them. And for someone or a team like that, I might just suggest using commit shots as your versions until you figure that out, or maybe using, um, dates as your version, but for the more for the team, with the larger application, they probably already know the answers to those questions. In which case they're either already using Sember or they, um, or they may be using some other version of the strategy and might be in December, might suit them better. So, um, you're going to hear me say, it depends a lot, and I'm just going to say here, it depends. Cause it really does. Carlos. >>I think you hit on something interesting beyond just how to version, but, um, when to consider it a major release and who makes those decisions, and if you leave it to engineers to version, you're kind of pushing business decisions down the pipe. Um, I think when it's a minor or a major should be a business decision and someone else needs to make that call someone closer to the business should be making that call as to when we want to call it major. >>That's a really good point. And I add some, I actually agree. Um, I absolutely agree with that. And again, it really depends on the team that on the team and the scope of it, it depends on the scope that they're maintaining, right? And so it's a business application. Of course, you're going to have a product manager and you're going to have, you're going to have a product manager who's going to want to make that call because that version is going to be out in marketing. People are going to use it. They're going to refer to and support calls. They're going to need to make those decisions. Sember again, works really, really well for that. Um, but for a team that's maintaining the scripts, you know, I don't know, having them say, okay, you must tell me what a major version is. It's >>A lot, but >>If they want it to use some birds great too, which is why I think going back to what you originally said, Sember in the absence of other options. I think that's a good strategy. >>Yeah. There's a, there's a, um, catching up on chat. I'm not sure if I'm ever going to catch up, but there's a lot of people commenting on their favorite CII systems and it's, and it, it just goes to show for the, the testing and deployment community. Like how many tools there are out there, how many tools there are to support the tools that you're using. Like, uh, it can be a crazy wilderness. And I think that's, that's part of the art of it, uh, is that these things are allowing us to build our workflows to the team's culture. Um, and, uh, but I do think that, you know, getting into like maybe what we hope to be at what's next is I do hope that we get to, to try to figure out some of these harder problems of consistency. Uh, one of the things that led me to Docker at the beginning to begin with was the fact that it wa it created a consistent packaging solution for me to get my code, you know, off of, off of my site of my local system, really, and into the server. >>And that whole workflow would at least the thing that I was making at each step was going to be the same thing used. Right. And that, that was huge. Uh, it was also, it also took us a long time to get there. Right. We all had to, like Docker was one of those ones that decade kind of ideas of let's solidify the, enter, get the consensus of the community around this idea. And we, and it's not perfect. Uh, you know, the Docker Docker file is not the most perfect way to describe how to make your app, but it is there and we're all using it. And now I'm looking for that next piece, right. Then hopefully the next step in that, um, that where we can all arrive at a consensus so that once you hop teams, you know, okay. We all knew Docker. We now, now we're all starting to get to know the manifests, but then there's this big gap in the middle where it's like, it might be one of a dozen things. Um, you know, so >>Yeah, yeah. To that, to that, Brett, um, you know, uh, just maybe more of a shameless plug here and wanting to kind of talk about one of the things that I'm on. So excited, but I work, I work at Tasha Corp. I don't know anyone, or I don't know if many people have heard of, um, you know, we tend to focus a lot on workflows versus technologies, right. Because, you know, as you can see, even just looking at the chat, there's, you know, ton of opinions on the different tooling, right. And, uh, imagine having, you know, I'm working with clients that have 10,000 developers. So imagine taking the folks in the chat and being partnered with one organization or one company and having to make decisions on how to build software. Um, but there's no way you can conversion one or, or one way or one tool, uh, and that's where we're facing in the industry. >>So one of the things that, uh, I'm pretty excited about, and I don't know if it's getting as much traction as you know, we've been focused on it. This is way point, which is a project, an open source project. I believe we got at least, uh, last year, um, which is, it's more of, uh, it's, it is aim to address that really, uh, uh, Brad set on, you know, to come to tool to, uh, make it extremely easy and simple. And, you know, to describe how you want to build, uh, deploy or release your application, uh, in, in a consistent way, regardless of the tools. So similar to how you can think of Terraform and having that pluggability to say Terraform apply or plan against any cloud infrastructure, uh, without really having to know exactly the details of how to do it, uh, this is what wave one is doing. Um, and it can be applied with, you know, for the CIA, uh, framework. So, you know, task plugability into, uh, you know, circle CEI tests to Docker helm, uh, Kubernetes. So that's the, you know, it's, it's a hard problem to solve, but, um, I'm hopeful that that's the path that we're, you know, we'll, we'll eventually get to. So, um, hope, you know, you can, you can, uh, see some of the, you know, information, data on it, on, on HashiCorp site, but I mean, I'm personally excited about it. >>Yeah. Uh I'm to gonna have to check that out. And, um, I told you on my live show, man, we'll talk about it, but talk about it for a whole hour. Uh, so there's another question here around, uh, this, this is actually a little bit more detailed, but it is one that I think a lot of people deal with and I deal with a lot too, is essentially the question is from Cameron, uh, D essentially, do you use compose in your CIO or not Docker compose? Uh, because yes I do. Yeah. Cause it, it, it, it solves so many problems am and not every CGI can, I don't know, there's some problems with a CIO is trying to do it for me. So there are pros and cons and I feel like I'm still on the fence about it because I use it all the time, but also it's not perfect. It's not always meant for CIA. And CIA sometimes tries to do things for you, like starting things up before you start other parts and having that whole order, uh, ordering problem of things anyway. W thoughts and when have thoughts. >>Yes. I love compose. It's one of my favorite tools of all time. Um, and the reason why it's, because what I often find I'm working with teams trying to actually let me walk that back, because Jack on the chat asked a really interesting question about what, what, what the hardest thing about CIS for a lot of teams. And in my experience, the hardest thing is getting teams to build an app that is the same app as what's built in production. A lot of CGI does things that are totally different than what you would do in your local, in your local dev. And as a result of that, you get, you got this application that either doesn't work locally, or it does work, but it's a completely different animal than what you would get in production. Right? So what I've found in trying to get teams to bridge that gap by basically taking their CGI, shifting the CII left, I hate the shift left turn, but I'll use it. >>I'm shifting the CIO left to your local development is trying to say, okay, how do we build an app? How do we, how do we build mot dependencies of that app so that we can build so that we can test our app? How do we run tests, right? How do we build, how do we get test data? And what I found is that trying to get teams to do all this in Docker, which is normally a first for a lot of teams that I'm working with, trying to get them all to do all of this. And Docker means you're running Docker, build a lot running Docker, run a lot. You're running Docker, RM a lot. You ran a lot of Docker, disparate Docker commands. And then on top of that, trying to bridge all of those containers together into a single network can be challenging without compose. >>So I like using a, to be able to really easily categorize and compartmentalize a lot of the things that are going to be done in CII, like building a Docker image, running tests, which is you're, you're going to do it in CII anyway. So running tests, building the image, pushing it to the registry. Well, I wouldn't say pushing it to the registry, but doing all the things that you would do in local dev, but in the same network that you might have a mock database or a mock S3 instance or some of something else. Um, so it's just easy to take all those Docker compose commands and move them into your Yammel file using the hub actions or your dankest Bob using Jenkins, or what have you. Right. It's really, it's really portable that way, but it doesn't work for every team. You know, for example, if you're just a team that, you know, going back to my script example, if it's a really simple script that does one thing on a somewhat routine basis, then that might be a lot of overhead. Um, in that case, you know, you can get away with just Docker commands. It's not a big deal, but the way I looked at it is if I'm, if I'm building, if I build something that's similar to a make bile or rate file, or what have you, then I'm probably gonna want to use Docker compose. If I'm working with Docker, that's, that's a philosophy of values, right? >>So I'm also a fan of Docker compose. And, um, you know, to your point, Carlos, the whole, I mean, I'm also a fan of shifting CEI lift and testing lift, but if you put all that logic in your CTI, um, it changes the L the local development experience from the CGI experience. Versus if you put everything in a compose file so that what you build locally is the same as what you build in CGI. Um, you're going to have a better experience because you're going to be testing something more, that's closer to what you're going to be releasing. And it's also very easy to look at a compose file and kind of, um, understand what the dependencies are and what's happening is very readable. And once you move that stuff to CGI, I think a lot of developers, you know, they're going to be intimidated by the CGI, um, whatever the scripting language is, it's going to be something they're going to have to wrap their head around. >>Um, but they're not gonna be able to use it locally. You're going to have to have another local solution. So I love the idea of a composed file use locally, um, especially if he can Mount the local workspace so that they can do real time development and see their changes in the exact same way as it's going to be built and tested in CGI. It gives developers a high level of confidence. And then, you know, you're less likely to have issues because of discrepancies between how it was built in your local test environment versus how it's built in NCI. And so Docker compose really lets you do all of that in a way that makes your solution more portable, portable between local dev and CGI and reduces the number of CGI cycles to get, you know, the test, the test data that you need. So that's why I like it for really, for local dev. >>It'll be interesting. Um, I don't know if you all were able to see the keynote, but there was a, there was a little bit, not a whole lot, but a little bit talk of the Docker, compose V two, which has now built into the Docker command line. And so now we're shifting from the Python built compose, which was a separate package. You could that one of the challenges was getting it into your CA solution because if you don't have PIP and you got down on the binary and the binary wasn't available for every platform and, uh, it was a PI installer. It gets a little nerdy into how that works, but, uh, and the team is now getting, be able to get unified with it. Now that it's in Golang and it's, and it's plugged right into the Docker command line, it hopefully will be easier to distribute, easier to, to use. >>And you won't have to necessarily have dependencies inside of where you're running it because there'll be a statically compiled binary. Um, so I've been playing with that, uh, this year. And so like training myself to do Docker going from Docker dash compose to Docker space, compose. It is a thing I I'm almost to the point of having to write a shell replacement. Yeah. Alias that thing. Um, but, um, I'm excited to see what that's going, cause there's already new features in it. And it, these built kit by default, like there's all these things. And I, I love build kit. We could make a whole session on build kit. Um, in fact there's actually, um, maybe going on right now, or right around this time, there is a session on, uh, from Solomon hikes, the seat, uh, co-founder of Docker, former CTO, uh, on build kit using, uh, using some other tool on top of build kit or whatever. >>So that, that would be interesting for those of you that are not watching that one. Cause you're here, uh, to do a check that one out later. Um, all right. So another good question was caching. So another one, another area where there is no wrong answers probably, and everyone has a different story. So the question is, what are your thoughts on CII build caching? There's often a debate between security. This is from Quentin. Thank you for this great question. There's often a debate between security reproducibility and build speeds. I haven't found a good answer so far. I will just throw my hat in the ring and say that the more times you want to build, like if you're trying to build every commit or every commit, if you're building many times a day, the more caching you need. So like the more times you're building, the more caching you're gonna likely want. And in most cases caching doesn't bite you in the butt, but that could be, yeah, we, can we get the bit about that? So, yeah. Yeah. >>I'm going to quote Carlos again and say, it depends on, on, you know, how you're talking, you know, what you're trying to build and I'm quoting your colors. Um, yeah, it's, it's got, it's gonna depend because, you know, there are some instances where you definitely want to use, you know, depends on the frequency that you're building and how you're building. Um, it's you would want to actually take advantage of cashing functionalities, um, for the build, uh, itself. Um, but if, um, you know, as you mentioned, there could be some instances where you would want to disable, um, any caching because you actually want to either pull a new packages or, um, you know, there could be some security, um, uh, disadvantages related to security aspects that would, you know, you know, using a cache version of, uh, image layer, for example, could be a problem. And you, you know, if you have a fleet of build, uh, engines, you don't have a good grasp of where they're being cashed. We would have to, um, disable caching in that, in that, um, in those instances. So it, it would depend. >>Yeah, it's, it's funny you have that problem on both sides of cashing. Like there are things that, especially in Docker world, they will cash automatically. And, and then, and then you maybe don't realize that some of that caching could be bad. It's, it's actually using old, uh, old assets, old artifacts, and then there's times where you would expect it to cash, that it doesn't cash. And then you have to do something extra to enable that caching, especially when you're dealing with that cluster of, of CIS servers. Right. And the cloud, the whole clustering problem with caching is even more complex, but yeah, >>But that's, that's when, >>Uh, you know, ever since I asked you to start using build kits and able to build kit, you know, between it's it's it's reader of Boston in, in detecting word, you know, where in, in the bill process needs to cash, as well as, uh, the, the, um, you know, the process. I don't think I've seen any other, uh, approach there that comes close to how efficient, uh, that process can become how much time it can actually save. Uh, but again, I think, I think that's, for me that had been my default approach, unless I actually need something that I would intentionally to disable caching for that purpose, but the benefits, at least for me, the benefits of, um, how bill kit actually been processing my bills, um, from the builds as well as, you know, using the cash up until, you know, how it detects the, the difference in, in, in the assets within the Docker file had been, um, you know, uh, pretty, you know, outweigh the disadvantages that it brings in. So it, you know, take it each case by case. And based on that, determine if you want to use it, but definitely recommend those enabling >>In the absence of a reason not to, um, I definitely think that it's a good approach in terms of speed. Um, yeah, I say you cash until you have a good reason not to personally >>Catch by default. There you go. I think you catch by default. Yeah. Yeah. And, uh, the trick is, well, one, it's not always enabled by default, especially when you're talking about cross server. So that's a, that's a complexity for your SIS admins, or if you're on the cloud, you know, it's usually just an option. Um, I think it also is this, this veers into a little bit of, uh, the more you cash the in a lot of cases with Docker, like the, from like, if you're from images and checked every single time, if you're not pinning every single thing, if you're not painting your app version, you're at your MPN versions to the exact lock file definition. Like there's a lot of these things where I'm I get, I get sort of, I get very grouchy with teams that sort of let it, just let it all be like, yeah, we'll just build two images and they're totally going to have different dependencies because someone happened to update that thing and after whatever or MPM or, or, and so I get grouchy about that, cause I want to lock it all down, but I also know that that's going to create administrative burden. >>Like the team is now going to have to manage versions in a very much more granular way. Like, do we need to version two? Do we need to care about curl? You know, all that stuff. Um, so that's, that's kind of tricky, but when you get to, when you get to certain version problems, uh, sorry, uh, cashing problems, you, you, you don't want those set those caches to happen because it, if you're from image changes and you're not constantly checking for a new image, and if you're not pinning that V that version, then now you, you don't know whether you're getting the latest version of Davion or whatever. Um, so I think that there's, there's an art form to the more you pen, the less you have, the less, you have to be worried about things changing, but the more you pen, the, uh, all your versions of everything all the way down the stack, the more administrative stuff, because you're gonna have to manually change every one of those. >>So I think it's a balancing act for teams. And as you mature, I to find teams, they tend to pin more until they get to a point of being more comfortable with their testing. So the other side of this argument is if you trust your testing, then you, and you have better testing to me, the less likely to the subtle little differences in versions have to be penned because you can get away with those minor or patch level version changes. If you're thoroughly testing your app, because you're trusting your testing. And this gets us into a whole nother rant, but, uh, yeah, but talking >>About penny versions, if you've got a lot of dependencies isn't that when you would want to use the cash the most and not have to rebuild all those layers. Yeah. >>But if you're not, but if you're not painting to the exact patch version and you are caching, then you're not technically getting the latest versions because it's not checking for all the time. It's a weird, there's a lot of this subtle nuance that people don't realize until it's a problem. And that's part of the, the tricky part of allow this stuff, is it, sometimes the Docker can be almost so much magic out of the box that you, you, you get this all and it all works. And then day two happens and you built it a second time and you've got a new version of open SSL in there and suddenly it doesn't work. Um, so anyway, uh, that was a great question. I've done the question on this, on, uh, from heavy. What do you put, where do you put testing in your pipeline? Like, so testing the code cause there's lots of types of testing, uh, because this pipeline gets longer and longer and Docker building images as part of it. And so he says, um, before staging or after staging, but before production, where do you put it? >>Oh man. Okay. So, um, my, my main thought on this is, and of course this is kind of religious flame bait, so sure. You know, people are going to go into the compensation wrong. Carlos, the boy is how I like to think about it. So pretty much in every stage or every environment that you're going to be deploying your app into, or that your application is going to touch. My idea is that there should be a build of a Docker image that has all your applications coded in, along with its dependencies, there's testing that tests your application, and then there's a deployment that happens into whatever infrastructure there is. Right. So the testing, they can get tricky though. And the type of testing you do, I think depends on the environment that you're in. So if you're, let's say for example, your team and you have, you have a main branch and then you have feature branches that merged into the main branch. >>You don't have like a pre-production branch or anything like that. So in those feature branches, whenever I'm doing CGI that way, I know when I freak, when I cut my poll request, that I'm going to merge into main and everything's going to work in my feature branches, I'm going to want to probably just run unit tests and maybe some component tests, which really, which are just, you know, testing that your app can talk to another component or another part, another dependency, like maybe a database doing tests like that, that don't take a lot of time that are fascinating and right. A lot of would be done at the beach branch level and in my opinion, but when you're going to merge that beach branch into main, as part of a release in that activity, you're going to want to be able to do an integration tasks, to make sure that your app can actually talk to all the other dependencies that it talked to. >>You're going to want to do an end to end test or a smoke test, just to make sure that, you know, someone that actually touches the application, if it's like a website can actually use the website as intended and it meets the business cases and all that, and you might even have testing like performance testing, low performance load testing, or security testing, compliance testing that would want to happen in my opinion, when you're about to go into production with a release, because those are gonna take a long time. Those are very expensive. You're going to have to cut new infrastructure, run those tests, and it can become quite arduous. And you're not going to want to run those all the time. You'll have the resources, uh, builds will be slower. Uh, release will be slower. It will just become a mess. So I would want to save those for when I'm about to go into production. Instead of doing those every time I make a commit or every time I'm merging a feature ranch into a non main branch, that's the way I look at it, but everything does a different, um, there's other philosophies around it. Yeah. >>Well, I don't disagree with your build test deploy. I think if you're going to deploy the code, it needs to be tested. Um, at some level, I mean less the same. You've got, I hate the term smoke tests, cause it gives a false sense of security, but you have some mental minimum minimal amount of tests. And I would expect the developer on the feature branch to add new tests that tested that feature. And that would be part of the PR why those tests would need to pass before you can merge it, merge it to master. So I agree that there are tests that you, you want to run at different stages, but the earlier you can run the test before going to production. Um, the fewer issues you have, the easier it is to troubleshoot it. And I kind of agree with what you said, Carlos, about the longer running tests like performance tests and things like that, waiting to the end. >>The only problem is when you wait until the end to run those performance tests, you kind of end up deploying with whatever performance you have. It's, it's almost just an information gathering. So if you don't run your performance test early on, um, and I don't want to go down a rabbit hole, but performance tests can be really useless if you don't have a goal where it's just information gap, uh, this is, this is the performance. Well, what did you expect it to be? Is it good? Is it bad? They can get really nebulous. So if performance is really important, um, you you're gonna need to come up with some expectations, preferably, you know, set up the business level, like what our SLA is, what our response times and have something to shoot for. And then before you're getting to production. If you have targets, you can test before staging and you can tweak the code before staging and move that performance initiative. Sorry, Carlos, a little to the left. Um, but if you don't have a performance targets, then it's just a check box. So those are my thoughts. I like to test before every deployment. Right? >>Yeah. And you know what, I'm glad that you, I'm glad that you brought, I'm glad that you brought up Escalades and performance because, and you know, the definition of performance says to me, because one of the things that I've seen when I work with teams is that oftentimes another team runs a P and L tests and they ended, and the development team doesn't really have too much insight into what's going on there. And usually when I go to the performance team and say, Hey, how do you run your performance test? It's usually just a generic solution for every single application that they support, which may or may not be applicable to the application team that I'm working with specifically. So I think it's a good, I'm not going to dig into it. I'm not going to dig into the rabbit hole SRE, but it is a good bridge into SRE when you start trying to define what does reliability mean, right? >>Because the reason why you test performance, it's test reliability to make sure that when you cut that release, that customers would go to your site or use your application. Aren't going to see regressions in performance and are not going to either go to another website or, you know, lodge in SLA violation or something like that. Um, it does, it does bridge really well with defining reliability and what SRE means. And when you have, when you start talking about that, that's when you started talking about how often do I run? How often do I test my reliability, the reliability of my application, right? Like, do I have nightly tasks in CGI that ensure that my main branch or, you know, some important branch I does not mean is meeting SLA is meeting SLR. So service level objectives, um, or, you know, do I run tasks that ensure that my SLA is being met in production? >>Like whenever, like do I use, do I do things like game days where I test, Hey, if I turn something off or, you know, if I deploy this small broken code to production and like what happens to my performance? What happens to my security and compliance? Um, you can, that you can go really deep into and take creating, um, into creating really robust tests that cover a lot of different domains. But I liked just using build test deploy is the overall answer to that because I find that you're going to have to build your application first. You're going to have to test it out there and build it, and then you're going to want to deploy it after you test it. And that order generally ensures that you're releasing software. That works. >>Right. Right. Um, I was going to ask one last question. Um, it's going to have to be like a sentence answer though, for each one of you. Uh, this is, uh, do you lint? And if you lint, do you lent all the things, if you do, do you fail the linters during your testing? Yes or no? I think it's going to depend on the culture. I really do. Sorry about it. If we >>Have a, you know, a hook, uh, you know, on the get commit, then theoretically the developer can't get code there without running Melinta anyway, >>So, right, right. True. Anyone else? Anyone thoughts on that? Linting >>Nice. I saw an additional question online thing. And in the chat, if you would introduce it in a multi-stage build, um, you know, I was wondering also what others think about that, like typically I've seen, you know, with multi-stage it's the most common use case is just to produce the final, like to minimize the, the, the, the, the, the image size and produce a final, you know, thin, uh, layout or thin, uh, image. Uh, so if it's not for that, like, I, I don't, I haven't seen a lot of, you know, um, teams or individuals who are actually within a multi-stage build. There's nothing really against that, but they think the number one purpose of doing multi-stage had been just producing the minimalist image. Um, so just wanted to kind of combine those two answers in one, uh, for sure. >>Yeah, yeah, sure. Um, and with that, um, thank you all for the great questions. We are going to have to wrap this up and we could go for another hour if we all had the time. And if Dr. Khan was a 24 hour long event and it didn't sadly, it's not. So we've got to make room for the next live panel, which will be Peter coming on and talking about security with some developer ex security experts. And I wanted to thank again, thank you all three of you for being here real quick, go around the room. Um, uh, where can people reach out to you? I am, uh, at Bret Fisher on Twitter. You can find me there. Carlos. >>I'm at dev Mandy with a Y D E N D Y that's me, um, >>Easiest name ever on Twitter, Carlos and DFW on LinkedIn. And I also have a LinkedIn learning course. So if you check me out on my LinkedIn learning, >>Yeah. I'm at Nicola Quebec. Um, one word, I'll put it in the chat as well on, on LinkedIn, as well as, uh, uh, as well as Twitter. Thanks for having us, Brett. Yeah. Thanks for being here. >>Um, and, and you all stay around. So if you're in the room with us chatting, you're gonna, you're gonna, if you want to go to see the next live panel, I've got to go back to the beginning and do that whole thing, uh, and find the next, because this one will end, but we'll still be in chat for a few minutes. I think the chat keeps going. I don't actually know. I haven't tried it yet. So we'll find out here in a minute. Um, but thanks you all for being here, I will be back a little bit later, but, uh, coming up next on the live stuff is Peter Wood security. Ciao. Bye.

(techno music) >> Live, from Washington DC, it's theCUBE. Covering AWS Public Sector Summit, 2018. Brought to you by Amazon Web Services and its ecosystem partners. >> Okay, welcome back, everyone, this is the live CUBE coverage here in Washington DC for AWS Public Sector Summit 2018. This is the, kind of like the reinvent for Public Sector. I'm John Furrier, f my co-host Stu Miniman, our next guest is Hardik Bhatt, Smart Cities Vertical Lead for Amazon Web Services, been a former CIO, knows the state and local governments cold. This is a very key area around Internet of Things and technology with cloud, because smart cities have to do not only technology roll outs for some of the new capabilities, but all manage some of the societal changes, like self-driving cars and a variety of other things, from instrumenting sensors and traffic lights and video cam ... I mean, this is a little, just a little ... Welcome to theCUBE. >> Thank you very much, John. Good to see you, Stu, good morning. Looking forward to having a great conversation. >> So, smart cities obviously is really hot, but we love it, because it brings life, and work, life, and play together, because we all live in towns, and we live in cities, and the cities provide services to the residents, transportation, sidewalks, and things that we take for granted in the analog world. Now there's a whole digital set of services coming big time. So, are they prepared? (laughs) It used to be buy a mainframe, then move it to a minicomputer, get a Local Area Network, buy some PCs, buy some network tablets, now the cloud's here. What's your assessment of the smart cities landscape for state and local governments? Because it really is something that's on the front burner, in terms of figuring it out. What's the architecture? Lot of questions. What's your, what's the state of the union, if you will, for-- >> You know it has been, like, how the governments have been for many years, right? Governments exist so that they can provide better services, they can provide better quality of life, they can create an environment where businesses thrive, jobs can be created, education can be given, and you can build a workforce and talent, et cetera. And smart cities is just, I'd say, a trend where, you know, you're using multitudes of technology to kind of help the government get its mission accomplished in a smoother, faster, better, cheaper manner. And a lot of times, I've seen, because how smart cities movement started a decade ago, we kind of compare smart cities with the Internet of Things or the sensors, but smart cities is much more than just the IoT, or the Internet of Things, I mean if you're talking about creating a new stream of data that is real-time, whether coming in from sensors, coming from video, you already as a government, I used to be a CIO for the City of Chicago, we used petabytes of data that was already sitting in my data center, and then there's also this whole third-party data. So smart cities is a lot about how do you as a city are aggregating this different sources of data and then making some action from it, so that ultimately, going back to the city's priorities, you are giving better public safety, or you're providing better public health, or you're providing better education or you're providing, better providing government services. So that's what we are seeing. Our customers are, when we say smart cities, they jump right into, "What problems are you solving?" And that, to me, is the core for Amazon, core for Amazon Web Services. We want to know our customers' problems and then work backwards to solve them. >> What are some of the problems right now that are low-hanging fruit? Because obviously it's an evolution. You set the architecture up, but ultimately governments would love to have some revenue coming in from businesses. You mention that. Education is certainly there. What are some of the challenges there? Is it pre-existing stuff, or is it new opportunities? What are some of the trends you're seeing for use cases? It is actually both pre-existing stuff that they are trying to solve, as well the new stuff, the new opportunities that are getting created, because the technology is much different than what it used to be 10 years ago. The cloud, especially, is creating a lot more new opportunities, because of the nimbleness it brings, the agility it brings. So, in transportation side, we are seeing on one hand, multiple departments, multi-jurisdictional, so state transportation department, as well as a local transportation department, working together to create kind of a virtual information sharing environment or a virtual command center, so that they can detect an accident, a traffic incident, much quicker and respond to that, because now they can aggregate this data. And they're also now adding to that some public safety information. So whether it is a police department, fire department, EMS, so that they can address that incident quickly and then not only clear the traffic and clear the congestion, or reduce the congestion time, but they can also address the, any public safety issue that may have arisen out of that incident that has happened. So, the Department of Transportation, the USDOT, through the Federal Highway Administration, has been giving out $60 million worth of grants to six to ten recipients. The grant, this year's grant period, just closed on Monday, and we worked with multiple customers who are looking to kind of respond to that. So on one hand, it is that. So this is an age-old problem, but new technology can help you solve that. On the other hand, another customer that we worked with is looking for on-demand micro-transit solutions. As you can see, all the ride-sharing applications are making easier to jump in a car and move to one place to the other. It is causing a dip in transit ridership. So the public transit agents, they are looking for solutions to that. So they are looking at, "Can we build an on-demand microtransit "so you can pool your friends and jump into a transit van, as opposed to a private car?" And then you can go from point A to point B in a much more affordable manner. So they are looking at that. On the public health side, you know, we have the DC Benefits Exchange, Health Benefits Exchange, is on AWS, and they have seen significant savings. They have seen $1.8 million of annual savings because they are using cloud and cloud services. On the other hand, you have State of Georgia, which is using Alexa. So they have built Alexa Skills where you can ask, as a resident of State of Georgia getting SNAP benefit, the Supplemental Nutritional Assistance, the food-stamp program, you can say, "Alexa, what's my SNAP balance?" So based on the answer then, based on the balance you know, you can plan your, you know, where you're going to use that money. So we are seeing large volume of data now coming on the cloud where the governments are looking to move kind of the needle. We are also seeing this nimble, quick solutions that can start going out. And we are seeing a lot of driver behind the innovation is our City on a Cloud challenge. So we have seen the City on a Cloud winners, since last so many years, are kind of the ones who are driving innovation and they're also driving a lot of collaboration. So I can, there are three trends that I can jump into as we kind of talk more. >> Yeah, it's interesting. I think back a decade ago, when you talk smarter cities, you'd see this video, and it would look like something out of a science fiction. It's like, you know, "Oh, the flying taxi'll come, "and it will get you and everything." But what I, the stories I have when I talk to CIOs in cities and the like, it's usually more about, it's about data. It's about the underlying data, and maybe it's a mobile app, maybe it's a thing like Alexa Skills. So help us understand a little bit, what does the average citizen, what do they see? How does their, you know, greater transparency and sharing of information and collaboration between what the agencies are doing and, you know, the citizenship. >> I think that's a great question. I mean that is what, as a former CIO, I always had to balance between, what I do creates internal government efficiency, but the citizens don't feel it, don't see it, they don't, it doesn't get in the news media. And on the other hand, I also have to, to my governor, to my mayor, to the agency directors, have to give them visible wins. So, I'll give you an example, so City of Chicago, back in the day, in 2010 when I was the CIO. We did a contract with our AWS, currently AWS Partner Socrata, to open up the data. So that was kind of the beginning of the Open Data Movement, and eventually, I left the city, I went work for Cisco, and the city government continued to kind of build on top of Socrata. And they build what they called the Windy Grid, which is basically bringing all of their various sets of data, so 311, code violations, inspections, crime, traffic, and they built an internal data analytics engine. So now, agencies can use that data. And now, what they did, two years ago, they were one of the City on a Cloud Challenge winners, and they, Uturn Data Solutions is our partner that was the winner of that, and they built Chicago Open Grid. So they basically opened that up on a map-based platform. So now as a citizen of Chicago, I can go on Chicago Open Grid, and I can see which restaurants in, surrounding my area, have failed inspections. Have they failed inspection because of a mice infestation, or was it something very minor, so I can decide whether I want to go to that restaurant or not. I can also look at the crime patterns in my area, I can look at the property values, I can look at the education kind of quality in the schools in my neighborhood. So, we have seen kind of now, and it's all on AWS cloud. >> This open data is interesting to me. Let's take that to another level. That's just the user side of it, there's also a delivery value. I saw use cases in Chicago around Health and Human Services, around being more efficient with either vaccines, or delivery of services based on demographics and other profile, all because of open data. So this brings up a question that comes up a lot, and we're seeing here is a trend, is Amazon Web Services public sector has been really good. Teresa Carlson has done an amazing job leaning on partners to be successful. Meaning it's a collaboration. What's that like in the state and local government? What's the partner landscape look like? What are the benefits for partners to work with AWS? Because it seems obvious to me, it might not be obvious to them. But if they have an innovative idea, whether it's to innovate something on the edge of the network in their business, they can do it, and they can scale with Amazon. What is the real benefits of partnering with AWS? >> You hit a key point on there. Teresa has done a fantastic job in customer management as well as building our partners. Similarly, we have a great leader within the state and local government, Kim Majerus. She leads all of our state and local government business. And her focus is exactly like Teresa: How can we help the customers, and also how can we enable partners to help customers? So I'll give you and example. The City of Louisville in Kentucky. They were a City on a Cloud winner, and they, basically what they're building with a partner of ours, Slingshot, they (laughs) get, I was, I used to be in Traffic Management Authority, back in my days, and we used to do traffic studies. So, basically, they send an intern out with clicker or have those black strips to count the number of cars, and based on that, we can plan whether we want to increase the signal timing on this approach, or we can plan the detours if we close the street, what's the, and it's all manual. It used to take, cost us anywhere from 10 to 50 thousand dollars, every traffic study. So what Louisville did with Slingshot is they got the free Waze data that they get gives all of the raw traffic information. Slingshot brought that on to a AWS platform, and now they are building a traffic analysis tool, which now you can do like a snap of a finger, get the analysis and you can manage the signal-approach timing. The cool thing about this is, they're building it in open source code. And the code's available on GitHub, and I was talking to the Chief Data Officer of Louisville, who's actually going to be speaking at this event later today. 12 other cities have already looked into this. They've started to download the code, and they are starting to use it. So, collaboration through partners also enables collaboration amongst all of our customers. >> And also, I'd just point out, that's a great example, love that, and that's new for me to hear that. But also, to me the observation is, it's new data. So being able to be responsive, to look at that opportunity. Now, it used to be in the old world, and I'm sure you can attest to this, being a CIO back in the day, is okay, just say there's new data available, you have to provision IT. >> Oh my God, yeah. >> I mean, what, old way, new way. I mean, compare and contrast the time it would take to do that with what you can do today. >> It's a big, huge difference. I'll tell you as the CIO for the State of Illinois, when I started in early 2015, in my first performance management session, I asked my Infrastructure Management Team to give me the average days it takes to build a server, 49 days. I mean, you're talking seven weeks or maybe, if you talk, 10 business weeks. It's not acceptable. I mean the way the pace of innovation is going, with AWS on cloud, you are talking about minutes you can spin up that server. And that's what we are seeing, a significant change, and that's why Louisville-- >> And I think you got to think it's even worse when you think about integration, personnel requirements, the meetings that have to get involved. It's a nightmare. Okay, so obviously cloud, we know cloud, we love cloud, we use cloud ourselves. So I got to ask you this could, City in a Cloud program, which we've covered in the past, so last year had some really powerful winners. This has been a very successful program. You're involved in it, you have unique insights, you've been on both sides of the table. How is that going? How is it inspiring other cities? What's the camaraderie like? What's the peer review? Is there a peer, is there a network building? How is that spreading? >> That is actually enabling collaboration in a significant manner. Because, you know, you are openly telling what you want to do, and then you are doing that. Everybody is watching you. Like Louisville is a perfect example where they built this, they're building this, and they're going to share it through open source code to all the cities. 12 is just the beginning. I'd not be surprised if there are 120 cities that are going to do this. Because who doesn't want to save two hundred, three hundred thousand dollars a year? And also lots of time to do the traffic studies. Same thing we have seen with, as Virginia Beach is building their Early Flood Warning System. There are other cities who are looking into, like how do we, New Orleans? And others are looking at, "How do we take what Virginia Beach has built? "And how can we use it for us?" And yesterday, they announced this year of the winners that includes Las Vegas, that includes LA Information Technology Department, that includes the City of Philadelphia, and I've been in conversations with all of the CIOs, CDOs, and the leaders of these agencies. The other thing, John, I have seen is, there's a phenomenal leadership that's out there right now in the cities and states that they want to innovate, they want to collaborate, and they want to kind of make a big difference. >> Hold on, hold on, so one more question, this is a really good question, want to get, follow-up on that. But this, what you're talking about to me signifies really the big trend going on right now in this modern era. You've got large cloud scale. You have open source, open sharing, and collaboration happening. This is the new network effect. This is the flywheel. This is uniquely different. This kind of categorizes cloud. And this wasn't available when IT systems and processes were built, 20, 30 years ago. I mean, this is the big shift, you, I mean do you agree? >> Absolutely, this is the big shift, the availability of the cloud, the ubiquitous nature of mobile platform that people have. The newer way of, like, the natural language processing, use of Alexa is becoming so prevalent in government. I mean, in City of Chicago, 50% of the 311 calls that we used to get in 2010, 3 1/2 million of those were informational in nature. If I could offload that on to my Alexa Skills, I can free up my workforce, the 311 call-takers, to do much better, higher-level, you know, call-taking, as opposed to this. So you're absolutely right. I've seen the trends we are seeing is, there is lots of collaboration going on between the governments and partners. I'm also seeing the governments are going at modernization from different points based on their pain points. And I'm also seeing a definite acceleration in modernization. Government, because the technology, AWS, the cloud, our services that we are seeing. And the pace of innovation that AWS brings is also enabling the acceleration in governments. >> Yeah, to help put a point on the, on the conversation here, there's been for years discussion about, "Well, what is the changing role of the CIO?" You've sat on that side of the table, you know, worked with lots of COs, what do you see is the role of the future for the CIO when, specifically when you talk state and local governments? >> I would say CIO is the kind of has to be an enabler of government services. Because if I go back to my city days and working with a mayor, or my state days, working with a governor, at the end of the day, the governor or the mayor is looking at creating better quality of life, providing better health, better education, better safety, et cetera. And CIO is kind of the key partner in that metrics to enable what the governor, what the mayor, the agency directors want to do. And because now data enables the CIO to kind of quickly give solutions, or AI services, Alexa and Polly and Rekog ... All of these things give you, give me as a CIO, ability to provide quick wins to the mayor, to the governor, and also very visible wins. We are seeing that, you know, CIO is becoming a uniquely positioned individual and leader to kind of enable the government. >> All right, thanks so much for comin' on theCUBE. Love the insight, love to follow up. You bring a great perspective and great insight and Amazon's lucky to have you on the team. Lot of great stuff goin' on in the cities and local governments. It's a good opportunity for you guys. Thanks for coming on, appreciate it. >> Thank you very much. >> It's theCUBE live here in Washington DC for AWS, Amazon Web Services Public Sector Summit, I'm John Furrier, Stu Miniman, again second year of live coverage. It's a packed house, a lot of great cloud action. Again, the game has changed. It's a whole new world, cloud scale, open source, collaboration, mobile, all this new data's here. This is the opportunity, this is what theCUBE's doing. We're doin' our part, sharing the data with you. Stay with us, more coverage from day two, here in Washington, after this short break. (techno music)