(upbeat jingle) >> Cameraman: Just look, Mike. >> Good afternoon everyone, and welcome back to Supercomputing. We're live here with theCUBE in Dallas. I'm joined by my cohost, David. Wonderful to be sharing the afternoon with you. And we are going to be kicking things off with a very thrilling discussion from two important thought leaders at Dell. Bhavesh and Shreya, thank you so much for being on the show. Welcome. How you doing? How does it feel to be at Supercomputing? >> Pretty good. We really enjoying the show and enjoying a lot of customer conversations ongoing. >> Yeah. Are most of your customers here? >> Yes. Most of the customers are, mostly in the Hyatt over there and a lot of discussions ongoing. >> Yeah. Must be nice to see everybody show off. Are you enjoying the show so far, Shreya? >> Yeah, I missed this for two years and so it's nice to be back and meeting people in person. >> Yeah, definitely. We all missed it. So, it's been a very exciting week for Dell. Do you want to talk about what you're most excited about in the announcement portfolio that we saw yesterday? >> Absolutely. >> Go for it, Shreya. >> Yeah, so, you know, before we get into the portfolio side of the house, you know, we really wanted to, kind of, share our thoughts, in terms of, you know, what is it that's, kind of, moving HPC and supercomputing, you know, for a long time- >> Stock trends >> For a long time HPC and supercomputing has been driven by packing the racks, you know, maximizing the performance. And as the work that Bhavesh and I have been doing over the last, you know, couple of generations, we're seeing an emerging trend and that is the thermal dissipated power is actually exploding. And so the idea of packing the racks is now turning into, how do you maximize your performance, but are able to deliver the infrastructure in that limited kilowatts per rack that you have in your data center. >> So I, it's been interesting walking around the show seeing how many businesses associated with cooling- >> Savannah: So many. >> are here. And it's funny to see, they open up the cabinet, and it's almost 19th-century-looking technology. It's pipes and pumps and- >> Savannah: And very industrial-like. >> Yeah, very, very industrial-looking. Yeah, and I think, so that's where the, the trends are more in the power and cooling. That is what everybody is trying to solve from an industry perspective. And what we did when we looked at our portfolio, what we want to bring up in this timeframe for targeting more the HPC and AI space. There are a couple of vectors we had to look at. We had to look at cooling, we had to look at power where the trends are happening. We had to look at, what are the data center needs showing up, be it in the cooler space, be it in the HPC space, be it in the large install happening out there. So, looking at those trends and then factoring in, how do you build a node out? We said, okay, we need to diversify and build out an infrastructure. And that's what me and Shreya looked into, not only looking at the silicon diversity showing up, but more looking at, okay, there is this power, there is this cooling, there is silicon diversity. Now, how do you start packing it up and bringing it to the marketplace? So, kind of, those are some of the trends that we captured. And that's what you see, kind of, in the exhibit floor today, even. >> And Dell technology supports both, liquid cooling, air cooling. Do you have a preference? Is it more just a customer-based? >> It is going to be, and Shreya can allude to it, it's more workload and application-focused. That is what we want to be thinking about. And it's not going to be siloed into, okay, is we going to be just targeting air-cooling, we wanted to target a breadth between air to liquid. And that's how we built into our portfolio when we looked at our GPUs. >> To add to that, if we look at our customer landscape, we see that there's a peak between 35 to 45 kilowatts per rack. We see another peak at 60, we see another peak at 80, and we've got selects, you know, very specialized customers above hundred kilowatts per rack. And so, if we take that 35 to 45 kilowatts per rack, you know, you can pack maybe three or four of these chassis, right? And so, to what Bhavesh is saying, we're really trying to provide the flexibility for what our customers can deliver in their data centers. Whether it be at the 35 end where air cooling may make complete sense. As you get above 45 and above, maybe that's the time to pivot to a liquid-cool solution. >> So, you said that there, so there are situations where you could have 90 kilowatts being consumed by a rack of equipment. So, I live in California where we are very, very closely attuned to things like the price for a kilowatt hour of electricity. >> Seriously. >> And I'm kind of an electric car nerd, so, for the folks who really aren't as attuned, 90 kilowatts, that's like over a hundred horsepower. So, think about a hundred horsepower worth of energy being used for compute in one of these racks. It's insane. So, we, you can kind of imagine a layperson can kind of imagine the variables that go into this equation of, you know, how do we, how do we bring the power and get the maximum bang for, per kilowatt hour. But, are there any, are there any kind of interesting odd twists in your equations that you find when you're trying to figure out. Do you have a- >> Yeah, and we, a lot of these trends when we look at it, okay, it's not, we think about it more from a power density that we want to try to go and solve. We are mindful about all the, from an energy perspective where the energy prices are moving. So, what we do is we try to be optimizing right at the node level and how we going to do our liquid-cooling and air cooled infrastructure. So, it's trying to, how do you keep a balance with it? That's what we are thinking about. And thinking about it is not just delivering or consuming the power that is maybe not needed for that particular node itself. So, that's what we are thinking about. The other way we optimize when we built this infrastructure out is we are thinking about, okay, how are we go going to deliver it at the rack level and more keeping in mind as to how this liquid-cooling plumbing will happen. Where is it coming into the data center? Is it coming in the bottom of the floor? Are we going to do it on the left hand side of your rack or the right hand side? It's a big thing. It's like it becomes, okay, yeah, it doesn't matter which side you put it on, but there is a piece of it going into our decision as to how we are going to build that, no doubt. So, there are multiple factors coming in and besides the power and cooling, which we all touched upon, But, Shreya and me also look at is where this whole GPU and accelerators are moving into. So, we're not just looking at the current set of GPUs and where they're moving from a power perspective. We are looking at this whole silicon diversity that is happening out there. So, we've been looking at multiple accelerators. There are multiple companies out there and we can tell you there'll be over three 30 to 50 silicon companies out there that we are actively engaged and looking into. So, our decision in building this particular portfolio out was being mindful about what the maturity curve is from a software point of view. From a hardware point of view and what can we deliver, what the customer really needs in it, yeah. >> It's a balancing act, yeah. >> Bhavesh: It is a balancing act. >> Let's, let's stay in that zone a little bit. What other trends, Shreya, let's go to you on this one. What other trends are you seeing in the acceleration landscape? >> Yeah, I think you know, to your point, the balancing act is actually a very interesting paradigm. One of the things that Bhavesh and I constantly think about, and we call it the Goldilocks syndrome, which is, you know, at that 90 and and a hundred, right? Density matters. >> Savannah: A lot. >> But, what we've done is we have really figured out what that optimal point is, 'cause we don't want to be the thinnest most possible. You lose a lot of power redundancy, you lose a lot of I/O capability, you lose a lot of storage capability. And so, from our portfolio perspective, we've really tried to think about the Goldilocks syndrome and where that sweet spot is. >> I love that. I love the thought of you all just standing around server racks, having a little bit of porridge and determining >> the porridge. Exactly the thickness that you want in terms of the density trade off there. Yeah, that's, I love that, though. I mean it's very digestible. Are you seeing anything else? >> No, I think that's pretty much, Shreya summed it up and we think about what we are thinking about, where the technology features are moving and what we are thinking, in terms of our portfolio, so it is, yeah. >> So, just a lesson, you know, Shreya, a lesson for us, a rudimentary lesson. You put power into a CPU or a GPU and you're getting something out and a lot of what we get out is heat. Is there a measure, is there an objective measure of efficiency in these devices that we look at? Because you could think of a 100 watt light bulb, an incandescent light bulb is going to give out a certain amount of light and a certain amount of heat. A 100 watt equivalent led, in terms of the lumens that it's putting out, in terms of light, a lot more light for the power going in, a lot less heat. We have led lights around us, thankfully, instead of incandescent lights. >> Savannah: Otherwise we would be melting. >> But, what is, when you put power into a CPU or a GPU, how do you measure that efficiency? 'Cause it's sort of funny, 'cause it's like, it's not moving, so it's not like measuring, putting power into a vehicle and measuring forward motion and heat. You're measuring this, sort of, esoteric thing, this processing thing that you can't see or touch. But, I mean, how much per watt of power, how do you, how do you measure it I guess? Help us out, from the base up understanding, 'cause people generally, most people have never been in a data center before. Maybe they've put their hand behind the fan in a personal computer or they've had a laptop feel warm on their lap. But, we're talking about massive amounts of heat being generated. Can you, kind of, explain the fundamentals of that? >> So, the way we think about it is, you know, there's a performance per dollar metric. There's a performance per dollar per watt metric and that's where the power kind of comes in. But, on the flip side, we have something called PUE, power utilization efficiency from a data center aspect. And so, we try to marry up those concepts together and really try to find that sweet spot. >> Is there anything in the way of harvesting that heat to do other worthwhile work, I mean? >> Yes. >> You know, it's like, hey, everybody that works in the data center, you all have your own personal shower now, water heated. >> Recirculating, too. >> Courtesy of Intel AMD. >> Or a heated swimming pool. >> Right, a heated swimming pool. >> I like the pool. >> So, that's the circulation of, or recycling of that thermal heat that you're talking about, absolutely. And we see that our customers in the, you know, in the Europe region, actually a lot more advanced in terms of taking that power and doing something that's valuable with it, right? >> Cooking croissant and, and making lattes, probably right? >> (laughing) Or heating your home. >> Makes me want to go on >> vacation, a pool, croissants. >> That would be a good use. But, do you, it's more on the PUE aspect of it. It's more thinking about how are we more energy efficient in our design, even, so we are more thinking about what's the best efficiency we can get, but what's the amount of heat capture we can get? Are we just kind of wasting any heat out there? So, that's always the goal when designing these particular platforms, so that's something that we had kept in mind with a lot of our power and cooling experts within Dell. When thinking about, okay, is it, how much can we get, can we capture? If we are not capturing anything, then what are we, kind of, recirculating it back in order to get much better efficiency when we think about it at a rack level and for the other equipment which is going to be purely air-cooled out there and what can we do about it, so. >> Do you think both of these technologies are going to continue to work in tandem, air cooling and liquid cooling? Yeah, so we're not going to see- >> Yeah, we don't, kind of, when we think about our portfolio and what we see the trends moving in the future, I think so, air-cooling is definitely going to be there. There'll be a huge amount of usage for customers looking into air-cooling. Air-cooling is not going to go away. Liquid-cooling is definitely something that a lot of customers are looking into adopting. PUE become the bigger factor for it. How much can I heat capture with it? That's a bigger equation that is coming into the picture. And that's where we said, okay, we have a transition happening. And that's what you see in our portfolio now. >> Yeah, Intel is, Intel, excuse me, Dell is agnostic when it comes to things like Intel, AMD, Broadcom, Nvidia. So, you can look at this landscape and I think make a, you know, make a fair judgment. When we talk about GPU versus CPU, in terms of efficiency, do you see that as something that will live on into the future for some applications? Meaning look, GPU is the answer or is it simply a question of leveraging what we think of as CPU cores differently? Is this going to be, is this going to ebb and flow back and forth? Shreya, are things going to change? 'Cause right now, a lot of what's announced recently, in the high performance computer area, leverages GPUs. But, we're right in the season of AMD and Intel coming out with NextGen processor architectures. >> Savannah: Great point. >> Shreya: Yeah >> Any thoughts? >> Yeah, so what I'll tell you is that it is all application dependent. If you rewind, you know, a couple of generations you'll see that the journey for GPU just started, right? And so there is an ROI, a minimum threshold ROI that customers have to realize in order to move their workloads from CPU-based to GPU-based. As the technology evolves and matures, you'll have more and more applications that will fit within that bucket. Does that mean that everything will fit in that bucket? I don't believe so, but as, you know, the technology will continue to mature on the CPU side, but also on the GPU side. And so, depending on where the customer is in their journey, it's the same for air versus liquid. Liquid is not an if, but it's a when. And when the environment, the data center environment is ready to support that, and when you have that ROI that goes with it is when it makes sense to transition to one way or the other. >> That's awesome. All right, last question for you both in a succinct phrase, if possible, I won't character count. What do you hope that we get to talk about next year when we have you back on theCUBE? Shreya, we'll start with you. >> Ooh, that's a good one. I'm going to let Bhavesh go first. >> Savannah: Go for it. >> (laughs) >> What do you think, Bhavesh? Next year, I think so, what you'll see more, because I'm in the CTI group, more talking about where cache coherency is moving. So, that's what, I'll just leave it at that and we'll talk about it more. >> Savannah: All right. >> Dave: Tantalizing. >> I was going to say, a little window in there, yeah. And I think, to kind of add to that, I'm excited to see what the future holds with CPUs, GPUs, smart NICs and the integration of these technologies and where that all is headed and how that helps ultimately, you know, our customers being able to solve these really, really large and complex problems. >> The problems our globe faces. Wow, well it was absolutely fantastic to have you both on the show. Time just flew. David, wonderful questions, as always. Thank you all for tuning in to theCUBE. Here live from Dallas where we are broadcasting all about supercomputing, high-performance computing, and everything that a hardware nerd, like I, loves. My name is Savannah Peterson. We'll see you again soon. (upbeat jingle)

>>Hey, welcome to the live panel. My name is Brett. I am your host, and indeed we are live. In fact, if you're curious about that, if you don't believe us, um, let's just show a little bit of the browser real quick to see. Yup. There you go. We're live. So, all right. So how this is going to work is I'm going to bring in some guests and, uh, in one second, and we're going to basically take your questions on the topic designer of the day, that continuous integration testing. Uh, thank you so much to my guests welcoming into the panel. I've got Carlos, Nico and Mandy. Hello everyone. >>Hello? All right, >>Let's go. Let's go around the room and all pretend we don't know each other and that the internet didn't read below the video who we are. Uh, hi, my name is Brett. I am a Docker captain, which means I'm supposed to know something about Docker. I'm coming from Virginia Beach. I'm streaming here from Virginia Beach, Virginia, and, uh, I make videos on the internet and courses on you to me, Carlos. Hey, >>Hey, what's up? I'm Carlos Nunez. I am a solutions architect, VMware. I do solution things with computers. It's fun. I live in Dallas when I'm moving to Houston in a month, which is where I'm currently streaming. I've been all over the Northeast this whole week. So, um, it's been fun and I'm excited to meet with all of you and talk about CIA and Docker. Sure. >>Yeah. Hey everyone. Uh, Nico, Khobar here. I'm a solution engineer at HashiCorp. Uh, I am streaming to you from, uh, the beautiful Austin, Texas. Uh, ignore, ignore the golden gate bridge here. This is from my old apartment in San Francisco. Uh, just, uh, you know, keeping that, to remember all the good days, um, that that lived at. But, uh, anyway, I work at Patrick Corp and I work on all things, automation, um, and cloud and dev ops. Um, and I'm excited to be here and Mandy, >>Hi. Yeah, Mandy Hubbard. I am streaming from Austin, Texas. I am, uh, currently a DX engineer at ship engine. Um, I've worked in QA and that's kind of where I got my, uh, my Docker experience and, um, uh, moving into DX to try and help developers better understand and use our products and be an advocate for them. >>Nice. Well, thank you all for joining me. Uh, I really appreciate you taking the time out of your busy schedule to be here. And so for those of you in chat, the reason we're doing this live, because it's always harder to do things live. The reason we're here is to answer a question. So we didn't come with a bunch of slides and demos or anything like that. We're here to talk amongst ourselves about ideas and really here for you. So we've, we obviously, this is about easy CII, so we're, we're going to try to keep the conversation around testing and continuous integration and all the things that that entails with containers. But we may, we may go down rabbit holes. We may go veer off and start talking about other things, and that's totally fine if it's in the realm of dev ops and containers and developer and ops workflows, like, Hey, it's, it's kinda game. >>And, uh, these people have a wide variety of expertise. They haven't done just testing, right? We, we live in a world where you all kind of have to wear many hats. So feel free to, um, ask what you think is on the top of your mind. And we'll do our best to answer. It may, might not be the best answer or the correct answer, but we're going to do our best. Um, well, let's get it start off. Uh, let's, let's get a couple of topics to start off with. Uh, th the, the easy CGI was my, one of my three ideas. Cause he's the, one of the things that I'm most excited about is the innovation we're seeing around easier testing, faster testing, automated testing, uh, because as much as we've all been doing this stuff for, you know, 15 years, since 20 years since the sort of Jenkins early days, um, it it's, it seems like it's still really hard and it's still a lot of work. >>So, um, let's go around the room real quick, and everybody can just kind of talk for a minute about like your experience with testing and maybe some of your pain points, like what you don't like about our testing world. Um, and we can talk about some pains, cause I think that will lead us to kind of talk about what, what are the things we're seeing now that might be better, uh, ideas about how to do this. I know for me, uh, testing, obviously there's the code part, but just getting it automated, but mostly getting it in the hands of developers so that they can control their own testing. And don't have to go talk to a person to run that test again, or the mysterious Jenkins platform somewhere. I keep mentioning Jenkins cause it's, it is still the dominant player out there. Um, so for me, I'm, I'm, I, I don't like it when I'm walking into a room and there's, there's only one or two people that know how the testing works or know how to make the new tests go into the testing platform and stuff like that. So I'm always trying to free those things so that any of the developers are enabled and empowered to do that stuff. So someone else, Carlos, anybody, um, >>Oh, I have a lot of opinions on that. Having been a QA engineer for most of my career. Um, the shift that we're saying is everyone is dev ops and everyone is QA. Th the issue I see is no one asked developers if they wanted to be QA. Um, and so being the former QA on the team, when there's a problem, even though I'm a developer and we're all running QA, they always tend to come to the one of the former QA engineers. And they're not really owning that responsibility and, um, and digging in. So that's kind of what I'm saying is that we're all expected to test now. And some people, well, some people don't know how it's, uh, for me it was kind of an intuitive skill. It just kind of fit with my personality, but not knowing what to look for, not knowing what to automate, not even understanding how your API end points are used by your front end to know what to test when a change is made. It's really overwhelming for developers. And, um, we're going to need to streamline that and, and hold their hands a little bit until they get their feet wet with also being QA. >>Right. Right. So, um, uh, Carlos, >>Yeah, uh, testing is like, Tesla is one of my favorite subjects to talk about when I'm baring with developers. And a lot of it is because of what Mandy said, right? Like a lot of developers now who used to write a test and say, Hey, QA, go. Um, I wrote my unit tests. Now write the rest of the test. Essentially. Now developers are expected to be able to understand how testing, uh, testing methodologies work, um, in their local environments, right? Like they're supposed to understand how to write an integration tasks federate into and tasks, a component test. And of course, how to write unit tests that aren't just, you know, assert true is true, right? Like more comprehensive, more comprehensive, um, more high touch unit tests, which include things like mocking and stubbing and spine and all that stuff. And, you know, it's not so much getting those tests. Well, I've had a lot of challenges with developers getting those tests to run in Docker because of usually because of dependency hell, but, um, getting developers to understand how to write tests that matter and mean something. Um, it's, it's, it can be difficult, but it's also where I find a lot of the enjoyment of my work comes into play. So yeah. I mean, that's the difficulty I've seen around testing. Um, big subject though. Lots to talk about there. >>Yeah. We've got, we've already got so many questions coming in. You already got an hour's worth of stuff. So, uh, Nico 81st thoughts on that? >>Yeah, I think I definitely agree with, with other folks here on the panel, I think from a, um, the shift from a skillset perspective that's needed to adopt the new technologies, but I think from even from, uh, aside from the organizational, um, and kind of key responsibilities that, that the new developers have to kinda adapt to and, and kind of inherit now, um, there's also from a technical perspective as there's, you know, um, more developers are owning the full stack, including the infrastructure piece. So that adds a lot more to the plate in Tim's oaf, also testing that component that they were not even, uh, responsible for before. Um, and, um, also the second challenge that, you know, I'm seeing is that on, you know, the long list of added, um, uh, tooling and, you know, there's new tool every other day. Um, and, um, that kind of requires more customization to the testing, uh, that each individual team, um, any individual developer Y by extension has to learn. Uh, so the customization, uh, as well as the, kind of the scope that had, uh, you know, now in conferences, the infrastructure piece, um, uh, both of act to the, to the challenges that we're seeing right now for, um, for CGI and overall testing, um, uh, the developers are saying, uh, in, in the market today. >>Yeah. We've got a lot of questions, um, about all the, all the different parts of this. So, uh, let me just go straight to them. Cause that's why we're here is for the people, uh, a lot of people asking about your favorite tools and in one of this is one of the challenges with integration, right? Is, um, there is no, there are dominant players, but there, there is such a variety. I mean, every one of my customers seems like they're using a different workflow and a different set of tools. So, and Hey, we're all here to just talk about what we're, what we're using, uh, you know, whether your favorite tools. So like a lot of the repeated questions are, what are your favorite tools? Like if you could create it from scratch, uh, what would you use? Pierre's asking, you know, GitHub actions sounds like they're a fan of GitHub actions, uh, w you know, mentioning, pushing the ECR and Docker hub and, uh, using vs code pipeline, I guess there may be talking about Azure pipelines. Um, what, what's your preferred way? So, does anyone have any, uh, thoughts on that anyone want to throw out there? Their preferred pipeline of tooling? >>Well, I have to throw out mine. I might as Jenkins, um, like kind of a honorary cloud be at this point, having spoken a couple of times there, um, all of the plugins just make the functionality. I don't love the UI, but I love that it's been around so long. It has so much community support, and there are so many plugins so that if you want to do something, you don't have to write the code it's already been tested. Um, unfortunately I haven't been able to use Jenkins in, uh, since I joined ship engine, we, most of our, um, our, our monolithic core application is, is team city. It's a dotnet application and TeamCity plays really well with.net. Um, didn't love it, uh, Ms. Jenkins. And I'm just, we're just starting some new initiatives that are using GitHub actions, and I'm really excited to learn, to learn those. I think they have a lot of the same functionality that you're looking for, but, um, much more simplified in is right there and get hubs. So, um, the integration is a lot more seamless, but I do have to go on record that my favorite CICT tools Jenkins. >>All right. You heard it here first people. All right. Anyone else? You're muted? I'm muted. Carlin says muted. Oh, Carla says, guest has muted themselves to Carlos. You got to unmute. >>Yes. I did mute myself because I was typing a lot, trying to, you know, try to answer stuff in the chat. And there's a lot of really dark stuff in there. That's okay. Two more times today. So yeah, it's fine. Yeah, no problem. So totally. And it's the best way to start a play more. So I'm just going to go ahead and light it up. Um, for enterprise environments, I actually am a huge fan of Jenkins. Um, it's a tool that people really understand. Um, it has stood the test of time, right? I mean, people were using Hudson, but 15 years ago, maybe longer. And, you know, the way it works, hasn't really changed very much. I mean, Jenkins X is a little different, but, um, the UI and the way it works internally is pretty familiar to a lot of enterprise environments, which is great. >>And also in me, the plugin ecosystem is amazing. There's so many plugins for everything, and you can make your own if you know, Java groovy. I'm sure there's a perfect Kotlin in there, but I haven't tried myself, but it's really great. It's also really easy to write, um, CIS code, which is something I'm a big fan of. So Jenkins files have been, have worked really well for me. I, I know that I can get a little bit more complex as you start to build your own models and such, but, you know, for enterprise enterprise CIO CD, if you want, especially if you want to roll your own or own it yourself, um, Jenkins is the bellwether and for very good reason now for my personal projects. And I see a lot on the chat here, I think y'all, y'all been agreed with me get hub actions 100%, my favorite tool right now. >>Um, I love GitHub actions. It's, it's customizable, it's modular. There's a lot of plugins already. I started using getting that back maybe a week after when GA and there was no documentation or anything. And I still, it was still my favorite CIA tool even then. Um, and you know, the API is really great. There's a lot to love about GitHub actions and, um, and I, and I use it as much as I can from my personal project. So I still have a soft spot for Travis CAI. Um, you know, they got acquired and they're a little different now trying to see, I, I can't, I can't let it go. I just love it. But, um, yeah, I mean, when it comes to Seattle, those are my tools. So light me up in the comments I will respond. Yeah. >>I mean, I, I feel with you on the Travis, the, I think, cause I think that was my first time experiencing, you know, early days get hub open source and like a free CIA tool that I could describe. I think it was the ammo back then. I don't actually remember, but yeah, it was kind of an exciting time from my experience. There was like, oh, this is, this is just there as a service. And I could just use it. It doesn't, it's like get hub it's free from my open source stuff. And so it does have a soft spot in my heart too. So yeah. >>All right. We've got questions around, um, cam, so I'm going to ask some questions. We don't have to have these answers because sometimes they're going to be specific, but I want to call them out because people in chat may have missed that question. And there's probably, you know, that we have smart people in chat too. So there's probably someone that knows the answer to these things. If, if it's not us, um, they're asking about building Docker images in Kubernetes, which to me is always a sore spot because it's Kubernetes does not build images by default. It's not meant for that out of the gate. And, uh, what is the best way to do this without having to use privileged containers, which privileged containers just implying that yeah, you, you, it probably has more privileges than by default as a container in Kubernetes. And that is a hard thing because, uh, I don't, I think Docker doesn't lie to do that out of the gate. So I don't know if anyone has an immediate answer to that. That's a pretty technical one, but if you, if you know the answer to that in chat, call it out. >>Um, >>I had done this, uh, but I'm pretty sure I had to use a privileged, um, container and install the Docker Damon on the Kubernetes cluster. And I CA I can't give you a better solution. Um, I've done the same. So, >>Yeah, uh, Chavonne asks, um, back to the Jenkins thing, what's the easiest way to integrate Docker into a Jenkins CICB pipeline. And that's one of the challenges I find with Jenkins because I don't claim to be the expert on Jenkins. Is there are so many plugins because of this, of this such a huge ecosystem. Um, when you go searching for Docker, there's a lot that comes back, right. So I, I don't actually have a preferred way because every team I find uses it differently. Um, I don't know, is there a, do you know if there's a Jenkins preferred, a default plugin? I don't even know for Docker. Oh, go ahead. Yeah. Sorry for Docker. And jacon sorry, Docker plugins for Jenkins. Uh, as someone's asking like the preferred or easy way to do that. Um, and I don't, I don't know the back into Jenkins that well, so, >>Well, th the new, the new way that they're doing, uh, Docker builds with the pipeline, which is more declarative versus the groovy. It's really simple, and their documentation is really good. They, um, they make it really easy to say, run this in this image. So you can pull down, you know, public images and add your own layers. Um, so I don't know the name of that plugin, uh, but I can certainly take a minute after this session and going and get that. Um, but if you really are overwhelmed by the plugins, you can just write your, you know, your shell command in Jenkins. You could just by, you know, doing everything in bash, calling the Docker, um, Damon directly, and then getting it working just to see that end to end, and then start browsing for plugins to see if you even want to use those. >>The plugins will allow more integration from end to end. Some of the things that you input might be available later on in the process for having to manage that yourself. But, you know, you don't have to use any of the plugins. You can literally just, you know, do a block where you write your shell command and get it working, and then decide if, for plugins for you. Um, I think it's always under important to understand what is going on under the hood before you, before you adopt the magic of a plugin, because, um, once you have a problem, if you're, if it's all a lockbox to you, it's going to be more difficult to troubleshoot. It's kind of like learning, get command line versus like get cracking or something. Once, once you get in a bind, if you don't understand the underlying steps, it's really hard to get yourself out of a bind, versus if you understand what the plugin or the app is doing, then, um, you can get out of situations a lot easier. That's a good place. That's, that's where I'd start. >>Yeah. Thank you. Um, Camden asks better to build test environment images, every commit in CII. So this is like one of those opinions of we're all gonna have some different, uh, or build on build images on every commit, leveraging the cash, or build them once outside the test pile pipeline. Um, what say you people? >>Uh, well, I I've seen both and generally speaking, my preference is, um, I guess the ant, the it's a consultant answer, right? I think it depends on what you're trying to do, right. So if you have a lot of small changes that are being made and you're creating images for each of those commits, you're going to have a lot of images in your, in your registry, right? And on top of that, if you're building those images, uh, through CAI frequently, if you're using Docker hub or something like that, you might run into rate limiting issues because of Docker's new rate, limiting, uh, rate limits that they put in place. Um, but that might be beneficial if the, if being able to roll back between those small changes while you're testing is important to you. Uh, however, if all you care about is being able to use Docker images, um, or being able to correlate versions to your Docker images, or if you're the type of team that doesn't even use him, uh, does he even use, uh, virgins in your image tags? Then I would think that that might be a little, much you might want to just have in your CIO. You might want to have a stage that builds your Docker images and Docker image and pushes it into your registry, being done first particular branches instead of having to be done on every commit regardless of branch. But again, it really depends on the team. It really depends on what you're building. It really depends on your workflow. It can depend on a number of things like a curse sometimes too. Yeah. Yeah. >>Once had two points here, you know, I've seen, you know, the pattern has been at every, with every, uh, uh, commit, assuming that you have the right set of tests that would kind of, uh, you would benefit from actually seeing, um, the, the, the, the testing workflow go through and can detect any issue within, within the build or whatever you're trying to test against. But if you're just a building without the appropriate set of tests, then you're just basically consuming almond, adding time, as well as all the, the image, uh, stories associated with it without treaty reaping the benefit of, of, of this pattern. Uh, and the second point is, again, I think if you're, if you're going to end up doing a per commit, uh, definitely recommend having some type of, uh, uh, image purging, um, uh, and, and, and garbage collection process to ensure that you're not just wasting, um, all the stories needed and also, um, uh, optimizing your, your bill process, because that will end up being the most time-consuming, um, um, you know, within, within your pipeline. So this is my 2 cents on this. >>Yeah, that's good stuff. I mean, those are both of those are conversations that could lead us into the rabbit hole for the rest of the day on storage management, uh, you know, CP CPU minutes for, uh, you know, your build stuff. I mean, if you're in any size team, more than one or two people, you immediately run into headaches with cost of CIA, because we have now the problem of tools, right? We have so many tools. We can have the CIS system burning CPU cycles all day, every day, if we really wanted to. And so you re very quickly, I think, especially if you're on every commit on every branch, like that gets you into a world of cost mitigation, and you probably are going to have to settle somewhere in the middle on, uh, between the budget, people that are saying you're spending way too much money on the CII platform, uh, because of all these CPU cycles, and then the developers who would love to have everything now, you know, as fast as possible and the biggest, biggest CPU's, and the biggest servers, and have the bills, because the bills can never go fast enough, right. >>There's no end to optimizing your build workflow. Um, we have another question on that. This is another topic that we'll all probably have different takes on is, uh, basically, uh, version tags, right? So on images, we, we have a very established workflow in get for how we make commits. We have commit shots. We have, uh, you know, we know get tags and there's all these things there. And then we go into images and it's just this whole new world that's opened up. Like there's no real consensus. Um, so what, what are your thoughts on the strategy for teams in their image tag? Again, another, another culture thing. Um, commander, >>I mean, I'm a fan of silver when we have no other option. Um, it's just clean and I like the timestamp, you know, exactly when it was built. Um, I don't really see any reason to use another, uh, there's just normal, incremental, um, you know, numbering, but I love the fact that you can pull any tag and know exactly when it was created. So I'm a big fan of bar, if you can make that work for your organization. >>Yep. People are mentioned that in chat, >>So I like as well. Uh, I'm a big fan of it. I think it's easy to be able to just be as easy to be able to signify what a major changes versus a minor change versus just a hot fix or, you know, some or some kind of a bad fix. The problem that I've found with having teams adopt San Bernardo becomes answering these questions and being able to really define what is a major change, what is a minor change? What is a patch, right? And this becomes a bit of an overhead or not so much of an overhead, but, uh, uh, uh, a large concern for teams who have never done versioning before, or they never been responsible for their own versioning. Um, in fact, you know, I'm running into that right now, uh, with, with a client that I'm working with, where a lot, I'm working with a lot of teams, helping them move their applications from a legacy production environment into a new one. >>And in doing so, uh, versioning comes up because Docker images, uh, have tags and usually the tax correlate to versions, but some teams over there, some teams that I'm working with are only maintaining a script and others are maintaining a fully fledged JAK, three tier application, you know, with lots of dependencies. So telling the script, telling the team that maintains a script, Hey, you know, you should use somber and you should start thinking about, you know, what's major, what's my number what's patch. That might be a lot for them. And for someone or a team like that, I might just suggest using commit shots as your versions until you figure that out, or maybe using, um, dates as your version, but for the more for the team, with the larger application, they probably already know the answers to those questions. In which case they're either already using Sember or they, um, or they may be using some other version of the strategy and might be in December, might suit them better. So, um, you're going to hear me say, it depends a lot, and I'm just going to say here, it depends. Cause it really does. Carlos. >>I think you hit on something interesting beyond just how to version, but, um, when to consider it a major release and who makes those decisions, and if you leave it to engineers to version, you're kind of pushing business decisions down the pipe. Um, I think when it's a minor or a major should be a business decision and someone else needs to make that call someone closer to the business should be making that call as to when we want to call it major. >>That's a really good point. And I add some, I actually agree. Um, I absolutely agree with that. And again, it really depends on the team that on the team and the scope of it, it depends on the scope that they're maintaining, right? And so it's a business application. Of course, you're going to have a product manager and you're going to have, you're going to have a product manager who's going to want to make that call because that version is going to be out in marketing. People are going to use it. They're going to refer to and support calls. They're going to need to make those decisions. Sember again, works really, really well for that. Um, but for a team that's maintaining the scripts, you know, I don't know, having them say, okay, you must tell me what a major version is. It's >>A lot, but >>If they want it to use some birds great too, which is why I think going back to what you originally said, Sember in the absence of other options. I think that's a good strategy. >>Yeah. There's a, there's a, um, catching up on chat. I'm not sure if I'm ever going to catch up, but there's a lot of people commenting on their favorite CII systems and it's, and it, it just goes to show for the, the testing and deployment community. Like how many tools there are out there, how many tools there are to support the tools that you're using. Like, uh, it can be a crazy wilderness. And I think that's, that's part of the art of it, uh, is that these things are allowing us to build our workflows to the team's culture. Um, and, uh, but I do think that, you know, getting into like maybe what we hope to be at what's next is I do hope that we get to, to try to figure out some of these harder problems of consistency. Uh, one of the things that led me to Docker at the beginning to begin with was the fact that it wa it created a consistent packaging solution for me to get my code, you know, off of, off of my site of my local system, really, and into the server. >>And that whole workflow would at least the thing that I was making at each step was going to be the same thing used. Right. And that, that was huge. Uh, it was also, it also took us a long time to get there. Right. We all had to, like Docker was one of those ones that decade kind of ideas of let's solidify the, enter, get the consensus of the community around this idea. And we, and it's not perfect. Uh, you know, the Docker Docker file is not the most perfect way to describe how to make your app, but it is there and we're all using it. And now I'm looking for that next piece, right. Then hopefully the next step in that, um, that where we can all arrive at a consensus so that once you hop teams, you know, okay. We all knew Docker. We now, now we're all starting to get to know the manifests, but then there's this big gap in the middle where it's like, it might be one of a dozen things. Um, you know, so >>Yeah, yeah. To that, to that, Brett, um, you know, uh, just maybe more of a shameless plug here and wanting to kind of talk about one of the things that I'm on. So excited, but I work, I work at Tasha Corp. I don't know anyone, or I don't know if many people have heard of, um, you know, we tend to focus a lot on workflows versus technologies, right. Because, you know, as you can see, even just looking at the chat, there's, you know, ton of opinions on the different tooling, right. And, uh, imagine having, you know, I'm working with clients that have 10,000 developers. So imagine taking the folks in the chat and being partnered with one organization or one company and having to make decisions on how to build software. Um, but there's no way you can conversion one or, or one way or one tool, uh, and that's where we're facing in the industry. >>So one of the things that, uh, I'm pretty excited about, and I don't know if it's getting as much traction as you know, we've been focused on it. This is way point, which is a project, an open source project. I believe we got at least, uh, last year, um, which is, it's more of, uh, it's, it is aim to address that really, uh, uh, Brad set on, you know, to come to tool to, uh, make it extremely easy and simple. And, you know, to describe how you want to build, uh, deploy or release your application, uh, in, in a consistent way, regardless of the tools. So similar to how you can think of Terraform and having that pluggability to say Terraform apply or plan against any cloud infrastructure, uh, without really having to know exactly the details of how to do it, uh, this is what wave one is doing. Um, and it can be applied with, you know, for the CIA, uh, framework. So, you know, task plugability into, uh, you know, circle CEI tests to Docker helm, uh, Kubernetes. So that's the, you know, it's, it's a hard problem to solve, but, um, I'm hopeful that that's the path that we're, you know, we'll, we'll eventually get to. So, um, hope, you know, you can, you can, uh, see some of the, you know, information, data on it, on, on HashiCorp site, but I mean, I'm personally excited about it. >>Yeah. Uh I'm to gonna have to check that out. And, um, I told you on my live show, man, we'll talk about it, but talk about it for a whole hour. Uh, so there's another question here around, uh, this, this is actually a little bit more detailed, but it is one that I think a lot of people deal with and I deal with a lot too, is essentially the question is from Cameron, uh, D essentially, do you use compose in your CIO or not Docker compose? Uh, because yes I do. Yeah. Cause it, it, it, it solves so many problems am and not every CGI can, I don't know, there's some problems with a CIO is trying to do it for me. So there are pros and cons and I feel like I'm still on the fence about it because I use it all the time, but also it's not perfect. It's not always meant for CIA. And CIA sometimes tries to do things for you, like starting things up before you start other parts and having that whole order, uh, ordering problem of things anyway. W thoughts and when have thoughts. >>Yes. I love compose. It's one of my favorite tools of all time. Um, and the reason why it's, because what I often find I'm working with teams trying to actually let me walk that back, because Jack on the chat asked a really interesting question about what, what, what the hardest thing about CIS for a lot of teams. And in my experience, the hardest thing is getting teams to build an app that is the same app as what's built in production. A lot of CGI does things that are totally different than what you would do in your local, in your local dev. And as a result of that, you get, you got this application that either doesn't work locally, or it does work, but it's a completely different animal than what you would get in production. Right? So what I've found in trying to get teams to bridge that gap by basically taking their CGI, shifting the CII left, I hate the shift left turn, but I'll use it. >>I'm shifting the CIO left to your local development is trying to say, okay, how do we build an app? How do we, how do we build mot dependencies of that app so that we can build so that we can test our app? How do we run tests, right? How do we build, how do we get test data? And what I found is that trying to get teams to do all this in Docker, which is normally a first for a lot of teams that I'm working with, trying to get them all to do all of this. And Docker means you're running Docker, build a lot running Docker, run a lot. You're running Docker, RM a lot. You ran a lot of Docker, disparate Docker commands. And then on top of that, trying to bridge all of those containers together into a single network can be challenging without compose. >>So I like using a, to be able to really easily categorize and compartmentalize a lot of the things that are going to be done in CII, like building a Docker image, running tests, which is you're, you're going to do it in CII anyway. So running tests, building the image, pushing it to the registry. Well, I wouldn't say pushing it to the registry, but doing all the things that you would do in local dev, but in the same network that you might have a mock database or a mock S3 instance or some of something else. Um, so it's just easy to take all those Docker compose commands and move them into your Yammel file using the hub actions or your dankest Bob using Jenkins, or what have you. Right. It's really, it's really portable that way, but it doesn't work for every team. You know, for example, if you're just a team that, you know, going back to my script example, if it's a really simple script that does one thing on a somewhat routine basis, then that might be a lot of overhead. Um, in that case, you know, you can get away with just Docker commands. It's not a big deal, but the way I looked at it is if I'm, if I'm building, if I build something that's similar to a make bile or rate file, or what have you, then I'm probably gonna want to use Docker compose. If I'm working with Docker, that's, that's a philosophy of values, right? >>So I'm also a fan of Docker compose. And, um, you know, to your point, Carlos, the whole, I mean, I'm also a fan of shifting CEI lift and testing lift, but if you put all that logic in your CTI, um, it changes the L the local development experience from the CGI experience. Versus if you put everything in a compose file so that what you build locally is the same as what you build in CGI. Um, you're going to have a better experience because you're going to be testing something more, that's closer to what you're going to be releasing. And it's also very easy to look at a compose file and kind of, um, understand what the dependencies are and what's happening is very readable. And once you move that stuff to CGI, I think a lot of developers, you know, they're going to be intimidated by the CGI, um, whatever the scripting language is, it's going to be something they're going to have to wrap their head around. >>Um, but they're not gonna be able to use it locally. You're going to have to have another local solution. So I love the idea of a composed file use locally, um, especially if he can Mount the local workspace so that they can do real time development and see their changes in the exact same way as it's going to be built and tested in CGI. It gives developers a high level of confidence. And then, you know, you're less likely to have issues because of discrepancies between how it was built in your local test environment versus how it's built in NCI. And so Docker compose really lets you do all of that in a way that makes your solution more portable, portable between local dev and CGI and reduces the number of CGI cycles to get, you know, the test, the test data that you need. So that's why I like it for really, for local dev. >>It'll be interesting. Um, I don't know if you all were able to see the keynote, but there was a, there was a little bit, not a whole lot, but a little bit talk of the Docker, compose V two, which has now built into the Docker command line. And so now we're shifting from the Python built compose, which was a separate package. You could that one of the challenges was getting it into your CA solution because if you don't have PIP and you got down on the binary and the binary wasn't available for every platform and, uh, it was a PI installer. It gets a little nerdy into how that works, but, uh, and the team is now getting, be able to get unified with it. Now that it's in Golang and it's, and it's plugged right into the Docker command line, it hopefully will be easier to distribute, easier to, to use. >>And you won't have to necessarily have dependencies inside of where you're running it because there'll be a statically compiled binary. Um, so I've been playing with that, uh, this year. And so like training myself to do Docker going from Docker dash compose to Docker space, compose. It is a thing I I'm almost to the point of having to write a shell replacement. Yeah. Alias that thing. Um, but, um, I'm excited to see what that's going, cause there's already new features in it. And it, these built kit by default, like there's all these things. And I, I love build kit. We could make a whole session on build kit. Um, in fact there's actually, um, maybe going on right now, or right around this time, there is a session on, uh, from Solomon hikes, the seat, uh, co-founder of Docker, former CTO, uh, on build kit using, uh, using some other tool on top of build kit or whatever. >>So that, that would be interesting for those of you that are not watching that one. Cause you're here, uh, to do a check that one out later. Um, all right. So another good question was caching. So another one, another area where there is no wrong answers probably, and everyone has a different story. So the question is, what are your thoughts on CII build caching? There's often a debate between security. This is from Quentin. Thank you for this great question. There's often a debate between security reproducibility and build speeds. I haven't found a good answer so far. I will just throw my hat in the ring and say that the more times you want to build, like if you're trying to build every commit or every commit, if you're building many times a day, the more caching you need. So like the more times you're building, the more caching you're gonna likely want. And in most cases caching doesn't bite you in the butt, but that could be, yeah, we, can we get the bit about that? So, yeah. Yeah. >>I'm going to quote Carlos again and say, it depends on, on, you know, how you're talking, you know, what you're trying to build and I'm quoting your colors. Um, yeah, it's, it's got, it's gonna depend because, you know, there are some instances where you definitely want to use, you know, depends on the frequency that you're building and how you're building. Um, it's you would want to actually take advantage of cashing functionalities, um, for the build, uh, itself. Um, but if, um, you know, as you mentioned, there could be some instances where you would want to disable, um, any caching because you actually want to either pull a new packages or, um, you know, there could be some security, um, uh, disadvantages related to security aspects that would, you know, you know, using a cache version of, uh, image layer, for example, could be a problem. And you, you know, if you have a fleet of build, uh, engines, you don't have a good grasp of where they're being cashed. We would have to, um, disable caching in that, in that, um, in those instances. So it, it would depend. >>Yeah, it's, it's funny you have that problem on both sides of cashing. Like there are things that, especially in Docker world, they will cash automatically. And, and then, and then you maybe don't realize that some of that caching could be bad. It's, it's actually using old, uh, old assets, old artifacts, and then there's times where you would expect it to cash, that it doesn't cash. And then you have to do something extra to enable that caching, especially when you're dealing with that cluster of, of CIS servers. Right. And the cloud, the whole clustering problem with caching is even more complex, but yeah, >>But that's, that's when, >>Uh, you know, ever since I asked you to start using build kits and able to build kit, you know, between it's it's it's reader of Boston in, in detecting word, you know, where in, in the bill process needs to cash, as well as, uh, the, the, um, you know, the process. I don't think I've seen any other, uh, approach there that comes close to how efficient, uh, that process can become how much time it can actually save. Uh, but again, I think, I think that's, for me that had been my default approach, unless I actually need something that I would intentionally to disable caching for that purpose, but the benefits, at least for me, the benefits of, um, how bill kit actually been processing my bills, um, from the builds as well as, you know, using the cash up until, you know, how it detects the, the difference in, in, in the assets within the Docker file had been, um, you know, uh, pretty, you know, outweigh the disadvantages that it brings in. So it, you know, take it each case by case. And based on that, determine if you want to use it, but definitely recommend those enabling >>In the absence of a reason not to, um, I definitely think that it's a good approach in terms of speed. Um, yeah, I say you cash until you have a good reason not to personally >>Catch by default. There you go. I think you catch by default. Yeah. Yeah. And, uh, the trick is, well, one, it's not always enabled by default, especially when you're talking about cross server. So that's a, that's a complexity for your SIS admins, or if you're on the cloud, you know, it's usually just an option. Um, I think it also is this, this veers into a little bit of, uh, the more you cash the in a lot of cases with Docker, like the, from like, if you're from images and checked every single time, if you're not pinning every single thing, if you're not painting your app version, you're at your MPN versions to the exact lock file definition. Like there's a lot of these things where I'm I get, I get sort of, I get very grouchy with teams that sort of let it, just let it all be like, yeah, we'll just build two images and they're totally going to have different dependencies because someone happened to update that thing and after whatever or MPM or, or, and so I get grouchy about that, cause I want to lock it all down, but I also know that that's going to create administrative burden. >>Like the team is now going to have to manage versions in a very much more granular way. Like, do we need to version two? Do we need to care about curl? You know, all that stuff. Um, so that's, that's kind of tricky, but when you get to, when you get to certain version problems, uh, sorry, uh, cashing problems, you, you, you don't want those set those caches to happen because it, if you're from image changes and you're not constantly checking for a new image, and if you're not pinning that V that version, then now you, you don't know whether you're getting the latest version of Davion or whatever. Um, so I think that there's, there's an art form to the more you pen, the less you have, the less, you have to be worried about things changing, but the more you pen, the, uh, all your versions of everything all the way down the stack, the more administrative stuff, because you're gonna have to manually change every one of those. >>So I think it's a balancing act for teams. And as you mature, I to find teams, they tend to pin more until they get to a point of being more comfortable with their testing. So the other side of this argument is if you trust your testing, then you, and you have better testing to me, the less likely to the subtle little differences in versions have to be penned because you can get away with those minor or patch level version changes. If you're thoroughly testing your app, because you're trusting your testing. And this gets us into a whole nother rant, but, uh, yeah, but talking >>About penny versions, if you've got a lot of dependencies isn't that when you would want to use the cash the most and not have to rebuild all those layers. Yeah. >>But if you're not, but if you're not painting to the exact patch version and you are caching, then you're not technically getting the latest versions because it's not checking for all the time. It's a weird, there's a lot of this subtle nuance that people don't realize until it's a problem. And that's part of the, the tricky part of allow this stuff, is it, sometimes the Docker can be almost so much magic out of the box that you, you, you get this all and it all works. And then day two happens and you built it a second time and you've got a new version of open SSL in there and suddenly it doesn't work. Um, so anyway, uh, that was a great question. I've done the question on this, on, uh, from heavy. What do you put, where do you put testing in your pipeline? Like, so testing the code cause there's lots of types of testing, uh, because this pipeline gets longer and longer and Docker building images as part of it. And so he says, um, before staging or after staging, but before production, where do you put it? >>Oh man. Okay. So, um, my, my main thought on this is, and of course this is kind of religious flame bait, so sure. You know, people are going to go into the compensation wrong. Carlos, the boy is how I like to think about it. So pretty much in every stage or every environment that you're going to be deploying your app into, or that your application is going to touch. My idea is that there should be a build of a Docker image that has all your applications coded in, along with its dependencies, there's testing that tests your application, and then there's a deployment that happens into whatever infrastructure there is. Right. So the testing, they can get tricky though. And the type of testing you do, I think depends on the environment that you're in. So if you're, let's say for example, your team and you have, you have a main branch and then you have feature branches that merged into the main branch. >>You don't have like a pre-production branch or anything like that. So in those feature branches, whenever I'm doing CGI that way, I know when I freak, when I cut my poll request, that I'm going to merge into main and everything's going to work in my feature branches, I'm going to want to probably just run unit tests and maybe some component tests, which really, which are just, you know, testing that your app can talk to another component or another part, another dependency, like maybe a database doing tests like that, that don't take a lot of time that are fascinating and right. A lot of would be done at the beach branch level and in my opinion, but when you're going to merge that beach branch into main, as part of a release in that activity, you're going to want to be able to do an integration tasks, to make sure that your app can actually talk to all the other dependencies that it talked to. >>You're going to want to do an end to end test or a smoke test, just to make sure that, you know, someone that actually touches the application, if it's like a website can actually use the website as intended and it meets the business cases and all that, and you might even have testing like performance testing, low performance load testing, or security testing, compliance testing that would want to happen in my opinion, when you're about to go into production with a release, because those are gonna take a long time. Those are very expensive. You're going to have to cut new infrastructure, run those tests, and it can become quite arduous. And you're not going to want to run those all the time. You'll have the resources, uh, builds will be slower. Uh, release will be slower. It will just become a mess. So I would want to save those for when I'm about to go into production. Instead of doing those every time I make a commit or every time I'm merging a feature ranch into a non main branch, that's the way I look at it, but everything does a different, um, there's other philosophies around it. Yeah. >>Well, I don't disagree with your build test deploy. I think if you're going to deploy the code, it needs to be tested. Um, at some level, I mean less the same. You've got, I hate the term smoke tests, cause it gives a false sense of security, but you have some mental minimum minimal amount of tests. And I would expect the developer on the feature branch to add new tests that tested that feature. And that would be part of the PR why those tests would need to pass before you can merge it, merge it to master. So I agree that there are tests that you, you want to run at different stages, but the earlier you can run the test before going to production. Um, the fewer issues you have, the easier it is to troubleshoot it. And I kind of agree with what you said, Carlos, about the longer running tests like performance tests and things like that, waiting to the end. >>The only problem is when you wait until the end to run those performance tests, you kind of end up deploying with whatever performance you have. It's, it's almost just an information gathering. So if you don't run your performance test early on, um, and I don't want to go down a rabbit hole, but performance tests can be really useless if you don't have a goal where it's just information gap, uh, this is, this is the performance. Well, what did you expect it to be? Is it good? Is it bad? They can get really nebulous. So if performance is really important, um, you you're gonna need to come up with some expectations, preferably, you know, set up the business level, like what our SLA is, what our response times and have something to shoot for. And then before you're getting to production. If you have targets, you can test before staging and you can tweak the code before staging and move that performance initiative. Sorry, Carlos, a little to the left. Um, but if you don't have a performance targets, then it's just a check box. So those are my thoughts. I like to test before every deployment. Right? >>Yeah. And you know what, I'm glad that you, I'm glad that you brought, I'm glad that you brought up Escalades and performance because, and you know, the definition of performance says to me, because one of the things that I've seen when I work with teams is that oftentimes another team runs a P and L tests and they ended, and the development team doesn't really have too much insight into what's going on there. And usually when I go to the performance team and say, Hey, how do you run your performance test? It's usually just a generic solution for every single application that they support, which may or may not be applicable to the application team that I'm working with specifically. So I think it's a good, I'm not going to dig into it. I'm not going to dig into the rabbit hole SRE, but it is a good bridge into SRE when you start trying to define what does reliability mean, right? >>Because the reason why you test performance, it's test reliability to make sure that when you cut that release, that customers would go to your site or use your application. Aren't going to see regressions in performance and are not going to either go to another website or, you know, lodge in SLA violation or something like that. Um, it does, it does bridge really well with defining reliability and what SRE means. And when you have, when you start talking about that, that's when you started talking about how often do I run? How often do I test my reliability, the reliability of my application, right? Like, do I have nightly tasks in CGI that ensure that my main branch or, you know, some important branch I does not mean is meeting SLA is meeting SLR. So service level objectives, um, or, you know, do I run tasks that ensure that my SLA is being met in production? >>Like whenever, like do I use, do I do things like game days where I test, Hey, if I turn something off or, you know, if I deploy this small broken code to production and like what happens to my performance? What happens to my security and compliance? Um, you can, that you can go really deep into and take creating, um, into creating really robust tests that cover a lot of different domains. But I liked just using build test deploy is the overall answer to that because I find that you're going to have to build your application first. You're going to have to test it out there and build it, and then you're going to want to deploy it after you test it. And that order generally ensures that you're releasing software. That works. >>Right. Right. Um, I was going to ask one last question. Um, it's going to have to be like a sentence answer though, for each one of you. Uh, this is, uh, do you lint? And if you lint, do you lent all the things, if you do, do you fail the linters during your testing? Yes or no? I think it's going to depend on the culture. I really do. Sorry about it. If we >>Have a, you know, a hook, uh, you know, on the get commit, then theoretically the developer can't get code there without running Melinta anyway, >>So, right, right. True. Anyone else? Anyone thoughts on that? Linting >>Nice. I saw an additional question online thing. And in the chat, if you would introduce it in a multi-stage build, um, you know, I was wondering also what others think about that, like typically I've seen, you know, with multi-stage it's the most common use case is just to produce the final, like to minimize the, the, the, the, the, the image size and produce a final, you know, thin, uh, layout or thin, uh, image. Uh, so if it's not for that, like, I, I don't, I haven't seen a lot of, you know, um, teams or individuals who are actually within a multi-stage build. There's nothing really against that, but they think the number one purpose of doing multi-stage had been just producing the minimalist image. Um, so just wanted to kind of combine those two answers in one, uh, for sure. >>Yeah, yeah, sure. Um, and with that, um, thank you all for the great questions. We are going to have to wrap this up and we could go for another hour if we all had the time. And if Dr. Khan was a 24 hour long event and it didn't sadly, it's not. So we've got to make room for the next live panel, which will be Peter coming on and talking about security with some developer ex security experts. And I wanted to thank again, thank you all three of you for being here real quick, go around the room. Um, uh, where can people reach out to you? I am, uh, at Bret Fisher on Twitter. You can find me there. Carlos. >>I'm at dev Mandy with a Y D E N D Y that's me, um, >>Easiest name ever on Twitter, Carlos and DFW on LinkedIn. And I also have a LinkedIn learning course. So if you check me out on my LinkedIn learning, >>Yeah. I'm at Nicola Quebec. Um, one word, I'll put it in the chat as well on, on LinkedIn, as well as, uh, uh, as well as Twitter. Thanks for having us, Brett. Yeah. Thanks for being here. >>Um, and, and you all stay around. So if you're in the room with us chatting, you're gonna, you're gonna, if you want to go to see the next live panel, I've got to go back to the beginning and do that whole thing, uh, and find the next, because this one will end, but we'll still be in chat for a few minutes. I think the chat keeps going. I don't actually know. I haven't tried it yet. So we'll find out here in a minute. Um, but thanks you all for being here, I will be back a little bit later, but, uh, coming up next on the live stuff is Peter Wood security. Ciao. Bye.

>>Fly from San Francisco. It's the cube covering RSA conference, 2020 San Francisco brought to you by Silicon angled medias. >>Hey, welcome back everybody. Jeff Frick here with the queue. We are wrapping up a Wednesday here at RSA 2020. Again, it's like 50,000 people. This is a huge conference. Everyone has got anything to do with cybersecurity is here. Uh, it's the biggest show, uh, that we cover outside I think of reinvent. So we're excited to have our next guest. She's been on the cube many times but never. And in her current role, she's Pam Murphy, the new CEO of Imperva. Pam, great to see you and congratulations on your new law, on your new gig. Thank you very much. Second month, your second month. So tell us about, you know, kind of what attracted you to the opportunity, you know, kinda, you haven't been there a whole long time, but what's, uh, what's kind of your first impression now that you've been at it for a couple of days? >>I, I'm extremely impressed. It really is. Uh, how would you describe it? Like a dark horse or sort of like the biggest kept secret. So in terms of my previous roles, as you know, we've, you've interviewed me many times, but I've always been in software vendors who basically build applications and sort of build to build databases. Um, and I guess for the last five to eight years it's been all about rebuilt again rearchitecting applications for the cloud. Right. Um, and through that I've managed dev op functions and CCL functions. And so I've been on the consuming side of security. Um, so it's always been a very, you know, area that interested me, Greg Lee, um, as a consumer, you know, obviously the landscape was very much changing. Um, and so I decided to jump over to the other side, right. And lead a company that created and delivered cybersecurity solutions. >>So, uh, so it's been awesome. As I said, month two, uh, Imperva has just amazing products. Um, I didn't quite know when I took the job exactly everything that had had, but when I came over and saw it, it was really working very hard over the last couple of years to acquire new products. And also build and innovate new solutions, uh, to have such a complete AppSec and dataset set of solutions today. I mean, I think, I can't see anybody else in the market right now that has as complete a solution covering ups and data stack that we have. So it's, it's been a really fun time. Um, I must say, you know, it's, uh, it's got a great culture as well. Um, there people have sort of a purpose and sort of, you know, have a feel that they'd be great responsibility sort of making great solutions, which really protect our customer's data and their applications. So it's been really cool. >>No, I saw it on the website. You know, the values are very clearly stated right up front and uh, it's a really important ones. But before we go deeper there, I want to kind of take you back to your old role from a, from a buyer of these services. Because as I, as I walk around the floor here, there are so many vendors, right? Big and small, established and new. So for when you were in your other role and now you, it'll be a great thing for you now that you're on this side of the house, how did you think about sorting it all out? How do you, you know, kind of keep up with, you know, the trusted and true, but yet, you know, kind of the new and innovative in this massive sea of vendors and technologies? >>Totally at one of the things that customers have been saying to me since I came to Imperva is they want a partnership from us because as you rightly said, we're in a sea of loads of vendors, a lot of whom claim to do the same, the same thing effectively. And it's becoming, and I found the same thing when I was on the other side. There is such a sea of clutter right now. It's really hard to sort of find your way through, um, costumers and like myself and my former role, you want fewer vendors, um, and you want to have more complete and integrated solutions. Uh, that's what I wanted and my former role. And that's really what I'm focused on now at Imperva is on the customer side of things. Um, making solutions easier to consume. Um, showing them the breadth of what we have, frankly speaking so that they don't have to go to other solutions. >>I mean your worst nightmare is going to a customer and finding out that you had a, B and C and they didn't realize that you actually had it. So from that perspective, I am bringing the voice of the customer with me from my previous role. It's been echoed and what I'm hearing from our customers now in terms of where they want to see us go and do. Um, so that's really what we're focused on is just doing a better job of giving customers more integrated solutions. Because, you know, as you said, the threat landscape right now, it's becoming really complex. Um, very much automated. Um, you know, in terms of automated attacks, I think by talking to my team this morning, we think based on the data we're seeing right now that bad bots are probably making up like 30% of web traffic right now. Yeah. Yeah. I mean it's getting really hard. Right? And that's in terms of, you know, what they do around account scraping, ATO, um, spam in terms of all the damage that that could do right to you as a customer. So that's what we're focused on. We're focused on, and again, it's bringing from my former old, what do customers need rather than what software companies or tech companies or security companies think that they need. Right. >>Such a good spot. Cause you were in that buyer's seat, you know, just a short, long, short time ago. Cause the other thing you've seen and where you guys applied across a lot of apps in your old space was AI and machine learning and really the power of that apply to lots of different challenges, opportunities and really changing the game now. Now you're fighting against those, those same forces that are being much more sophisticated in their, in their attacks. So when you, when you sit with the team and you look at kind of the evolution of AI, you look at the evolution of 5g and all the IOT connectivity that's going to happen in the increased vulnerabilities. Um, where do you see kind of the solution evolving? Is it just a constant, you know, kind of grind and trying to keep up? Or are there some big strategic things that you see now that you've been here for whatever, all the 60 days? Um, to kind of take advantage of these opportunities. >>So we have this, uh, we call it a threat research group within the company. And their job is to take all the data from the sensors we have. I mean, we have, we look at about 25 petabytes of data every day. All our solutions are cloud solutions as well as on-prem. So we get the benefit of basically seeing all the datas that are hitting our customers every day. I mean, we block a bed 1 million attacks every minutes, like every minute, basically every minute, right? We protect over 3 million databases and you know, we've mitigated some of the largest DDoSs, um, attacks that's ever been reported. So we have a lot of data, right, that we're seen. And the interesting thing is that you're right, we are having to always, we're using that threat research data to see what's happening, how the threat threat landscape is changing there for guiding us on how we need to augment an add to our products to prevent that. >>But interestingly, we're also consuming AI and machine learning as well on our products because we're able to use those solutions to actually do a lot of attack analytics and do a lot of predictive and research for our customers that can kind of guide them about, you know, where things are happening. Because what's happening is that before a lot of the talks were just, um, sort of fast and furious, now we're seeing a pattern towards snow, snow, and continuous, if that makes sense. And so we're seeing all these patterns and threats coming in. Uh, so we're fighting against those technologies like AI, but we're also using those technologies to help us soon, you know, decide where we need to continue to, to add capabilities to stop it. You know, the whole bad box thing wasn't a problem right. A number of years ago. And so it's, it's ever changing in our world, which frankly speaking makes it an interesting place to be because who wants to be in a static, >>in a boring place, no boring here. So another kind of interesting thing about this, this particular industry is the coopertition, you know, kind of aspects to it where there is a lot of sharing across competitors on information when there is some new new type of threat or new kind of threat pattern. So it's a little bit different than, than just a pure competition because there is a, a shared benefit in sharing some of this late breaking news. I don't know if you've started to get into to some of that or had an instant, yeah, it's probably a little bit early, but that's, that's a unique trade I think. >>No, it is for sure. And we make all of our data publicly available. If you go to our website, you look at the CTI index whereby we literally index what we, you know, see the level as being and we're providing all of this data. I mean we get that from our own sensors, but obviously we pull it as well from other third party data sources as well and bring it all together. Um, you know, T to hide that and not make it available to everyone would be would be would be just a very bad thing. Um, for us we are, and I, I'm still trying to find someone, but in terms of most of the vendors out there focus on pieces of apps or pieces of data where we've got both combined, right? Which gives us a huge closed loop advantage of being able to mesh that data together and see the full track record of what's happening from the data from the, from the application down to the data on back again. So that's a benefit that we have that literally we're taking great advantage of right now because in other cases, our competition is sort of point solution based, right? For every one of the best of breed solutions that we have. Right, >>right. It always goes back to the data, right? I mean it's always about the data. >>That's the thing. I mean at the end of the day, uh, why, why is all these things happening? HEOS and attacks and spamming. It's your, as you said, it's to get to the data. And that's why we say we protect data and all paths leading to it because fundamentally that's what customers care about, right? >>Right. So it's crazy. The date is the business and the date is what you're protecting and the business. All right, so put you on the spot. So what are some of your kind of top priorities, you know, kind of out of the gate, they brought you in, you're all excited, you see this great team and opportunity. You know, what are some of the things if we sit down a year from now or maybe six months of black hat that you, uh, that you've got on your plate that you're working on? >>So I think innovation will always be, you know, first and foremost, um, we have Gardner magic quadrant and Forester leading edge products. But in this industry, you need to be paranoid. You always need to be staying ahead. So from an innovation perspective, that's where we're focused. We're working on a lot of cool stuff which we'll be rolling out through the rest of the year. Um, platform as well is really important. I mentioned that we have the unique advantage of having a huge amount of data at the application level and also at the database level and that's allowing us to give use cases and value back to our customers that they don't have right now from any other vendor. So we're working with customers on, on getting that done. Um, I think as well, just purely in terms of, um, publicizing what we have. Right. I think we could do a, I found a lot of things right coming to Imperva and I feel we didn't communicate exactly, exactly. So I think there's a lot of capabilities that we're going to do, um, a lot in terms of publicizing them this year. So there's a lot of really, really cool stuff happening and uh, you know, great momentum going on in the company. Right. >>Well, uh, well, good for them for getting you there. Very fortunate to have you, uh, have you on board. Alright. Right. Well, thanks for taking a few minutes and again, congratulations on your new role. We really look forward to watching this story unfold. All. Alright. Geez, Pam, I'm Jeff. You're watching the cube where at R S a 2020 fear. We're supposed to know everything with the benefit of hindsight, but we're still learning. Thanks for watching. We'll see you next time.

>>From our studios in the heart of Silicon Valley, Palo Alto, California. This is a cute conversation. >>Hi and welcome to the cube studios in Palo Alto, California for another cube conversation where we go in depth with the tech leaders driving innovation across the technology industry. I'm your host, Peter Burris. Well, it's that time of quarter again. Every quarter we get together with Fortinet to discuss their threat landscape report, which is one of the industry's best and most comprehensive views into how the bad guys are utilizing bad software and bad access to compromise digital business and steal digital assets. Now, this quarter's report suggests that there's not as much new stuff going on. If you look at the numbers, they're relatively flat compared to previous quarters, but that doesn't tell the real story. Underneath those numbers, we see that there is a churn. There's an incredibly dynamic world of bad actors doing bad things with old and new bad stuff to try to compromise digital business, to learn more about this dynamism and what's really happening. Once again, we've got a great cube guest, Tony Gian. Medico is a senior security strategist and researcher and CTI lead at Fordanet. Tony, welcome back to the cube. >>Hey Peter, it's great to be here. >>So Tony, I started off by making this observation that the index suggests that we're in kind of a steady state, but that's not really what's happening. Is it? What's really going on? Where it's going on inside the numbers? >>Yeah, no, we start to see a little bit of a shift of tactics. Um, what has happened, I think, uh, not all the time, but sometimes with the adversaries like to do is penetrate an organization where maybe us as defenders aren't necessarily as focused in on, and a great example is this. For many years we were focused on and rightfully so. And we continue to be focused on this is being able to block a phishing email, right? We have our email security gateways to be able to not allow that email to come into the network. We also then for for whatever reason, if it happens to get into the network, we focus on user awareness training to educate our users to make sure that they can identify a malicious email. They're not clicking that link or clicking that attachment. Now with that said, we look at the actual data in our queue three threat last grade and what we're seeing is the adversaries are targeting vulnerabilities that if they were successfully exploited would give them remote code execution, meaning that they, they, they can compromise that box and then move further and further inside the network. >>Now granted that's been happening for many years, but we have actually seen an increase order. As a matter of fact, it was number one prevalence across all the actual regions. So with that said, I think it's worth making sure that you're looking at your edge devices or your edge services that are publicly exposed out there. Make sure that there's no vulnerabilities on them, make sure that they're not misconfigured and also make sure that you have some type of multifactor authentication. And I think like we've talked about many times that threat landscape or that, you know, threat attack surface continues really to expand, right? You've got, you've got cloud, you have IOT. So it's becoming more and more difficult to be able to secure all those edge services. Definitely. You know, something you should take a look at >>and you got more people using more mobile devices to do more things. So, so it sounds as though it's a combination of two things. It's really driving this dynamism, right, Tony? It's one, just the raw numbers of growth and devices and opportunities and the threat surface is getting larger and the possibility that something's misconfigured is going up and to that they're just trying to catch your organization's by surprise. One of those is just make sure you're doing things right, but the other one is don't keep, take your eye off the ball, isn't it? How are organizations doing as they try to, uh, expand their ability to address all of these different issues, including a bunch that are tried and true and mature, uh, that we may have stopped focusing on? >>Yeah. You know, it's really hard, right? I always say this and um, you know, I get some mixed kind of reaction sometimes, but you can't protect and monitor everything. I mean, depending on how large your network is, it's really difficult. So I mean, really focusing on what's important, what's critical in your organization is probably really the best approach, right? Really kind of focusing on that. Now with that said though, the reason why it becomes so, so difficult these days is the volumes of threats that we're seeing. I'm kind of come out of what I refer to the cybercrime ecosystem, right? Where anytime, do you know anybody who wants to get into a life of cyber crime, they really don't need to know much. They just need to understand, right? Where to get these particular services that they can sort of rent, right? You have malware as a service, right? You got kind of ransomware as a service. So that's an important to make sure we understand. Um,, Hey, anybody can get into a life of cyber crime and that volume is really sort of being driven by the cyber crime ecosystem. >>Well, the threat report noted, uh, specifically that the, uh, as you said, the life of crime is getting cheaper for folks to get into because just as we're moving from products to services in technology and in other parts of the industry, we're moving from products to services in, uh, the threat world. To talk a little bit about this, what you just said, this notion of, you know, bad guy as a service, what's happening. >>Yeah, I like that bad guy as a service. Um, what's really kind of popular these days is ransomware as a service. Um, then two, three we saw two more variants, uh, ramps and wears as a service, uh, you know, Soden and then also, um, I think I can pronounce it empty. I always have a hard time pronouncing all of these malware name. But anyway, these are new variants now that are coming up. Um, and of course anytime you get something new, the malware usually has more, you know, more a more advanced kind of capabilities. And you know, these malwares have, you know, ways to evade a Vieta taction you know, they're looking for different services that may be on the, the operating system, finding ways to be able to the war, the detection of their particular malware or if someone is analyzing that particular threat, making it longer for an analyst to be able to figure out what's going on. >>Mmm. And as well as trying to avoid different types of sandbox technologies. Now I think that's something bad to actually, you know, really worry about. But what really gets me, and I might've said this, um, in some of the previous conversations this year is that the tactics are also kind of changing a bit for ransomware as a service coming out of the cyber-crime ecosystem. It used to be more opportunistic. There was a spray and pray approach, let's hope something sticks. Right. Totally changed. They're becoming a lot more targeted. And one of the main reasons why it was because organizations are paying large amounts of money or the ransom depending large amounts of money to the group yo yo to have 'em the ability to decrypt their files after they get hit with ransomware. And you've seen this right now, the adversaries are targeting organizations or industries that may not have the most robust security posture. >>They're focused on municipalities. Yeah, they're focused on, okay. Cities also state local government. Um, well we saw it earlier on this year, the city of Baltimore, we had a bunch of cities in Florida, actually one city in Florida ended up having to pay $600,000 in a ransom to be able to have their files decrypted. And also in the state of Texas we saw, Mmm. A, uh, malware variant or ransomware variant hit about 22 municipalities throughout the state of Texas. And you know, the one other thing I think seems to be common amongst all of these victims is a lot of them have some type of insurance. So I think the bad guys are also doing some research or doing their homework to make sure, Hey, if I'm going to spend the money to target this individual or this organization, I want to make sure that they're going to be able to >>painting the ransom. They're refining their targets based on markers, which is how bad guys operate everywhere, right? You decide who your Mark is and what their attributes are. And because these are digital, there's also a lot more data flying around about who these marks are, how they work. Uh, as you said, the availability of insurance means that there is no process for payment in place because insurance demands it and it accelerates, uh, the, the, the time from hitting them to getting paid if I got that right. >>Yeah, that is 100% spot on, you know, efficiency, efficiency, officio. I mean, we all want to get paid as fast as possible, right? Yeah. >>Peter. Yeah, that's true. That's true. All right, so it's time for prescription time, Tony. It's a, uh, we've talked about this for probably six or eight quarters now and every time I ask you, and what do folks do differently in the next few months? Uh, what should they do differently in the next few months? >>You know, I like to talk a lot about how we, you know, you have to have that foundational, uh, it kind of infrastructure in place, having visibility and all that debt and that's 100% sort of true. Um, that doesn't change. But I think one thing that we can start doing, um, and this is wonderful. Um, I'm sort of project that had transpired over the last few years from the MITRE, uh, organization is the MITRE attack framework. Uh, what had happened was miter had gone out there and brought in, um, through all these open source outlets, different types of threat reports. Mmm. That the adversaries, um, you know, we're, di we're documented actually doing, they took all those tactics and corresponding techniques and documented all of them in one location. So now you have a common language for you to be able to determine and be able to learn what the actors are actually doing to come their cyber mission. >>And because now we have that there's a trend. Now organizations are starting to look at this data, understand it, and then operationalizing it into their environment. And what I mean by that is they're looking at the axle the, uh, tactic and the technique and not know, understanding what it is, looking at, what is the actual digital dust that it might leave behind, what's the action and making sure that they have the right protections and and they're grabbing the right logs at least to be able to determine when that particular threat actor, using that technique happens to be in there environment. >>But it also sounds as though you, you know, you noted the use of common language that it sounds as though, uh, you're suggesting that enterprises should be taking a look at these reports, studying them, uh, reaching agreement about, uh, what they mean, the language so that they are acculturating themselves to this more common way of doing things. Because it's the ability to not have to negotiate with each other when something happens and to practice how to respond. That really leads to a faster, more certain, uh, more protecting response if I got that right. Yeah. >>You know, 100%. And I'll also add though, um, as you start to operationalize this no miter attack framework and understanding what the adversaries are kind of doing, you get more visibility. Yeah. But then also what you're seeing is there's a trend of vendors starting to create what's referred to as threat actor playbooks, right? So there, as they discover these actual threads, they're mapping the actual tactics and techniques back to this common language. So now you have the ability to be able to say, Hey, I just seen a, you know, Fordanet just put this report out on this particular, you know, threat actor or this malware because we're leveraging a common language. They can more easily go back and see how they're actually defending against these particular, you know, TTPs. Well, and the latest one, you know, that we put out, uh, just this week was, um, uh, uh, a playbook on the malware that's a banking Trojan. >>Well, at least it started out as a banking Trojan. It's kind of morphed into something a little more now. You see it delivering a bunch of malware variants, um, you know, different malware families. It's almost like a botnet now. And, uh, we hadn't actually seen it, um, really for a little while. But in Q three we saw a bunch of different campaigns spawn. And like I always say, malware a hibernate for a little bit, but when it comes back, it comes back bigger, faster, stronger. There's always new tactics, there's only new capabilities. And then this case, that's no exception. What they did, Mmm. And I thought was very unique, uh, at being able to, again, crayon, Mmm. The humans to be able to make a mistake. So what they did is they as a victim, they would grab the email, thread from the emails, grab those threads, I put it in a spoofed email, and then email that to the next victim. And they'll actually, um, so know when the victim opens up that particular email, they see that thread that looks like, Hey, I've had this correspondence, you know, before this has to be a good email, I'm going to clip that attachment. And when they do, now they're compromised and that whole process happens over and over and over again. >>So there's, they're scraping the addressees and they are taking the email and creating a new AML and sending it onto new, uh, addressees hopefully before the actual real email gets there. Right. >>Uh, you know, yes. But also say that, um, they're actually, they're taking the context of the email, right? So the email sort of thread. So it makes it, it's an actual real thread. Well, they're just kind of adding it in there. So it's really it really looks like it's, hello. Hey, I've had that correspondence before. Um, I'm just going to click that link. >>So that's me. This notion of operationalizing through the minor and these new playbooks, uh, is a, a way ultimately that more people, presumably we're creating more of a sense of professionalism that will diffuse into new domains. So, for example, you mentioned early on, uh, municipalities and whatnot that may not have the same degree of sophistication through this playbook approach, through the utilizing these new resources and tools that Fortinet and others are providing. It means that you can raise to some degree, the level of responsiveness in shops that may not have the same degree of sophistication. Correct? >>Yeah, I didn't, you know, I definitely would have to agree. And it also, I think as you start to understand these techniques, you will never just have one technique as a standalone, right? These techniques are Holies chained together, right? You're going to have, once this technique is there, you're going to know that there's a few techniques are probably have a happen before and there's some, they're going to happen later. A great example of this, let's say, when you know, when an adversary is moving laterally inside the network, there's really three basic things that they have to be able to have. One is they have to have the authorization, the access, you know, to be able to move from system to system. Once they have that, you know, and there's a way a variety of ways that they can do that. Once they're there, now they have to somehow copy that malware from system to system. >>And you know, you can do that through, you know, ah, remote desktop protocol. You can do that through no P S exact. It's a variety of different ways you can do that. And then once the malware's there, then you have to execute it somehow. And there's ways to do that. Now if you have a common language for each one of those, now you start chaining these things together, you know, the digital dust or the actual behaviors and what's actually left behind with these actual tactics. And now as manually you can start better understanding how to, you know, thread hunt more efficiently and also start to actually let the technology do this kind of threat hunting for you. So I guarantee you we're going to see innovation and technology where they're going to be doing automatic through hunting for you based on these types of understandings in the future. >>Tony, what's growing? Once again, great cube conversation. Thanks again for being on the cube. Tony John, John de Medico is, I'm going to just completely shorten your title, uh, threat landscape expert Fort Tony. Thanks again. >>Yeah, it's great to be here. Peter. Thanks a lot, >>and thanks once again for joining us for another cube conversation on Peter Burris. See you next time..

>>Our studios. Silicon Valley, Palo Alto, California is a Q conversation. Hi and welcome to the cube studios in Palo Alto, California for another cube conversation where we go in depth with the tech leaders driving innovation across the technology industry. I'm your host Peter Burris. Well, it's that time of quarter again. Every quarter we get together with Fortinet to discuss their threat landscape report, which is one of the industry's best and most comprehensive views into how the bad guys are utilizing bad software and bad access to compromise digital business and steel digital assets. Now, this quarter's report suggests that there's not as much new stuff going on. If you look at the numbers, they're relatively flat compared to previous quarters, but that doesn't tell the real story. Underneath those numbers, we see that there is a churn. There's an incredibly dynamic world of bad actors doing bad things with old and new bad stuff to try to compromise digital business to learn more about this dynamism and what's really happening. Once again, we've got a great cube guest, Tony Gian. Medico is a senior security strategist and researcher and CTI lead at Fortinet. Tony, welcome back to the cube. >>Hey Peter, it's great to be here. >>So Tony, I started off by making this observation that the index suggests that we're in kind of a steady state, but that's not really what's happening. Is it? What's really going on? Where it's going on inside the numbers? >>Yeah, no, we start to see a little bit of a shift of tactics. Um, what has happened, I think, uh, not all the time, but sometimes with the adversaries like to do is penetrate an organization where maybe us as defenders aren't necessarily as focused in on, and a great example is this. For many years we were focused on at and rightfully so, and we continue to be focused on this is being able to block a phishing email, right? We have our email security gateways to be able to not allow that email to come into the network. We also then for for whatever reason, if it happens to get into the network, we focus on user awareness training to educate our users to make sure that they can identify a malicious email. They're not clicking that link are clicking that attachment. Now with that said, we look at the actual data in our Q three threat last grade report and what we're seeing is the adversaries are targeting vulnerabilities that if they were successfully exploited would give them remote code execution, meaning that they, they they can compromise that box further and further inside the network. >>Now granted that's been happening for many years but we have actually seen an increase order. As a matter of fact, it was number one prevalence across all the actual regions. So with that said, I think it's worth making sure that you're looking at your edge devices or your edge services that are publicly exposed out there. Make sure that there's no vulnerabilities on them, make sure that they're not misconfigured and also make sure that you have some type of multifactor authentication. And I think like we've talked about many times that threat landscape or that no threat attack surface continues really to expand, right? You got, you got cloud, you have IOT. So it's becoming more and more difficult to be able to secure all those edge services. But definitely you know, something you should take a look at >>and you got more people using more mobile devices to do more things. So, so it sounds as though it's a combination of two things. It's really driving this dynamism, right, Tony? It's one, just the raw numbers of growth and devices and opportunities and the threat surface is getting larger and the possibility that something's misconfigured is going up and to that they're just trying to catch organizations by surprise. One of those is just make sure you're doing things right, but the other one is don't keep, take your eye off the ball, isn't it? How are organizations doing as they try to, uh, expand their ability to address all of these different issues, including a bunch that are tried and true and mature, uh, that we may have stopped focusing on? >>Yeah. You know, it's really hard, right? I always say this and um, you know, I get some mixed kind of reacts in sometimes, but you can't protect and monitor everything. I mean, depending on how large your network is, it's really difficult. So, I mean really focusing on what's important, what's critical in your organization is probably really the best approach. I mean, really kind of focusing on that. Now with that said though, the reason why it becomes so, so difficult these days is the volumes of threats that we're seeing. Um, kind of come out of what I refer to the cybercrime ecosystem, right? Where anytime, do you know anybody who wants to get into a life of cyber crime, they really don't need to know much. They just need to understand, right, where to get these particular services that they can sort of rent, right? You have malware as a service, right? You got kind of ransomware as a service. So it's an important to make sure we understand, um, Hey, anybody can get into a life of cyber crime and that volume is really sort of being driven by the cyber crime ecosystem. >>Well, the threat report noted, uh, specifically that the, uh, as you said, the life of crime is getting cheaper for folks to get into because just as we're moving from products to services in technology and in other parts of the industry, we're moving from products to services in, uh, the threat world. To talk a little bit about this, what you just said, this notion of, you know, bad guy as a service, what's happening? >>Yeah, I actually that bad guy as a service. Um, what's really kind of popular these days is ransomware as a service. Um, as a matter of fact, uh, In Fortiguard labs, we were tracking for about two years or so, one of the most prolific ransomware-as-a-service GandCrab. Matter of fact, over the two year period, they gleaned off about over $2 billion  dollars worth of ransoms. Now, they said that they kind of shut down and as they started closing down operations in Q3, we saw two more variants of ransomware as a service. You know, Soden and, and also, uh, I think I can pronounce it ... "Nempty". I always have a hard time pronouncing all of these malware name. But anyway, these are new variants now that are coming up. And of course anytime you get something new, the malware usually has more, you know, more a more advanced kind of capabilities in, you know, these malwares have, you know, ways to evade detection, you know, they're looking for different services that may be on the, the operating system, finding ways to be able to thwart the detection of their particular malware, or if someone is analyzing that particular threat, making it longer for an analyst to be able to figure out what's going on. >>Um, and as well as trying to avoid different types of sandbox technologies. Now I think that's something bad that actually, you know, really worry about. But what really gets me, and I might have said this, um, in some of the previous conversations this year, is that the tactics are also kind of changing a bit for ransomware as a service coming out of the cyber-crime ecosystem. It used to be more opportunistic. There was a spray and pray approach, let's hope something sticks. Right? Totally changed. They're becoming a lot more targeted. And one of the main reasons why it is because organizations are paying large amounts of money or the ransom depending large amounts of money to the group. Yo yo to have 'em the ability to decrypt their files after they get hit with ransomware. And you've seen this right now, the adversaries are targeting organizations or industries that may not have the most robust security posture. >>They're focused on municipalities. No, they're focused on, you know, cities also state local government. Um, well we saw it earlier on this year, the city of Baltimore. We had a bunch of cities in Florida, actually one city in Florida ended up having to pay $600,000 in a ransom to be able to have their files decrypted. And also in the state of Texas we saw, um, a uh, malware variant or ransomware variant hit about 22 municipalities throughout the state of Texas. And you know, the one other thing I think seems to be common amongst all of these victims is a lot of them have some type of insurance. So I think the bad guys are also doing some research or doing their homework to sure, Hey, if I'm going to spend the money to target this individual or this organization, I want to make sure that they're going to be able to, yeah, pay me the ransom. >>They're refining their targets based on markers, which is how bad guys operate everywhere, right? You decide who your market is and what their attributes are. And because these are digital, there's also a lot more data flying around about who these marks are, how they work. Uh, as you said, the of the availability of insurance means that there's now a process for payment in place because insurance demands it and it accelerates, uh, the, the, the time from hitting them to getting paid. If I got that right. >>Yeah, that is 100% spot on, you know, efficiency, efficiency, officio. I mean, we all want to get paid as fast as possible. Right? Right. >>Peter? Yeah, that's true. That's true. Alright, so it's time for prescription time, Tony. It's a, a, we've talked about this for probably six or eight quarters now and every time I ask you and what do folks do differently in the next few months? Uh, what should they do differently and the next few months? >>Ah, you know, I like to talk a lot about how we, you know, you have to have that foundational, it kind of infrastructure in plays, having visibility and all that debt and that's 100% sort of true. Um, that doesn't change. But I think one thing that we can start doing, um, and this is wonderful. Um, I'm sort of project that had transpired over the last few years from the MITRE, uh, organization is the MITRE attack framework. Uh, what had happened was MITRE had gone out there and brought in, um, through all these open source outlets, different types of threat reports, um, that the adversaries, um, you know, we're di we're documented actually doing, they took all those tactics and corresponding techniques and documented all of them in one location. So now you have a common language for you to be able to determine and be able to learn what the actors are actually doing to come cyber mission. >>And because now we have that there's a trend. Now organizations are starting to look at this data, understand it and then operationalizing it into their environment. And what I mean by that is they're looking at the actual, the uh, tactic and the technique and you know, understanding what it is, looking at, what is the actual digital dust that it might leave behind, what's the action and making sure that they, I have the right protections and the Texans and they're grabbing the right logs at least to be able to determine when that particular threat actor, using that technique happens to be in there environment. >>But it also sounds as though you, you know, you noted the, uh, use of common language that it sounds as though, uh, you're suggesting that enterprises should be taking a look at these reports, studying them, uh, reaching agreement about what they mean, the language so that they are acculturating themselves to this more common way of doing things. Because it's the ability to not have to negotiate with each other when something happens and to practice how to respond. That really leads to a faster, more certain, more protecting response if I got that right. Yeah. >>You know, 100%. And I'll also add though, um, as you start to operationalize this no miter attack framework and understanding what the adversaries are kind of doing, you get more visibility. Yeah. But then also what you're seeing is it's a trend of vendors starting to create what's referred to as threat actor playbooks, right? So there, as they discover these actual threads, they're mapping the actual tactics and techniques back to this common language. So now you have the ability to be able to say, Hey, I just seen, uh, you know, Fordanet just put this report out on this particular, you know, threat actor or this malware because we're leveraging a common language. They can more easily go back and see how they're actually defending against these particular, you know, TTPs. Well, and the latest one, you know, that we put out, uh, just this week was, um, uh, Oh, a playbook on the malware it's a banking Trojan. >>Uh, well at least it started out as a banking Trojan. It's kinda morphed into something a little more now. You see it delivering a bunch of malware variants, um, you know, different malware families. It's almost like a botnet now. And, uh, we hadn't actually seen it, um, really for a little while. But in Q three we saw a bunch of different campaigns spawn. And like I always say, malware a hibernate for a little bit, but when it comes back, it comes back bigger, faster, stronger. There's always new tactics, there's always new capabilities. And then this case, that's no exception. What they did, um, and I thought was very unique, uh, at being able to, again, Ray on, um, the humans to be able to make a mistake. So what they did is they, as a victim, they would grab the email thread from the emails, grab those threads, I put it in a spoofed email, and then email that to the next victim. And they'll actually, um, so you know, when the victim opens up that particular email, they see that thread that looks like, Hey, I've had this correspondence, you know, before this has to be a good email, I'm going to click that attachment. And when they do, now they're compromised and that whole process happens over and over and over again. >>So there's, they're scraping the addressees and they are taking the email and creating a new AML and sending it onto new, uh, addressees hopefully before the actual real email gets there. Right? >>No, yes, but also say that, um, they're actually, they're taking the context of the email, right? So the email sort of thread, so it makes it, it's an actual real thread. Well, they're just kind of adding it in there. So it's really. It really looks like it's, hello. Hey, I've had that correspondence before. Um, I'm just going to click that link for attachments. >>This notion of operationalizing through the minor framework and these new playbooks, uh, is a, a way ultimately that more people, presumably we're creating more of a sense of professionalism that will diffuse into new domains. So, for example, you mentioned early on, uh, municipalities and whatnot that may not have the same degree of sophistication through this playbook approach, through the utilizing these new resources and tools that Fort Dannon and others are providing. It means that you can raise to some degree, the level of responsiveness in shops that may not have the same degree of sophistication. Correct? >>Yeah, I did. You know, I, I definitely would have to agree. And then also, I think as you start to understand these techniques, you will never just have one technique as a standalone, right? These techniques are Holies chained together, right? You're going to have, once this technique is there, you're going to know that there's a few techniques or probably have happened before and there's some, they're going to happen later. A great example of this, let's say, when you know, when an adversary is moving laterally inside the network, there's really three basic things that they have to be able to have. One is they have to have the authorization, the access, you know, to be able to move from system to system. Once they have that, you know, and there's a way a variety of ways that they can do that. Once they're there, now they have to somehow copy that malware from system to system. >>And you know, you can do that through, you know, ah, remote desktop protocol. You can do that through no P S exact. There's a variety of different ways you can do that. And then once the malware's there, then you have to execute it somehow. And there's ways to do that now if you have a common language for each one of those, now you start chaining these things together, you know, the digital dust or the actual behaviors and what's actually left behind with these actual tactics. And now as manually you can start better understanding how to, you know, threat hunt more efficiently and also start to actually let the technology do this kind of threat hunting for you. So I guarantee you we're going to see innovation and technology where they're going to be doing automatic through hunting for you based on these types of understandings in the future. >>Tony, what's growing? Once again, great cube conversation. Thanks again for being on the cube. Tony John, John de Medico is, I'm going to just completely shorten your title, uh, threat landscape expert Fort net. Tony, thanks again. >>Hey, it's great to be here, Peter. >>Thanks a lot, and thanks once again for joining us for another cube conversation on Peter Burris. See you next time..

from our studios in the heart of Silicon Valley Palo Alto California this is a cute conversation welcome to the cube studios for another cube conversation where we go in-depth with thought leaders driving business outcomes with technology I'm your host Peter Burris every Enterprise that is trying to do digital transformation finds themselves facing two challenges one their digital assets themselves are a source of value and to other assets that are sources of value are becoming increasingly digitized and that creates a lot of challenges a lot of security concerns that bad agents out in the internet are exploiting and requires a programmatic fundamental response to try to ensure that the digital assets or digitized assets aren't mucked with by bad guys so to have that conversation we're here with Tony Jian Domenico Tony's a senior security strategist and a researcher and the CTI lead at Ford NIT Tony welcome back to the cube hey Pete it's great to be here man so as you get to see you yeah well we've been doing this for a couple of years now Tony and so let's get just kick it off what's new so what's new should we start talking about a little bit about the index here what we saw with the overall threat landscape sure well cool so you know y'all like you know like we always do we always like to start off with an overall threat landscape at least they give an overview of what that index looks like and it really consists of malware botnets application exploits and what we looked at over the quarter there was a lot of volatility throughout the quarter but at the end of the day it ended up only 1% higher than the quarter before now some of that volatility really is being driven by what we've talked about a lot of times Peter and a lot of these other episodes is that swarm like activity whenever an actual vulnerability is successfully exploited by an adversary everybody swarms in on that vulnerability and our fertig are labs you see that really like super spike up a great example of that would be in the last year in December think PHP which is an application that's a framework to rapidly develop web apps they had a vulnerability that if you successfully exploited it it would give you remote the remote access or I'm sorry remote code execution and they were exploiting that and we definitely seen a huge uptick now that wasn't the only one for the quarter but that and along with some of the other ones it's really what's kind of driving on volume so the index has been around for a few quarters now and it's a phenomenal way for folks out there to observe how overall trends are evolving but as you said one of the key things that's being discovered is that or you're discovering as you do this research is this notion of swarming it seems as though there ought to be a couple of reasons why that's the case Tony it's it's we've talked about this in the past there's folks who want to get a little bit more creative in creating bad stuff and there's other folks who just want to keep the cost low and just leverage what's out there which approach are the bad guys tend to using more and or is there an approach one of the other approach is more targeted to one or another kind of attack well it's funny you usually see the folks in the cyber crime ecosystem that are really focusing on you know identifying them not so much where they're doing more sort of targeted attacks it's more of a you know pray and spray you know type of thing and you see a lot of that you know anytime they can hire you can get a life of cybercrime right in the leverage some of these common you know you know services you have code reuse you know which is out there so you have that sort of like group there right and then you have more of the you know more of the you know hands-on sort of keyboard the more you know targeted attacks that are really focused on specific you know victims so you have those you know those two groups I say now with that though there kind of is a commonality there where there's this concept and it's nothing new we've been talking about this for years in the cybersecurity industry it's living off the land right where once a victim is on the actual machine itself they start leveraging some of the tools that are already available there and usually these tools their administration tools to be able to minister the actual network but these tools can also be used in the farías ways from example here would be you know PowerShell they you know a lot of admins use PowerShell for efficiencies on the network but that also can be used in the forest ways and the bad guys are using that and then this past quarter you know we did see a lot of PowerShell activity now you know Peter having said that though I think as a whole with the security community we're getting better at being able to identify these types of PowerShell attacks one we got better technology on the endpoint and I think to Microsoft is in a better job of being able to provide us more hardening capabilities for PowerShell like being able to restrict access to PowerShell as well as giving us better logging capability to be able to identify that malicious activity so we are getting better and the bad guys know this so I think what we can probably look for in the future is them leveraging either a different interface or different language because all they really need to do is interface with that dotnet framework which is part of a Windows system and they can start doing the same exact things they were doing with PowerShell and we're seeing that it in the open-source community now things like Silent Trinity open source tool that allows you to do those same things so for C an open source pretty much guarantee we're gonna see it out there in the wild here soon so we've got a group of bad actors that are using this living off the land approach to leverage technology that's out there and we've still got kind of the big guys having to worry about being targeted because you know that's how you make a lot of money if you're successful but it certainly does sound is that a general business practice for a lot of these guys is to leverage common infrastructure and that this common infrastructure is increasingly becoming you know better understood have I got that right no I you know Peter you're spot-on here what we did we did some exploratory research in this last quarter and what we found out is with the exploits within that quarter or or or the axe will come threats sixty percent of those threats are using the same infrastructure what I mean by infrastructure you know I I mean things like you know infrastructure to download malware maybe to redirect you to some other site and then downloads malware and that makes a lot of sense Peter you know why because in this cybercrime ecosystem if you didn't realize this it's a vicious competitive market everybody is trying to sell their wares and they want to make sure that their service is the best it's better than someone else's and they want to make sure that it's stable so they find these you know community you know infrastructures that are tried-and-true you know some of them are from you know bulletproof hosting so you know services you know things of that nature so you see a lot of the folks in a cybercrime ecosystem using them now on the flip side though you definitely see some of the thread actors that are more sort of you know more the advanced threat actors maybe what they want to do is hide a little bit so they'll hide in that larger community to be able to possibly be able to bypass that that attribution back to them because they don't want to be sort of labeled with oh hey this particular thread actor always uses this infrastructure so if they can blend in a lot harder to find them so they can use what is available but at the same time differentiate themselves in this bad actor ecosystem to take on even more challenging the potentially lucrative exploits now tell me if we know something about this common infrastructure as you said sixty percent of these attacks are using this common infrastructure that suggests we can bring a common set of analysis frameworks to bear as we consider who these actors are and what their practices are have I got that right yeah yeah absolutely if you can align your PlayBook defenses with the offensive actual playbook that the threat actors are using they're better off you're gonna be right because then you can be able to combat them a lot better and as a matter of fact I mean we've kind of introduced this sort of concept in conjunction with our our partnership with the cyber threat Alliance we're actually producing these thread actor play books you know and what we're doing is the idea behind this is if we can identify the malicious activity the threat actors are actually doing to complete their cyber mission expose some of them tactics those techniques those procedures we could possibly disrupt some of that malicious activity and you know this past this past quarter here we focused on a group you know Peter called the the silence group and they're really focused on identifying and stealing financial data they're looking at banks banking infrastructure and ATM machines and you'll get a kick out of this with the ATM machines they're doing something called jackpot II where they if they can find the axle software behind the ATM machine find that ATM process they can inject a malicious DLL into that process giving them total control over the ATM machine and now they can dispense money at will and they can have these money mules on the other side receive that actual money so you know we have a lot of different campaigns in play books that we've identified on our website and that once we understand that we align that with our security fabric and ensure that our customers are protected against that particular playbook Tony I'm not happy to hear that so this is this is my distressed face that I use during these types of interviews but it's if if we're able to look at how bad guy play books are operating then we ought to be able to say and what are those fundamentals that a shop should be using the security professionals should be using that are just you know so basic and so consistent and it seems that are you guys have identified three to do a better job of taking a fabric approach that starts to weave together all assets into a more common security framework to to do a better job of micro and macro segmentation so that you can identify where problems are and then finally increase your overall use of automation with AI and m/l how is this translating into your working with customers as they try to look at these playbooks and apply their own playbooks for how they set up their response regimes yeah so I mean I think overall I mean I think you can hit it on the head computer you kind of nailed down really those some it was kind of fundamental sort of concepts here now you can identify and you can document as many playbooks as you want but if you're not able to quickly respond when you identify those actual playbooks you know that's really half the battle I mean if you need to be able to identify you know one not only when the threat actors in your environment but then also you need to be able to quickly you know take action and like you were saying with that fabric if we can have that actual fabric being able to talk to the other controls within that fabric and take some action they're better off you're gonna be because you can align your defenses there and that's a great would you gotta make sure that all the controls within that fabric are all communicating together they're working together they're sharing information and they're responding together sure enough yeah are you starting to advise customers I'm curious you advising customers that even as they increase the capabilities of their fabric and how they handle their architectures from a micro macro segmentation and increase their use of automation or are there things that they can do from a practice standpoint just to ensure that their responses are appropriate fast and accurate yeah sure sure I mean I think a lot of the actual fabric once you actually build that fabric there's certain you know playbook responses that you can program into that fabric and I'll also even go I know we talked about you know fundamentals but I'll even dive a little bit lower here and you know you have that fabric but you also have to make sure you understand all the assets you have in your in you know your environment because that that information and that knowledge helps you with that macro and micro segmentation because when you can isolate you know different areas if there is a certain area that gets infected you can quickly turn the knobs to isolate that particular threat and that specific you know area or that's a specific segmented area and that is really gonna allow you to fight through the attack give you more time and ultimately reduce the impact of that particular breach so Tony we got the summer months coming up that means more vacations which is you jest less activity but then we got summer interns coming in which you know may involve additional clicking on things that shouldn't be clicked on any ideas what what should security pros be thinking about in the summer months what's the trend show well I think we're gonna continue to see that you know I I think the same type of threats that we've seen in the first quarter but I would say you know there may be a slight sort of drop-off right we got kind of kids that are gonna be out on vacation so you know schools may not see as much activity you got you know folks gonna be taking vacations and at the end of the day most of these exploits are client-side exploits which means you know a lot of times you need somebody to do something on the actual computer either you know clicking that link or clicking the attachment and if they're not there to do that they'll just sit there and you'll see less activity over time so we might see a little reduction in volume but I still think we'll see very similar types of you know threats in the coming months so good time good time are a good opportunity for security pros to double down on putting in place new architecture practices and response regime so that when stuff kicks up in the fall they're that much more prepared da Tony G on Domenico fort Ned great once again thanks very much for being on the cube hey you know Peter it's always a pleasure being here man hope to see you again soon you will and once again I'm Peter Burroughs until next time [Music]