(upbeat music) >> Welcome to theCUBE. I'm Dave Nicholson, and this is part of the continuing conversation about Managing Risk in the Digital Supply Chain. I have with me today Vincent Danen, vice president of product security from Red Hat and Luke Hines security engineering lead from the office of the CTO at Red Hat. Gentlemen, welcome to theCUBE. >> Thank you. >> Great to be here. >> So let's just start out and dive right into this, Vincent, what is the software or digital supply chain? What are we talking about? Yeah, that's a good question. Software supply chain is basically the software that an end user would get from a vendor or in our case, we're talking about open source, so upstream. It is the software that comes in that is part of your package, operating system, applications. It could be something that you get from one vendor, multiple vendors. So we look at in the example of Red Hat, we are one part of the customer's software supply chain. >> So it's interesting that it's coming in from different areas. Do we have a sense for the ratio of kind of commercial software versus open source software that makes up an enterprise today? >> I think that's a really hard thing to answer and I think every enterprise or every company would have a little bit different. Depends if you have an open source vendor that you choose, you may get a significant amount of software from them. Certainly you're not going to get it all. As an example, Red Hat provides thousands of open source packages. We certainly can't provide all of them. There are millions that are out there. So when you're looking at a specific application that you're building, chances are, you could be running that on a managed platform or an enterprise supply platform, but there are going to be packages that you're going to be obtaining from other sources in other communities as well in order to power your applications. >> So, Luke, that sounds like a kind of a vague situation we're looking at in terms of where all of our software is coming from. So what do we need to know about our software supply chain in that context? What do we need to understand? Before we even get anywhere near the idea of securing it, what are some of the issues that arise from that? >> Yeah, so Vincent's touchpoint is a very wide range in ecosystem, multiple sources when we're talking about open source. So essentially awareness is key really. I think a lot of people are really not aware of the sources that they're drawing from to create their own supply chain. So there's multiple supply chains. You can be somebody like Red Hat that the provide software, and then people will leverage Red Hats for their own supply chain. And then you have the cloud provider and they have their own source of software. So I think that the key thing is the awareness of how much you rely upon that ecosystem before we look at the security of the supply chain. It's really understanding your supply chain. >> And just to follow up on that. So can you... I'm sort of checking my own level of understanding on this subject. When you talk about open source code, you're talking about a code base that is often maintained essentially by volunteers, isn't that correct? >> A mix of volunteers and paid professionals where a company has an interest in the open source project, but predominantly I would say it's... Well, I'm not entirely sure, but volunteers make up a substantial part of the ecosystem that is for sure. So it's a mix really. Some people do it because they enjoy writing software. They want to share software. Other people also enjoy working software, but they're in the position that a company pays for them to work on that software. So it's a mix of both. >> Vincent, give us a reminder of reminder of why this is important from a little bit of a higher level. Step back from the data center view of things, from the IT view of things, just from a societal perspective, Vincent, what happens when we don't secure our digital supply chain? What are the things that are put at risk? >> Okay, well, there's a significant number of things that are placed at risk, the security of the enterprise itself. So your own customer data, your own internal corporate data is place at risk if there were a supply chain breach. But further to that for a software provider, and I think that in a lot of cases, most companies today are software providers or software developers. You actually put your own customers at risk as well, not just their data, but their actual... The things that they're working on, any workloads that they may have, an order that they might place as an example. So there's a number of areas where you want to have the security of that supply chain and the software components that you have figured out. You want to be on top of that because there is that risk that trickles down when it comes to an event. I mean, we've seen that with breaches earlier this year, one company is breached multiple companies end up being breached as a result of that. So it's really important. I think we all have a part to play in that I always view it as it's not just about the company itself. So I mean, speaking from a Red Hat perspective, I don't look at it as we're just securing Red Hat, we're securing our customers, and then we're also doing that for their customers as well, because they're writing software that's running on the software that we're providing to them. So there is this trickle down effect that comes, and so I think that every link in that chain, I mean, it's wonderful that it's called a supply chain. It's only as strong as its weakest link. So our view is how do we strengthen every link in that chain? And we're one part of it, but we're kind of looking a little broader, what can we do upstream and how can we help our customers to ensure the security of their part in that supply chain? >> Yeah, I want to talk about that in a broad sense, but let's see if we can get a little bit more specific in terms of what some of the chains look like because it's not just really one chain when you think about it, there's the idea of inherent flaws that can be caught and then there are the things that bad actors might be doing to leverage those flaws. So you've got all of these different things that are converging. So first and Vincent, if you want to toss this to Luke back and forth, it's up to you guys. What about this issue of inherent flaws in code? We referenced this idea of the maintainer community. What are best practices for locking that down to make sure that there aren't inherent flaws or security risks? >> I'll take a stab at it, and then I'll let Luke follow up with maybe some of the technologies that Red Hat provides. And again, speaking to Red Hat as part of that chain. When we're talking about inherent risk, there's a vulnerability that's present upstream. We pull that software to Red Hat. We package it as a component of one of the pieces of software that we provide to our customers. It's our responsibility to pay attention to those upstream potential vulnerabilities, potential risks, and correct them in our code. So that might be taking a patch from upstream, applying it to our software, might be grabbing the latest version from upstream, whatever the case might be, but it's our responsibility to provide that protection for that software to actually remediate that risk, and then our customers can then install the update and apply the mitigation themselves. If we take a look at it from, when we're looking at multiple suppliers where you'd asked earlier about, what part of it is Red Hat and what part of it is self-service open source? When you look at that, the work that Red Hat's doing there as a commercial provider of open source and end user for that little bit that they're going to grab themselves, that Red Hat doesn't provide, it's going to have to do all of those things as well. They're going to have to pay attention to that risk from upstream. They're going to have to pay attention to any potential vulnerabilities and pull that in to figure out, do I need to patch? Where do I need to patch it? And that's something we didn't really touch on was an inventory of the software that you have in place. I mean, you don't know that you need to fix something. You don't even know that it's running. So, I mean, there's a lot of considerations there where you have to pay attention to a lot of sources. Certainly there's metadata, automation, all of these things that make it easier, but it doesn't absolve us of the responsibility across the board to pay attention to these things, whether you're grabbing it from upstream directly or from the vendor. And it's the vendor's responsibility to then be paying attention to things upstream. >> Yeah, so Luke, I want you to kind of riff on that from the perspective that let's just assume that Vincent was just primarily talking about the idea that, okay, we've established that this code is solid and we've got gold copy of it and we know it's okay. There aren't inherent problems in the code as far as we can tell. Well, that's fine. I'm a developer. I go out to pull code and to use. How do I know if it's not been tampered with? How do I know if it's in fact the code that was validated during this process before? What do you do about that? >> So there's several methods there, but I just like to loop back to that point, because I think this is really interesting around, so if you look at a software supply chain, this is a mix of humans and machines, and both have flaws, probably humans a bit more. And a supply chain, you have developers. You have code reviewers, you have your systems administrators that set up the systems, and then you have your machine actors. So you've got your build systems, the various machines that are part of that supply chain. Now the humans, there's a as an attack factor there 'cause typically they will have some sort of identity, which they leverage for access to the supply chain. So quite often a developer's identity can be compromised. So a lot of the time people will have a corporate account that gives them some sort of single sign on access to multiple systems. So the developers are coming and this could be somebody in the community as well. Their account is compromised, then they're able to easily backdoor systems. So that's one aspect. And then there is machines as well. There's the whole premise of machines software not being up to date. So when the latest nasty vulnerability is released, machines are updated, then the machines have their flaws. They can be exploited. So I would say it's not just a technical problem. There is a humanistic element to this as well around protecting your supply chain. And I would say a really good perspective to carry when you're looking to, how do I secure my supply chain is treat it like you would a production system. So what do I mean by that? When we put something into production and we've got this very long legacy of treating it with a very strict security context around who can access that people, okay. How much it's upgraded and it's patched? And we seem to not have this same perception around our supply chain and our build systems, the integrity of those, the access of those, the policy around the access and so forth. So that's one giveaway that I would say is a real key focus that you should have is treat it like a production system. Be very mindful about what you're bringing in, who can access it because it is the keys to the kingdom, because if somebody compromises your supply chain, your build systems and so forth, they can compromise the whole chain because the chain is only as strong as the weakest link. So that's what I draw upon it. And around the verifications, there is multiple technologies that you can leverage. So Red Hat, we've got a very robust sign in system that we use so that you can be sure that the packages that we get you have non-repudiation that they've been produced by Red Hat. When you update your system, that's automatically looked after. And there are other systems as well, there's other new technologies that are starting to get a foothold around the provenance of aspects of your build system. So when you're pulling in from these multiple sources of open source communities, you can have some provenance around what you're putting in as well. And yeah, I don't want to bite share too much on the technologies, but there's some exciting stuff starting to happen there as well. >> So let's look at an example of something, because I think it's important to understand all of these different aspects. Recently, I think actually still in the news, we found that some logging software distributed by Apache that's widely used in people's websites to gather information about... To help from a security perspective and to help developers improve things that are going on in websites. A vulnerability was discovered. I guess, first Alibaba, some folks were reported it directly to some folks at Apache and the Apache Organization. And then of all people, some folks from Minecraft mentioned it in a blog. That seems like a crazy way to find out about something that's a critical flaw. Now we're looking at this right now with hindsight. So with hindsight, what could we have done to not be in the circumstances that we're in right now? Vincent, I'll toss that to you first, but again, if Luke is more appropriate, let us know. >> No, it's a great question, and it's a hard question. >> How did you let this happen, Vincent? How did you let this happen? >> It wasn't me, I promise. (Dave laughs) >> What I mean, it's a challenging question I mean, and there's a number of areas where we focused on a lot of what we perceived as critical software. So it comes to web server applications, DNS, a number of the kind of the critical infrastructure that powers the internet. Right or wrong. Do we look at logging software as a critical piece of that? Well, maybe, maybe we should, right? Logging is definitely important as part of an incident response or just an awareness of what's going on. So, I mean, yeah, it probably should have been considered critical software, but I mean, it's open source, right? So there's a number of different logging applications. I imagine now we're scrutinizing those a little bit more, but looking beforehand, how do you determine what's critical until an event like this happens, and it's unfortunate that it happens. And I like to think of these as learning opportunities, and certainly not just for Red Hat, but for this (talking over each other) >> Certainly this is not... Yeah, this is not an indictment of our entire industry. We are all in this together and learning every day. It just highlights how complex the situation is that we're dealing with, right? >> It really is. And I mean, a lot of what we're looking at now is how do we get tools into the hands of developers who can catch some of these things earlier. And there's a lot of commercial offerings, there's a lot of open source tools that are available and being produced that are going to help with these sorts of situations moving forward. But I mean, all the tools on the planet aren't going to help if they're not being used. So, I mean, there has to be an education and an incentive for these developers, particularly, maybe in some upstream communities where they are labors of love and they're passionate projects they're not sponsored or backed by a corporation who's paying for these tools, to be able to use some of them and move that forward. I think that looking at things now, there is work to be done. Obviously there's always going to be work to be done. Not all of these tools, and we have to recognize this, they're not all perfect. They're not going to catch everything. These tools could have been... I mean, I don't know if they were running these tools or not, they could have been, and the tool simply could not have picked them up. So part of it is the proactive part. We talk a lot about shift left and moving these things earlier into the development process and that's great, and we should do it. It certainly should never be seen as a silver bullet or a replacement for a good response. And I think the really important thing to highlight with respect to this, and I mean, this touches on the supply chain issue as well, companies, especially those who never maybe saw themselves as a software development company really have to figure out and understand how to do appropriate response. Part of that is awareness, what do you have installed? Part of it is sources of information. Like how do I find out about a new vulnerability or a potential vulnerability? And then it's just the speed to respond. We know that a number of companies they have, maybe it's a Patch Tuesday, maybe it's a patch 26th of the month, maybe it's patch day of the quarter, we have to learn how to respond to these things quickly so that we can apply these mitigations and these fixes as quickly as possible to them protect ourselves and protect the end users or customers that we have, or to keep the kids from using some backdoors in Minecraft is the word. >> (laughs) Yeah. Look, this is an immensely important subject. To wrap us up on this, Luke, I'd like you to pretend that you just got into an elevator in a moderately tall building, and you have 60 seconds to share with me someone who already trusts you, you don't have to convince me of your credentials or anything. I trust you. What tools specifically do you need me to be running, tools and processes. You've got 60 seconds to say, Dave, if you're not doing these things right now, you're unnecessarily vulnerable. So ready, and go, Luke. >> So automatically update all packages. Always stay up-to-date so that when an issue does hit, you're not having to go back 10 versions and work your way forward. That's the key thing. Ensure that everything you pull in, you're not going to have 100%, but have a very strict requirement that there is non-repudiation, is signed content, so you can verify that it's not being tampered with. For your developers that are producing code, run static, dynamic analysis, API fuzzes, all of these sorts of tools. They will find some vulnerabilities for you. Be part of communities. Be part of communities, help chop the wood and carry the water because the log for Jay, the thing is that was found because it was in the open. If it wasn't any open, it wouldn't have been found. And I've been in this business for a long time. Software developers will always write bugs. I do. Some of them will be security bugs. That's never going to change. So it's not about stopping something that's inevitable. It's about being prepared to react accordingly in our right and correct manner when it does happen so that you can mitigate against those risks. >> Well, we're here on the 35th floor. That was amazing. Thank you, Luke. Vincent, you were in the elevator also listening in on this conversation. Did we miss anything? >> No, I mean, the only thing I'll say is that it's really helpful to partner with an enterprise open source provider, be it Red Hat or anybody else. I don't want to toot our own horn. They do a lot of that work on your behalf that you don't have to do. A lot of the things that Luke was talking about, those providers do, so you don't have to. And that's where you.. I liked that you talked about, hey, you don't have to convince me that I'm trusted, or that I trust you. Trust those vendors. They're literally here to do a lot of that heavy lifting for you and trust the process. Yeah, it's a very, very good point. And I know that sometimes it's hard to get to that point where you are the trusted advisor. Both of you certainly are. And with that, I would like to thank you very much for an interesting conversation. Gentlemen, let's keep in touch. You're always welcome on theCUBE. Luke, second time, getting a chance to talk to you on theCUBE personally. Fantastic. With that, I would like to thank everyone for joining this very special series on theCUBE. Managing risk in the digital supply chain is a critical topic to keep on top of. Thanks for tuning into theCUBE. We'll be back soon. I'm Dave Nicholson saying, thanks again. (upbeat music)

(upbeat music) >> Welcome to theCUBE. I'm Dave Nicholson, and this is part of the continuing conversation about Managing Risk in the Digital Supply Chain. I have with me today Vincent Danen, vice president of product security from Red Hat and Luke Hines security engineering lead from the office of the CTO at Red Hat. Gentlemen, welcome to theCUBE. >> Thank you. >> Great to be here. >> So let's just start out and dive right into this, Vincent, what is the software or digital supply chain? What are we talking about? Yeah, that's a good question. Software supply chain is basically the software that an end user would get from a vendor or in our case, we're talking about open source, so upstream. It is the software that comes in that is part of your package, operating system, applications. It could be something that you get from one vendor, multiple vendors. So we look at in the example of Red Hat, we are one part of the customer's software supply chain. >> So it's interesting that it's coming in from different areas. Do we have a sense for the ratio of kind of commercial software versus open source software that makes up an enterprise today? >> I think that's a really hard thing to answer and I think every enterprise or every company would have a little bit different. Depends if you have an open source vendor that you choose, you may get a significant amount of software from them. Certainly you're not going to get it all. As an example, Red Hat provides thousands of open source packages. We certainly can't provide all of them. There are millions that are out there. So when you're looking at a specific application that you're building, chances are, you could be running that on a managed platform or an enterprise supply platform, but there are going to be packages that you're going to be obtaining from other sources in other communities as well in order to power your applications. >> So, Luke, that sounds like a kind of a vague situation we're looking at in terms of where all of our software is coming from. So what do we need to know about our software supply chain in that context? What do we need to understand? Before we even get anywhere near the idea of securing it, what are some of the issues that arise from that? >> Yeah, so Vincent's touchpoint is a very wide range in ecosystem, multiple sources when we're talking about open source. So essentially awareness is key really. I think a lot of people are really not aware of the sources that they're drawing from to create their own supply chain. So there's multiple supply chains. You can be somebody like Red Hat that the provide software, and then people will leverage Red Hats for their own supply chain. And then you have the cloud provider and they have their own source of software. So I think that the key thing is the awareness of how much you rely upon that ecosystem before we look at the security of the supply chain. It's really understanding your supply chain. >> And just to follow up on that. So can you... I'm sort of checking my own level of understanding on this subject. When you talk about open source code, you're talking about a code base that is often maintained essentially by volunteers, isn't that correct? >> A mix of volunteers and paid professionals where a company has an interest in the open source project, but predominantly I would say it's... Well, I'm not entirely sure, but volunteers make up a substantial part of the ecosystem that is for sure. So it's a mix really. Some people do it because they enjoy writing software. They want to share software. Other people also enjoy working software, but they're in the position that a company pays for them to work on that software. So it's a mix of both. >> Vincent, give us a reminder of reminder of why this is important from a little bit of a higher level. Step back from the data center view of things, from the IT view of things, just from a societal perspective, Vincent, what happens when we don't secure our digital supply chain? What are the things that are put at risk? >> Okay, well, there's a significant number of things that are placed at risk, the security of the enterprise itself. So your own customer data, your own internal corporate data is place at risk if there were a supply chain breach. But further to that for a software provider, and I think that in a lot of cases, most companies today are software providers or software developers. You actually put your own customers at risk as well, not just their data, but their actual... The things that they're working on, any workloads that they may have, an order that they might place as an example. So there's a number of areas where you want to have the security of that supply chain and the software components that you have figured out. You want to be on top of that because there is that risk that trickles down when it comes to an event. I mean, we've seen that with breaches earlier this year, one company is breached multiple companies end up being breached as a result of that. So it's really important. I think we all have a part to play in that I always view it as it's not just about the company itself. So I mean, speaking from a Red Hat perspective, I don't look at it as we're just securing Red Hat, we're securing our customers, and then we're also doing that for their customers as well, because they're writing software that's running on the software that we're providing to them. So there is this trickle down effect that comes, and so I think that every link in that chain, I mean, it's wonderful that it's called a supply chain. It's only as strong as its weakest link. So our view is how do we strengthen every link in that chain? And we're one part of it, but we're kind of looking a little broader, what can we do upstream and how can we help our customers to ensure the security of their part in that supply chain? >> Yeah, I want to talk about that in a broad sense, but let's see if we can get a little bit more specific in terms of what some of the chains look like because it's not just really one chain when you think about it, there's the idea of inherent flaws that can be caught and then there are the things that bad actors might be doing to leverage those flaws. So you've got all of these different things that are converging. So first and Vincent, if you want to toss this to Luke back and forth, it's up to you guys. What about this issue of inherent flaws in code? We referenced this idea of the maintainer community. What are best practices for locking that down to make sure that there aren't inherent flaws or security risks? >> I'll take a stab at it, and then I'll let Luke follow up with maybe some of the technologies that Red Hat provides. And again, speaking to Red Hat as part of that chain. When we're talking about inherent risk, there's a vulnerability that's present upstream. We pull that software to Red Hat. We package it as a component of one of the pieces of software that we provide to our customers. It's our responsibility to pay attention to those upstream potential vulnerabilities, potential risks, and correct them in our code. So that might be taking a patch from upstream, applying it to our software, might be grabbing the latest version from upstream, whatever the case might be, but it's our responsibility to provide that protection for that software to actually remediate that risk, and then our customers can then install the update and apply the mitigation themselves. If we take a look at it from, when we're looking at multiple suppliers where you'd asked earlier about, what part of it is Red Hat and what part of it is self-service open source? When you look at that, the work that Red Hat's doing there as a commercial provider of open source and end user for that little bit that they're going to grab themselves, that Red Hat doesn't provide, it's going to have to do all of those things as well. They're going to have to pay attention to that risk from upstream. They're going to have to pay attention to any potential vulnerabilities and pull that in to figure out, do I need to patch? Where do I need to patch it? And that's something we didn't really touch on was an inventory of the software that you have in place. I mean, you don't know that you need to fix something. You don't even know that it's running. So, I mean, there's a lot of considerations there where you have to pay attention to a lot of sources. Certainly there's a metadata automation, all of these things that make it easier, but it doesn't absolve us of the responsibility across the board to pay attention to these things, whether you're grabbing it from upstream directly or from the vendor. And it's the vendor's responsibility to then be paying attention to things upstream. >> Yeah, so Luke, I want you to kind of riff on that from the perspective that let's just assume that Vincent was just primarily talking about the idea that, okay, we've established that this code is solid and we've got gold copy of it and we know it's okay. There aren't inherent problems in the code as far as we can tell. Well, that's fine. I'm a developer. I go out to pull code and to use. How do I know if it's not been tampered with? How do I know if it's in fact the code that was validated during this process before? What do you do about that? >> So there's several methods there, but I just like to loop back to that point, because I think this is really interesting around, so if you look at a software supply chain, this is a mix of humans and machines, and both have flaws, probably humans a bit more. And a supply chain, you have developers. You have code reviewers, you have your systems administrators that set up the systems, and then you have your machine actors. So you've got your build systems, the various machines that are part of that supply chain. Now the humans, there's a as an attack factor there 'cause typically they will have some sort of identity, which they leverage for access to the supply chain. So quite often a developer's identity can be compromised. So a lot of the time people will have a corporate account that gives them some sort of single sign on access to multiple systems. So the developers are coming and this could be somebody in the community as well. Their account is compromised, then they're able to easily backdoor systems. So that's one aspect. And then there is machines as well. There's the whole premise of machines software not being up to date. So when the latest nasty vulnerability is released, machines are updated, then the machines have their flaws. They can be exploited. So I would say it's not just a technical problem. There is a humanistic element to this as well around protecting your supply chain. And I would say a really good perspective to carry when you're looking to, how do I secure my supply chain is treat it like you would a production system. So what do I mean by that? When we put something into production and we've got this very long legacy of treating it with a very strict security context around who can access that people, okay. How much it's upgraded and it's patched? And we seem to not have this same perception around our supply chain and our build systems, the integrity of those, the access of those, the policy around the access and so forth. So that's one giveaway that I would say is a real key focus that you should have is treat it like a production system. Be very mindful about what you're bringing in, who can access it because it is the keys to the kingdom, because if somebody compromises your supply chain, your build systems and so forth, they can compromise the whole chain because the chain is only as strong as the weakest link. So that's what I draw upon it. And around the verifications, there is multiple technologies that you can leverage. So Red Hat, we've got a very robust sign in system that we use so that you can be sure that the packages that we get you have non-repudiation that they've been produced by Red Hat. When you update your system, that's automatically looked after. And there are other systems as well, there's other new technologies that are starting to get a foothold around the provenance of aspects of your build system. So when you're pulling in from these multiple sources of open source communities, you can have some provenance around what you're putting in as well. And yeah, I don't want to bite share too much on the technologies, but there's some exciting stuff starting to happen there as well. >> So let's look at an example of something, because I think it's important to understand all of these different aspects. Recently, I think actually still in the news, we found that some logging software distributed by Apache that's widely used in people's websites to gather information about... To help from a security perspective and to help developers improve things that are going on in websites. A vulnerability was discovered. I guess, first Alibaba, some folks were reported it directly to some folks at Apache and the Apache Organization. And then of all people, some folks from Minecraft mentioned it in a blog. That seems like a crazy way to find out about something that's a critical flaw. Now we're looking at this right now with hindsight. So with hindsight, what could we have done to not be in the circumstances that we're in right now? Vincent, I'll toss that to you first, but again, if Luke is more appropriate, let us know. >> No, it's a great question, and it's a hard question. >> How did you let this happen, Vincent? How did you let this happen? >> It wasn't me, I promise. (Dave laughs) >> What I mean, it's a challenging question I mean, and there's a number of areas where we focused on a lot of what we perceived as critical software. So it comes to web server applications, DNS, a number of the kind of the critical infrastructure that powers the internet. Right or wrong. Do we look at logging software as a critical piece of that? Well, maybe, maybe we should, right? Logging is definitely important as part of an incident response or just an awareness of what's going on. So, I mean, yeah, it probably should have been considered critical software, but I mean, it's open source, right? So there's a number of different logging applications. I imagine now we're scrutinizing those a little bit more, but looking beforehand, how do you determine what's critical until an event like this happens, and it's unfortunate that it happens. And I like to think of these as learning opportunities, and certainly not just for Red Hat, but for this (talking over each other) >> Certainly this is not... Yeah, this is not an indictment of our entire industry. We are all in this together and learning every day. It just highlights how complex the situation is that we're dealing with, right? >> It really is. And I mean, a lot of what we're looking at now is how do we get tools into the hands of developers who can catch some of these things earlier. And there's a lot of commercial offerings, there's a lot of open source tools that are available and being produced that are going to help with these sorts of situations moving forward. But I mean, all the tools on the planet aren't going to help if they're not being used. So, I mean, there has to be an education and an incentive for these developers, particularly, maybe in some upstream communities where they are labors of love and they're passionate projects they're not sponsored or backed by a corporation who's paying for these tools, to be able to use some of them and move that forward. I think that looking at things now, there is work to be done. Obviously there's always going to be work to be done. Not all of these tools, and we have to recognize this, they're not all perfect. They're not going to catch everything. These tools could have been... I mean, I don't know if they were running these tools or not, they could have been, and the tool simply could not have picked them up. So part of it is the proactive part. We talk a lot about shift left and moving these things earlier into the development process and that's great, and we should do it. It certainly should never be seen as a silver bullet or a replacement for a good response. And I think the really important thing to highlight with respect to this, and I mean, this touches on the supply chain issue as well, companies, especially those who never maybe saw themselves as a software development company really have to figure out and understand how to do appropriate response. Part of that is awareness, what do you have installed? Part of it is sources of information. Like how do I find out about a new vulnerability or a potential vulnerability? And then it's just the speed to respond. We know that a number of companies they have, maybe it's a Patch Tuesday, maybe it's a patch 26th of the month, maybe it's patch day of the quarter, we have to learn how to respond to these things quickly so that we can apply these mitigations and these fixes as quickly as possible to them protect ourselves and protect the end users or customers that we have, or to keep the kids from using some backdoors in Minecraft is the word. >> (laughs) Yeah. Look, this is an immensely important subject. To wrap us up on this, Luke, I'd like you to pretend that you just got into an elevator in a moderately tall building, and you have 60 seconds to share with me someone who already trusts you, you don't have to convince me of your credentials or anything. I trust you. What tools specifically do you need me to be running, tools and processes. You've got 60 seconds to say, Dave, if you're not doing these things right now, you're unnecessarily vulnerable. So ready, and go, Luke. >> So automatically update all packages. Always stay up-to-date so that when an issue does hit, you're not having to go back 10 versions and work your way forward. That's the key thing. Ensure that everything you pull in, you're not going to have 100%, but have a very strict requirement that there is non-repudiation, is signed content, so you can verify that it's not being tampered with. For your developers that are producing code, run static, dynamic analysis, API fuzzes, all of these sorts of tools. They will find some vulnerabilities for you. Be part of communities. Be part of communities, help chop the wood and carry the water because the log for Jay, the thing is that was found because it was in the open. If it wasn't any open, it wouldn't have been found. And I've been in this business for a long time. Software developers will always write bugs. I do. Some of them will be security bugs. That's never going to change. So it's not about stopping something that's inevitable. It's about being prepared to react accordingly in our right and correct manner when it does happen so that you can mitigate against those risks. >> Well, we're here on the 35th floor. That was amazing. Thank you, Luke. Vincent, you were in the elevator also listening in on this conversation. Did we miss anything? >> No, I mean, the only thing I'll say is that it's really helpful to partner with an enterprise open source provider, be it Red Hat or anybody else. I don't want to toot our own horn. They do a lot of that work on your behalf that you don't have to do. A lot of the things that Luke was talking about, those providers do, so you don't have to. And that's where you.. I liked that you talked about, hey, you don't have to convince me that I'm trusted, or that I trust you. Trust those vendors. They're literally here to do a lot of that heavy lifting for you and trust the process. Yeah, it's a very, very good point. And I know that sometimes it's hard to get to that point where you are the trusted advisor. Both of you certainly are. And with that, I would like to thank you very much for an interesting conversation. Gentlemen, let's keep in touch. You're always welcome on theCUBE. Luke, second time, getting a chance to talk to you on theCUBE personally. Fantastic. With that, I would like to thank everyone for joining this very special series on theCUBE. Managing risk in the digital supply chain is a critical topic to keep on top of. Thanks for tuning into theCUBE. We'll be back soon. I'm Dave Nicholson saying, thanks again. (upbeat music)