>> theCube presents Dell Technologies World, brought to you by Dell. >> Hi everyone, welcome back to Dell Technologies World 2022. You're watching theCube's coverage of this, three-day coverage wall to wall. My name is David Vellante John Furrier's here, Lisa Martin, David Nicholson. Talk of the town here is data. And one of the big announcements at the show is Snowflake and Dell partnering up, building ecosystems. Snowflake reaching into on-prem, allowing customers to actually access the Snowflake Data Cloud without moving the data or if they want to move the data they can. This is really one of the hotter announcements of the show. Martin Glynn is here, he's the Senior Director of Storage Product Management at Dell Technologies. And Clark Patterson, he's the Head of Product Marketing for Snowflake. Guys, welcome. >> Thanks for having us. >> So a lot of buzz around this and, you know, Clark, you and I have talked about the need to really extend your data vision. And this really is the first step ever you've taken on-prem. Explain the motivation for this from your customer's perspective. >> Yeah. I mean, if you step back and think about Snowflake's vision and our mission of mobilizing the world's data, it's all around trying to break down silos for however customers define what a silo is, right? So we've had a lot of success breaking down silos from a workload perspective where we've expanded the platform to be data warehousing, and data engineering, and machine learning, and data science, and all the kind of compute intensive ways that people work with us. We've also had a lot of success in our sharing capabilities and how we're breaking down silos of organizations, right? So I can share data more seamlessly within my team, I can do it across totally disparate organizations, and break down silos that way. So this partnership is really like the next leg of the stool, so to speak, where we're breaking down the silos of the the data and where the data lives ultimately, right? So up until this point, Cloud, all focus there, and now we have this opportunity with Dell to expand that and into on-premises world and people can bring all those data sets together. >> And the data target for this Martin, is Dell ECS, right? Your object store, and it's got S3 compatibility. Explain that. >> Yeah, we've actually got sort of two flavors. We'll start with ECS, which is our turnkey object storage solution. Object storage offers sort of the ultimate in flexibility, you know, potential performance, ease of use, right? Which is why it fits so well with Snowflake's mission for sort of unlocking, you know, the data within the data center. So we'll offer it to begin with ECS, and then we also recently announced our software defined object scale solution. So add even more flexibility there. >> Okay. And the clock, the way it works is I can now access non-native Snowflake data using what? Materialized views, external tables, how does that work? >> Some combination of all the above. So we've had in Snowflake a capability called external tables which we refer to, it goes hand in hand with this notion of external stages. Basically through the combination of those two capabilities, it's a metadata layer on data wherever it resides. So customers have actually used this in Snowflake for data lake data outside of Snowflake in the Cloud up until this point. So it's effectively an extension of that functionality into the Dell on-premises world, so that we can tap into those things. So we use the external stages to expose all the metadata about what's in the Dell environment. And then we build external tables in Snowflake so that data looks like it is in Snowflake. And then the experience for the analyst or whomever it is, is exactly as though that data lives in the Snowflake world. >> Okay. So for a while you've allowed non-native Snowflake data but it had to be in the Cloud. >> Correct. >> It was the first time it's on-prem, >> that's correct >> that's the innovation here. Okay. And if I want to bring it into the Cloud, can I? >> Yeah, the connection here will help in a migration sense as well, right? So that's the good thing is, it's really giving the user the choice. So we are integrating together as partners to make connection as seamless as possible. And then the end user will say like, look I've got data that needs to live on-premises, for whatever reasons, data sovereignty whatever they decide. And they can keep it there and still do the analytics in another place. But if there's a need and a desire to use this as an opportunity to migrate some of that data to Cloud, that connection between our two platforms will make that easier. >> Well, Michael always says, "Hey, it's customer choice, we're flexible." So you're cool with that? That's been the mission since we kind of came together, right? Is if our customers needed to stay in their data center, if that makes more sense from a cost perspective or, you know, a data gravity perspective, then they can do that. But we also want to help them unlock the value of that data. So if they need to copy it up to the public Cloud and take advantage of it, we're going to integrate directly with Snowflake to make that really easy to do. >> So there are engineering integrations here, obviously that's required. Can you describe what that looks like? Give us the details on when it's available. >> Sure. So it's going to be sort of second half this year that you'll see, we're demoing it this week, but the availability we second half this year. And fundamentally, it's the way Clark described it, that Snowflake will reach into our S3 interface using the standard S3 interface. We're qualifying between the way they expect that S3 interface to present the data and the way our platform works, just to ensure that there's smooth interaction between the two. So that's sort of the first simplest use case. And then the second example we gave where the customer can copy some of that data up to the public Cloud. We're basically copying between two S3 buckets and making sure that Snowflake's Snowpipe is aware that data's being made available and can easily ingest it. >> And then that just goes into a virtual warehouse- >> Exactly. >> and customer does to know or care. >> Yep Exactly. >> Yeah. >> The compute happens in Snowflake the way it does in any other manner. >> And I know you got to crawl, walk, run second half of this year, but I would imagine, okay, you're going to start with AWS, correct? And then eventually you go to other Clouds. I mean, that's going to take other technical integrations, I mean, obviously. So should we assume there's a roadmap here or is this a one and done? >> I would assume that, I mean, based on our multi-Cloud approach, that's kind of our approach at least, yeah. >> Kind of makes sense, right? I mean, that would seem to be a natural progression. My other thought was, okay, I've got operational systems. They might be transaction systems running on a on a PowerMax. >> Yeah. >> Is there a way to get the data into an object store and make that available, now that opens up even more workloads. I know you're not committing to doing that, but it just, conceptually, it seems like something a customer might want to do. >> Yeah. I, a hundred percent, agree. I mean, I think when we brought our team together we started with a blank slate. It was what's the best solution we can build. We landed on this sort of first step, but we got lots of feedback from a lot of our big joint customers about you know, this system over there, this potential integration over here, and whether it's, you know, PowerMax type systems or other file workloads with native Snowflake data types. You know, I think this is just the beginning, right? We have lots of potential here. >> And I don't think you've announced pricing, right? It's premature for that. But have you thought about, and how are you thinking about the pricing model? I mean, you're a consumption based pricing, is that kind of how this is going to work? Or is it a sort of a new pricing model or haven't you figured that out yet? >> I don't know if you've got any details on that, but from a Snowflake perspective, I would assume it's consistent with how our customers engage with us today. >> Yeah. >> And we'll offer both possibilities, right? So you can either continue with the standard, you know, sort of CapEx motion, maybe that's the most optimal for you from a cost perspective, or you can take advantage through our OpEx option, right? So you can do consumption on-prem also. >> Okay. So it could be a dual model, right? Depending on what the customer wants. If they're a Snowflake customer, obviously it's going to be consumption based, however, you guys price. What's happening, Clark, in in the market? Explain why Snowflake has so much momentum and, you know, traction in the marketplace. >> So like I spent a lot of time doing analysis on why we win and lose, core part of my role. And, you know, there's a couple of, there's really three things that come up consistently as to why people people are really excited about Snowflake platform. One is the most simplest thing of all. It feels like is just ease of use and it just works, right? And I think the way that this platform was built for the Cloud from the ground up all the way back 10 years ago, really a lot allows us to deliver that seamless experience of just like instant compute when you want it, it goes away, you know, only pay for what you use. Very few knobs to turn and things like that. And so people absolutely love that factor. The other is multi-Cloud. So, you know, there's definitely a lot of organizations out there that have a multi-Cloud strategy, and, you know, what that means to them can be highly variable, but regardless, they want to be able to interact across Clouds in some capacity. And of course we are a single platform, like literally one single interface, consistent across all the three Cloud providers that we work upon. And it gives them that flexibility to mix and match Cloud infrastructure under any Snowflake however they see fit. The last piece of it is sharing. And, you know, I think it's that ability as I kind of alluded to around like breaking down organizational silos, and allow people to be able to actually connect with each other in ways that you couldn't do before. Like, if you think about how you and I would've shared data before, I'd be like, "Hey, Dave, I'm going to unload this table into a spreadsheet and I'm going to send it over in email." And there's the whole host of issues that get introduced in that and world, now it's like instantly available. I have a lot of control over it, it's governed it's all these other things. And I can create kind of walled gardens, so to speak, of how far out I want that to go. It could be in a controlled environment of organizations that I want to collaborate with, or I can put it on our marketplace and expose it to the whole world, because I think there's a value in that. And if I choose I can monetize it, right? So those, you know, the ease of use aspect of it, absolutely, it's just a fantastic platform. The multi-Cloud aspect of it and our unique differentiation around sharing in our marketplace and monetization. >> Yeah, on the sharing front. I mean, it's now discoverable. Like if you send me an email, like what'd you call that? When did you send that email? And then the same time I can forward that to somebody else's not governed. >> Yeah. >> All right. So that just be creates a nightmare for the compliance. >> Right. Yeah. You think about how you revoke access in that situation. You just don't, right? Now I can just turn it off and you go in to run your query. >> Don't get access on that data anymore. Yeah. Okay. And then the other thing I wanted to ask you, Clark is Snowflake started really as analytics platform, simplifying data warehousing, you're moving into that world of data science, you know, the whole data lake movement, bringing those two worlds together. You know, I was talking to Ben Ward about this, maybe there's a semantic layer that helps us kind of talk between those two worlds, but you don't care, right? If it's in an object store, it can play in both of those worlds, right? >> That's right. >> Yeah, it's up to you to figure it out and the customer- >> Yeah. >> from a storage standpoint. Here it is, serve it up. >> And that's the thrust of this announcement, right? Is bringing together two great companies, the Dell platform, the Snowflake platform, and allowing organizations to bring that together. And they decide like it, as we all know, customers decide how they're going to build their architecture. And so this is just another way that we're helping them leverage the capabilities of our two great platforms. >> Does this push or pull or little bit of both? I mean, where'd this come from? Or customers saying, "Hey, it would be kind of cool if we could have this." Or is it more, "Hey, what do you guys think?" You know, where are you at with that? >> It was definitely both, right? I mean, so we certainly started with, you know, a high level idea that, you know, the technologies are complimentary, right? I mean, as Clark just described, and at the same time we had customers coming to us saying, "Hey, wait a minute, I'm doing this over here, and this over here, how can I make this easier?" So that was like I said, we started with a blank sheet and lots of long customer conversations and this is what resulted. So >> So what are the sequence of events to kind of roll this out? You said it's second half, you know, when do you start getting customers involved? Do you have your already, you know, to poke at this and what's that look like? >> Yeah, sure. I can weigh in there. So, absolutely. We've had a few of our big customers that have been involved sort of in the design already who understand how they want to use it. So I think our expectation is that now that the sort of demonstrations have been in place, we have some pre functionality, we're going to see some initial testing and usage, some beta type situations with our customers. And then second half, we'll ramp from there. >> It's got to be a huge overlap between Dell customers and Snowflake customers. I mean, it's hundred billion. You can't not bump into Dell somewhere. >> Exactly. Yeah, you know. >> So where do you guys want to see this relationship go, kind of how should we measure success? Maybe you could each give your perspectives of that. >> I mean, for us, I think it's really showing the value of the Snowflake platform in this new world where there's a whole new ecosystem of data that is accessible to us, right? So seeing those organizations that are saying like, "Look, I'm doing new things with on-premises data that I didn't think that I could do before", or, "I'm driving efficiency in how I do analytics, and data engineering, and data science, in ways that I couldn't do before," 'cause they were locked out of using a Snowflake-like technology, right? So I think for me, that's going to be that real excitement. I'm really curious to see how the collaboration and the sharing component comes into this, you know, where you can think of having an on-premises data strategy and a need, right? But you can really connect to Cloud native customers and partners and suppliers that live in the Snowflake ecosystem, and that wasn't possible before. And so that is very conceivable and very possible through this relationship. So seeing how those edges get created in in our world and how people start to collaborate across data, both in the Cloud and on-prem is going to be really exciting. >> I remember I asked Frank, it was kind of early in the pandemic. I asked him, come on, tell me about how you're managing things. And he was awesome. And I asked him to at the time, you know, "You're ever going to do, you know, bring this platform on-prem?" He's like unequivocal, "No way, that's never going to happen. We're not going to do it halfway house ware Cloud only." And I kept thinking, but there's got to be a way to expand that team. There's so much data out there, and so boom, now we see the answer . Martin, from your standpoint, what does success look like? >> I think it starts with our partnership, right? So I've been doing this a long time. Probably the first time I've worked so closely with a partner like Snowflake. Joint customer conversations, joint solutioning, making sure what we're building is going to be really, truly as useful as possible to them. And I think we're going to let them guide us as we go forward here, right? You mentioned, you know, systems or record or other potential platforms. We're going to let them tell us where exactly the most value will come from the integration between the two companies. >> Yeah. Follow data. I mean, remember in the old days a hardware company like Dell would go to an ISP like Snowflake and say, "Hey, we ran some benchmarks. Your software runs really fast on our hardware, can we work together?" And you go, "Yeah, of course. Yeah, no problem." But wow! What a different dynamic it is today. >> Yeah. Yeah, absolutely. >> All right guys. Hey, thanks so much for coming to theCube. It's great to see you. We'll see you at the Snowflake Summit in June. >> Snowflake Summit in a month and a half. >> Looking forward to that. All right. Thank you again. >> Thank you Dave. >> All right. Keep it right there everybody. This is Dave Vellante, wall to wall coverage of Dell Tech World 2022. We'll be right back. (gentle music)

>> theCube presents Dell Technologies World, brought to you by Dell. >> Hi everyone, welcome back to Dell Technologies World 2022. You're watching theCube's coverage of this, three-day coverage wall to wall. My name is David Vellante John Furrier's here, Lisa Martin, David Nicholson. Talk of the town here is data. And one of the big announcements at the show is Snowflake and Dell partnering up, building ecosystems. Snowflake reaching into on-prem, allowing customers to actually access the Snowflake Data Cloud without moving the data or if they want to move the data they can. This is really one of the hotter announcements of the show. Martin Glynn is here, he's the Senior Director of Storage Product Management at Dell Technologies. And Clark Patterson, he's the Head of Product Marketing for Snowflake. Guys, welcome. >> Thanks for having us. >> So a lot of buzz around this and, you know, Clark, you and I have talked about the need to really extend your data vision. And this really is the first step ever you've taken on-prem. Explain the motivation for this from your customer's perspective. >> Yeah. I mean, if you step back and think about Snowflake's vision and our mission of mobilizing the world's data, it's all around trying to break down silos for however customers define what a silo is, right? So we've had a lot of success breaking down silos from a workload perspective where we've expanded the platform to be data warehousing, and data engineering, and machine learning, and data science, and all the kind of compute intensive ways that people work with us. We've also had a lot of success in our sharing capabilities and how we're breaking down silos of organizations, right? So I can share data more seamlessly within my team, I can do it across totally disparate organizations, and break down silos that way. So this partnership is really like the next leg of the stool, so to speak, where we're breaking down the silos of the the data and where the data lives ultimately, right? So up until this point, Cloud, all focus there, and now we have this opportunity with Dell to expand that and into on-premises world and people can bring all those data sets together. >> And the data target for this Martin, is Dell ECS, right? Your object store, and it's got S3 compatibility. Explain that. >> Yeah, we've actually got sort of two flavors. We'll start with ECS, which is our turnkey object storage solution. Object storage offers sort of the ultimate in flexibility, you know, potential performance, ease of use, right? Which is why it fits so well with Snowflake's mission for sort of unlocking, you know, the data within the data center. So we'll offer it to begin with ECS, and then we also recently announced our software defined object scale solution. So add even more flexibility there. >> Okay. And the clock, the way it works is I can now access non-native Snowflake data using what? Materialized views, external tables, how does that work? >> Some combination of all the above. So we've had in Snowflake a capability called external tables which we refer to, it goes hand in hand with this notion of external stages. Basically through the combination of those two capabilities, it's a metadata layer on data wherever it resides. So customers have actually used this in Snowflake for data lake data outside of Snowflake in the Cloud up until this point. So it's effectively an extension of that functionality into the Dell on-premises world, so that we can tap into those things. So we use the external stages to expose all the metadata about what's in the Dell environment. And then we build external tables in Snowflake so that data looks like it is in Snowflake. And then the experience for the analyst or whomever it is, is exactly as though that data lives in the Snowflake world. >> Okay. So for a while you've allowed non-native Snowflake data but it had to be in the Cloud. >> Correct. >> It was the first time it's on-prem, >> that's correct >> that's the innovation here. Okay. And if I want to bring it into the Cloud, can I? >> Yeah, the connection here will help in a migration sense as well, right? So that's the good thing is, it's really giving the user the choice. So we are integrating together as partners to make connection as seamless as possible. And then the end user will say like, look I've got data that needs to live on-premises, for whatever reasons, data sovereignty whatever they decide. And they can keep it there and still do the analytics in another place. But if there's a need and a desire to use this as an opportunity to migrate some of that data to Cloud, that connection between our two platforms will make that easier. >> Well, Michael always says, "Hey, it's customer choice, we're flexible." So you're cool with that? That's been the mission since we kind of came together, right? Is if our customers needed to stay in their data center, if that makes more sense from a cost perspective or, you know, a data gravity perspective, then they can do that. But we also want to help them unlock the value of that data. So if they need to copy it up to the public Cloud and take advantage of it, we're going to integrate directly with Snowflake to make that really easy to do. >> So there are engineering integrations here, obviously that's required. Can you describe what that looks like? Give us the details on when it's available. >> Sure. So it's going to be sort of second half this year that you'll see, we're demoing it this week, but the availability we second half this year. And fundamentally, it's the way Clark described it, that Snowflake will reach into our S3 interface using the standard S3 interface. We're qualifying between the way they expect that S3 interface to present the data and the way our platform works, just to ensure that there's smooth interaction between the two. So that's sort of the first simplest use case. And then the second example we gave where the customer can copy some of that data up to the public Cloud. We're basically copying between two S3 buckets and making sure that Snowflake's Snowpipe is aware that data's being made available and can easily ingest it. >> And then that just goes into a virtual warehouse- >> Exactly. >> and customer does to know or care. >> Yep Exactly. >> Yeah. >> The compute happens in Snowflake the way it does in any other manner. >> And I know you got to crawl, walk, run second half of this year, but I would imagine, okay, you're going to start with AWS, correct? And then eventually you go to other Clouds. I mean, that's going to take other technical integrations, I mean, obviously. So should we assume there's a roadmap here or is this a one and done? >> I would assume that, I mean, based on our multi-Cloud approach, that's kind of our approach at least, yeah. >> Kind of makes sense, right? I mean, that would seem to be a natural progression. My other thought was, okay, I've got operational systems. They might be transaction systems running on a on a PowerMax. >> Yeah. >> Is there a way to get the data into an object store and make that available, now that opens up even more workloads. I know you're not committing to doing that, but it just, conceptually, it seems like something a customer might want to do. >> Yeah. I, a hundred percent, agree. I mean, I think when we brought our team together we started with a blank slate. It was what's the best solution we can build. We landed on this sort of first step, but we got lots of feedback from a lot of our big joint customers about you know, this system over there, this potential integration over here, and whether it's, you know, PowerMax type systems or other file workloads with native Snowflake data types. You know, I think this is just the beginning, right? We have lots of potential here. >> And I don't think you've announced pricing, right? It's premature for that. But have you thought about, and how are you thinking about the pricing model? I mean, you're a consumption based pricing, is that kind of how this is going to work? Or is it a sort of a new pricing model or haven't you figured that out yet? >> I don't know if you've got any details on that, but from a Snowflake perspective, I would assume it's consistent with how our customers engage with us today. >> Yeah. >> And we'll offer both possibilities, right? So you can either continue with the standard, you know, sort of CapEx motion, maybe that's the most optimal for you from a cost perspective, or you can take advantage through our OpEx option, right? So you can do consumption on-prem also. >> Okay. So it could be a dual model, right? Depending on what the customer wants. If they're a Snowflake customer, obviously it's going to be consumption based, however, you guys price. What's happening, Clark, in in the market? Explain why Snowflake has so much momentum and, you know, traction in the marketplace. >> So like I spent a lot of time doing analysis on why we win and lose, core part of my role. And, you know, there's a couple of, there's really three things that come up consistently as to why people people are really excited about Snowflake platform. One is the most simplest thing of all. It feels like is just ease of use and it just works, right? And I think the way that this platform was built for the Cloud from the ground up all the way back 10 years ago, really a lot allows us to deliver that seamless experience of just like instant compute when you want it, it goes away, you know, only pay for what you use. Very few knobs to turn and things like that. And so people absolutely love that factor. The other is multi-Cloud. So, you know, there's definitely a lot of organizations out there that have a multi-Cloud strategy, and, you know, what that means to them can be highly variable, but regardless, they want to be able to interact across Clouds in some capacity. And of course we are a single platform, like literally one single interface, consistent across all the three Cloud providers that we work upon. And it gives them that flexibility to mix and match Cloud infrastructure under any Snowflake however they see fit. The last piece of it is sharing. And, you know, I think it's that ability as I kind of alluded to around like breaking down organizational silos, and allow people to be able to actually connect with each other in ways that you couldn't do before. Like, if you think about how you and I would've shared data before, I'd be like, "Hey, Dave, I'm going to unload this table into a spreadsheet and I'm going to send it over in email." And there's the whole host of issues that get introduced in that and world, now it's like instantly available. I have a lot of control over it, it's governed it's all these other things. And I can create kind of walled gardens, so to speak, of how far out I want that to go. It could be in a controlled environment of organizations that I want to collaborate with, or I can put it on our marketplace and expose it to the whole world, because I think there's a value in that. And if I choose I can monetize it, right? So those, you know, the ease of use aspect of it, absolutely, it's just a fantastic platform. The multi-Cloud aspect of it and our unique differentiation around sharing in our marketplace and monetization. >> Yeah, on the sharing front. I mean, it's now discoverable. Like if you send me an email, like what'd you call that? When did you send that email? And then the same time I can forward that to somebody else's not governed. >> Yeah. >> All right. So that just be creates a nightmare for the compliance. >> Right. Yeah. You think about how you revoke access in that situation. You just don't, right? Now I can just turn it off and you go in to run your query. >> Don't get access on that data anymore. Yeah. Okay. And then the other thing I wanted to ask you, Clark is Snowflake started really as analytics platform, simplifying data warehousing, you're moving into that world of data science, you know, the whole data lake movement, bringing those two worlds together. You know, I was talking to Ben Ward about this, maybe there's a semantic layer that helps us kind of talk between those two worlds, but you don't care, right? If it's in an object store, it can play in both of those worlds, right? >> That's right. >> Yeah, it's up to you to figure it out and the customer- >> Yeah. >> from a storage standpoint. Here it is, serve it up. >> And that's the thrust of this announcement, right? Is bringing together two great companies, the Dell platform, the Snowflake platform, and allowing organizations to bring that together. And they decide like it, as we all know, customers decide how they're going to build their architecture. And so this is just another way that we're helping them leverage the capabilities of our two great platforms. >> Does this push or pull or little bit of both? I mean, where'd this come from? Or customers saying, "Hey, it would be kind of cool if we could have this." Or is it more, "Hey, what do you guys think?" You know, where are you at with that? >> It was definitely both, right? I mean, so we certainly started with, you know, a high level idea that, you know, the technologies are complimentary, right? I mean, as Clark just described, and at the same time we had customers coming to us saying, "Hey, wait a minute, I'm doing this over here, and this over here, how can I make this easier?" So that was like I said, we started with a blank sheet and lots of long customer conversations and this is what resulted. So >> So what are the sequence of events to kind of roll this out? You said it's second half, you know, when do you start getting customers involved? Do you have your already, you know, to poke at this and what's that look like? >> Yeah, sure. I can weigh in there. So, absolutely. We've had a few of our big customers that have been involved sort of in the design already who understand how they want to use it. So I think our expectation is that now that the sort of demonstrations have been in place, we have some pre functionality, we're going to see some initial testing and usage, some beta type situations with our customers. And then second half, we'll ramp from there. >> It's got to be a huge overlap between Dell customers and Snowflake customers. I mean, it's hundred billion. You can't not bump into Dell somewhere. >> Exactly. Yeah, you know. >> So where do you guys want to see this relationship go, kind of how should we measure success? Maybe you could each give your perspectives of that. >> I mean, for us, I think it's really showing the value of the Snowflake platform in this new world where there's a whole new ecosystem of data that is accessible to us, right? So seeing those organizations that are saying like, "Look, I'm doing new things with on-premises data that I didn't think that I could do before", or, "I'm driving efficiency in how I do analytics, and data engineering, and data science, in ways that I couldn't do before," 'cause they were locked out of using a Snowflake-like technology, right? So I think for me, that's going to be that real excitement. I'm really curious to see how the collaboration and the sharing component comes into this, you know, where you can think of having an on-premises data strategy and a need, right? But you can really connect to Cloud native customers and partners and suppliers that live in the Snowflake ecosystem, and that wasn't possible before. And so that is very conceivable and very possible through this relationship. So seeing how those edges get created in in our world and how people start to collaborate across data, both in the Cloud and on-prem is going to be really exciting. >> I remember I asked Frank, it was kind of early in the pandemic. I asked him, come on, tell me about how you're managing things. And he was awesome. And I asked him to at the time, you know, "You're ever going to do, you know, bring this platform on-prem?" He's like unequivocal, "No way, that's never going to happen. We're not going to do it halfway house ware Cloud only." And I kept thinking, but there's got to be a way to expand that team. There's so much data out there, and so boom, now we see the answer . Martin, from your standpoint, what does success look like? >> I think it starts with our partnership, right? So I've been doing this a long time. Probably the first time I've worked so closely with a partner like Snowflake. Joint customer conversations, joint solutioning, making sure what we're building is going to be really, truly as useful as possible to them. And I think we're going to let them guide us as we go forward here, right? You mentioned, you know, systems or record or other potential platforms. We're going to let them tell us where exactly the most value will come from the integration between the two companies. >> Yeah. Follow data. I mean, remember in the old days a hardware company like Dell would go to an ISP like Snowflake and say, "Hey, we ran some benchmarks. Your software runs really fast on our hardware, can we work together?" And you go, "Yeah, of course. Yeah, no problem." But wow! What a different dynamic it is today. >> Yeah. Yeah, absolutely. >> All right guys. Hey, thanks so much for coming to theCube. It's great to see you. We'll see you at the Snowflake Summit in June. >> Snowflake Summit in a month and a half. >> Looking forward to that. All right. Thank you again. >> Thank you Dave. >> All right. Keep it right there everybody. This is Dave Vellante, wall to wall coverage of Dell Tech World 2022. We'll be right back. (gentle music)

>> Narrator: From around the globe, it's the CUBE, with digital coverage of Dell Technologies world. Digital experience brought to you by Dell Technologies. >> Welcome to the CUBE'S coverage of Dell Technologies World 2020, The Digital Experience. I'm Lisa Martin joined by a couple of guys from Dell Technology. Please welcome Martin Glynn, the senior director for product management for PowerMax Martin good morning. >> Good morning. >> Nice to see you. And joining Martin is Matthew Paul, the senior director of product management for PowerFlex at Dell Technologies. Matthew, nice to see you. >> Nice to see you thanks for having us Lisa. >> So our virtual cube this year can't be with you guys in person or the 14,000 other folks that usually attend at Dell Technologies World but a lot of opportunities to engage customers and partners and present analysts digitally, which is great. So Matthew, let's go ahead and start with you. Talk to us about what's new with PowerFlex, this was the kind of the end of the rebrand under the power portfolio that Dell Technologies undertook the last couple of years formerly the VXFlex excuse me, from Scale IO, what's new with PowerFlex? >> Yeah, that's a spot on. So really the idea of us aligning the full power portfolio is kind of a big deal, right? Part of the winning roadmap to at IO, kind of assigned to our customers and our field and everyone that software defined storage is a critical part of the Dell Technologies strategy. If you think about PowerFlex, just to kind of level set, it's really a software defined infrastructure kind of system that brings you the best of traditional three tier infrastructure and the best of HCI infrastructure while being able to make that experience really simple in the enterprise while still delivering exemplary really great performance and scale. In terms of new things, well, just real quick, in terms of kind of new things, we brought interesting topics like native Async replication, secure snapshots, some end to end lifecycle management pieces. So a lot of great innovation in the last year. >> And that was some of the recent announcements. Tell me Matthew, from a customer perspective since you've announced Asynchronous replication snapshots, what's the customer adoption, customer feedback been like? >> Yeah, it's been fantastic. We continue to grow this market really strong, you know, we're focusing on high end large enterprise customers working towards, bringing down also into kind of enterprise and commercial customers, so it'll make things easier to use. But very strong adoption and great investments here at Dell with this product. >> All right, so PowerFlex, Martin, let's go to you PowerMax, talk to us about PowerMax. And then also how it kind of fits into the whole power portfolio. >> Sure, yeah, so thanks Lisa. The PowerMax products, I think was the first product other than of course, the server products to be powered up in the storage portfolio, PowerMax is the sort of flagship sort of derived product that we've had now for, you know, a few decades really been a leader in mission critical data centers. But I think that pace of innovation over the last year just like Matt describing the PowerFlex side has been a really phenomenal. Just about a year ago he came out with a storage class memory, we did fiber channel Endymion over fiber channel, and more recently brought in a few really interesting new technologies, like support for replication, with VVols, cloud mobility, and now, efficient encryption. So the set of things we're enabling our customers to do with their you know, sort of traditional three tier SAN infrastructure is really just unmatched. >> So Matt talk to me about the last six seven months, where are these enterprise customers in terms of leveraging PowerMax for example, when everything just changed dramatically almost overnight. Enterprises in every industry had to suddenly remote workforce. How did PowerMax help your customers pivot and ensure that their digital transformation could support this business surviving? >> Yeah, well, like everybody we were a little worried at the outset, you know lot of uncertainty about how things would play out and the response from our customers has been amazing. You know, they've all sort of really doubled down on using our technology to support their businesses through this new model. So, you know, the business has been really amazing really incredible, and it's been great to partner with our customers that help them continue to deliver the services that they need you know, in this new model. So that part's been, been really wonderful, and as we work really closely with them, some of the things we just came out with, you know, they've helped us to design and deliver in a way that they can best take advantage of so, you know, for example the new cloud mobility functionality that's letting them take information directly off of their mission, critical sort of bedrock sand infrastructure and push it up to an object store. And that could be a local private object store, it could be a public object store like AWS. And so that's you know, it's enabling them to take advantage of some new models and a new approach to doing things. And I think ultimately that's going to help them work through this you know, new normal, we're all participating in. >> Yeah, we want to help those businesses not just survive this time, but be able to thrive, especially as we don't know how much of this remote scattered workforce is going to remain. We're hearing estimates from some of the big technology leaders at all. 50% percent of the workforce is going to remain at home so really helping organizations to maneuver and navigate these challenging landscapes is a big priority I know for Dell Technologies we talked about that with some other guests. Matthew, over to you talk to me about PowerFlex from a workloads perspective, so we can get a good idea for the workloads that it's really ideally best suited for. >> Yeah, I think wanted to just take a quick second on the COVID piece, because we have a couple of really big customers that we had to enable really quickly for curbside checkout and, you know, they were trying to run things, they were putting it on their existing infrastructure, their existing systems, and it just wasn't fast enough, it wasn't keeping up. And by working closely with the customer and designing a system with PowerFlex as the core, allowed us to enable them really quickly to turn from a customer who didn't have this idea of curbside checkout to enabling curbside checkout. So I think working and partnering closely with our customers is a critical part of how Dell Tech is successful and enabling them to kind of work through these tough times. With workloads, Yeah, oh, go ahead sorry. >> That's okay go ahead. >> I was going to say with workloads in general, the way that we have to think about them with enterprise quality or enterprise requirements is really in kind of a scheme of looking at performance, understanding scalability, ensuring we have enterprise class availability, and then last but definitely not least is like how we manage that and how we make it easier for customers to work through those. And when I think about Flex there's two or three key areas that we try to go after, if you, one of the key differentiation pieces around Flex is the fact that we can deploy it in multiple manners. So you can deploy it in an HCI mode, where you have the compute and networking together, or you can go deploy it in a dis-aggregated mode where you have compute and networking, I mean, compute and storage separate. And if those are separate that allows you to scale those independently work really, really well for key database workloads, key workloads like, let's say even like Honda, where you maybe have really high compute but little less storage requirements. So that really allows customers to dial up and down what makes the most sense for them right? The other angle that we're seeing pretty big adoption is around this idea of re-platform or realigning the data center with transformation with software defined scale all block storage. So think about deploying Powerflex in an environment and then being able to use that in a virtual environment in a physical environment, in a container environment being able to have your traditional applications like SQL or Oracle, right along really cool new applications like the ELK Stack or Mongo DB or other things, because of the way that we design our layout, it's really aligned towards being able to re-platform and align in a software defined infrastructure. So customers are using to kind of align those pieces meaning platforms, re-platforming and then also aligning specific applications that require high performance. >> I heard a lot in that and one word that pops up is no, that's good. >> No, I can tell you're passionate about it. >> I love it, yeah. >> And also the customer influence is absolutely critical. I think this is a time you mentioned the curbs I check in, and then I was reading a few months ago about some of the huge brands that were filing for chapter 11 and companies like big retailers that simply couldn't pivot, couldn't digitally transform to even offer curbside check in so that factor alone since us consumers are so demanding was table stakes a few months ago. It still is, but getting an organization able to pivot so quickly is key. Martin let's go over to you, PowerMax, workloads. Talk to me about some differentiators as well. >> Yeah Aatually, if I could I'll start with sort of some similar examples that Matt laid out there, you know, just like we have customers who chose PowerFlex you know, were in environments that made sense for them. We had customers who chose PowerMax to meet similar new demands with the whole, you know pandemic. So we had some really big customers just so okay, now we have sort of line of sight and, you know, across both products, I think the thing that our customers value most is you know, the quality of the experience, the performance of the experience, some of the things Matt mentioned already. But they really pull forward, you know, huge numbers of systems and business, and be able to support you know, where they saw things going. So that was really great to partner with them on that and be ready to help support them and provide a product that they felt really good about making such huge investments in, you know, it was great to see their trust in us and be able to deliver for them. So, that was, I think a big part of the first half of the year, that sort of new, you know, new workloads and new use cases for us on the PowerMax side really revolve around giving our customers new capabilities that can deliver new services for their end users. So one of those is our new support for VVols remote replication. And this really lets us tie together the way that the infrastructure is managed at the VMware level, much more closely to the way that the storage infrastructure is managed. And the result is that our, our customers can do more granular operations for their end users, they can simplify the whole process, and now they can do it on top of our remote replication solution, which, you know going on 20 plus years now, it's really been sort of the gold standard in which they've come to rely on so much. So that's really exciting to be able to offer that to them now, to have it be part of the whole VMware stack that they're deploying and let them use you know, new things like, you know the way VVols works with our cyber site recovery manager, to let them automate you know, the testing, I feel always in the actual fail over. There's an interesting example of how I think our customers are going to take advantage of some of these new technologies as we go forward. >> You mentioned giving customers the ability with the right infrastructure to offer new services. And that's another critical component as we've seen in 2020 is businesses needing to pivot continuously and come up with new creative ideas, products, and services and new ways of delivering those to their existing customers holding onto them and hopefully growing their customer base. And that ability to leverage technology, to deliver new services is also one of the key kind of foundations that will allow businesses to be the winners of tomorrow. Matthew, to you talk to me when you're in customer situations, customers have choice, we know this, ding into me, give me the top three differentiators when you're talking to customers, why PowerFlex is the ideal solution for them? >> That's a great question. I'm glad you asked. (laughs) So I think, you know, as part of being a product guy it's really cool when the intellectual property within your product is software that your company owns and hardware, your company owns. So we're able to do some really cool stuff together to deliver innovative solutions for our customers. But, you know, when I think about my product I think first and foremost, around performance and scale right? You know, several million, IO'S a sub-millisecond response time and anytime someone wants more performance they just add another server, right? So this idea that we scale literally is a key differentiator for the product. A second key differentiator is this idea that I talked a little bit about before that we, you can kind of multi-platform this. So when you roll this out, you can deploy to use it with virtual environments, whether it's VMware or Hyper-V or other virtual environments. You can have bare metal deployment. So if you want to run this with Linux and use software defined storage in the bare metal, we can support that. Or we can go directly to containers. So you can use containers, bare metal or virtual. And so this idea of choice is a huge differentiator. And then the last one is anchored around this idea that when you scale and you get the benefit of management, you don't have to scale everything at the same time. So in traditional software defined infrastructure on the HCI side you have to scale compute and storage together. So every time you add a node you add compute power and storage power. With power flex, we've been able to effectively split those two pieces off, so a customer could actually only scale what they need. And in fact, if they only want to buy storage side of the solution, you can just buy storage side solution and then you can have existing infrastructure connect to that and it behaves just like a traditional three tier model. So those are, I think are the key things that I think differentiate the product and kind of make it special here at Dell and for our customers. >> Matthew, sticking with you, are there any, I think of things like compliance and healthcare and financial services, especially right now, what are some of the key benefits that PowerFlex delivers, say for some of those essential industries right now? >> Yeah, I think, you know it's interesting 'cause those are two of our largest space and financial is probably our largest space. And really for them, it comes down to, you talked about compliance, you talk about scale and then you talk about management. So we said some really interesting requirements because of scale so large, for example, in our last release we're able to start to do rack level firmware and software updates. So when you look at other solutions they might be doing system at a time, doing updates taking them offline and then running those around. But in our scenario, since we kind of own the SDS layer and the compute side, we can actually do update these for an entire rack in one shot. Dramatically reducing the complexity, dramatically reducing the amount of time it takes to do updates. So that's a real big deal in financial space. And then in terms of healthcare, for example we're the only software defined solution product that can run all of Epic healthcare, all pieces of Epic within our product. All other products run out of bandwidth, run out of performance. So they end up not being able to run all sides of the requirement, whether it's the database back end, or the VDI front end, we're the only one on the market that can do all of that. >> It seems to really be a big differentiator in healthcare as a lot of organizations run on Epic or try to, to help with patient care and care delivery. Martin, last question for you. Give me a snapshot of the partner's perspective over the last couple of years with the rebrand under Dell Technologies, with the power portfolio, how have your partners embraced the simplification? >> So, you know, I think that the overall, this gave them clearer understanding of where and what to sell and what made sense for power max in particular, you know, I think it let them anchor on, you know the flagship product of the legendary performance and reliability of that platform and, you know, gave them an easy way to think about where to position that with, you know, our end customers and, you know, in what ways that the products would benefit their customers the most. So, you know, as Matt described on the PowerFlex side, it starts with our performance and reliability and then ultimately, you know enabling them to do whatever they need to do, so across all the different data services and we got to talk ready about some of the new ones you know, but we also have a lot that we've you know, refined over the years and, you know making it sort of official and sort of the PowerMax envelope what everyone really just sort of simplify how they would consume it all. So, you know, I think, you know maybe one of the thing, you know, worth mentioning in all these new use cases and environments and, you know, all the different applications that our customers are trying to operate and deliver on is, you know, security, you know, so we developed a new capability that we call end-to-end efficient encryption. And this really lets customers do encryption all the way from the host through to the storage. And, you know I think ultimately that's going to help them sleep better at night and also, you know help them avoid some of the things that you've seen crop up now. Now that the world is so digital and all the different threats that our customers face. So we're keeping our finger on the pulse of a lot of different needs you know, whether it's flexibility, performance reliability, but all these new new technologies as well to make sure that we set our customers up to be successful as possible. >> That's exactly what they want to be, successful. Martin, Matthew, thank you so much for joining me on the Cube, sharing the updates for PowerMax, PowerFlex, the differentiators. We appreciate your time. >> Thank you, Lisa. >> Yeah, thank you Lisa this was fun. Alright from my guests, I'm Lisa Martin. You've been watching the cubes coverage, Dell Technologies World at 2020, the digital experience. (gentle music)

>>from around the globe. It's >>the Cube with digital coverage of Dev >>Ops Virtual Forum Brought to You by Broadcom. Welcome to Broadcom, Step Ups, Virtual Forum I and Lisa Martin and I'm joined by another Martin very socially. Distance from me all the way. Coming from Birmingham, England, is Glynn Martin, head of Q. A transformation at BT Glenn. It's great to have you on the program. >>Thank you, Lisa. I'm looking forward, Toa. >>As we said before, we went live to Martin's for the price of one in one segment. So this is gonna be an interesting segment, Guesses. What we're gonna do is Glen's gonna give us a really kind of deep inside out view of Dev ops. From an evolution perspective, Soglo's Let's start transformation is at the heart of what you dio. It's obviously been a very transformative year. How have the events of this year affected the transformation that you are so responsible for driving? >>Yeah. Thank you, Leigh. So I mean, yeah, it has been a difficult year Bond, although working for BT, which is ah, global telecommunications company. Relatively resilient, I suppose, as an industry through covert, it obviously still has been affected and has got its challenges on bond. If anything is actually caused us to accelerate of our transformation journey, you know, we had to do some great things during this time around. You know, in the UK for our emergency and health workers give them unlimited data and for vulnerable people to support them and that spent that we've had to deliver changes quickly. Um, but what? We want to be able to do it, deliver those kind of changes quickly, but sustainably for everything that we do, not just because there's an emergency eso we were already on the kind of journey to by John, but ever so ever more important now that we are what we're able to do, those that kind of work, do it more quickly on. But it works because the implications of it not working is could be terrible in terms of, you know, we've been supporting testing centers, new hospitals to treat covert patients, so we need to get it right and therefore the coverage of what we do, the quality of what we do and how quickly we do. It really has taken on a new scowling what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously deal with the fact that you know, Cove in 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less cost, but they're having to deliver more value quicker on bond, you know, to higher quality. So, yeah, certainly the finances is on our minds. And that's why we need flexible models, cost models that allow us to kind of do growth. But we get that growth by showing that we're delivering value, especially in, you know, these times when there are financial challenges on companies. >>So one of the things that I want to ask you about again looking at, develops from the inside out on the evolution that you've seen you talked about the speed of things really accelerating in this last nine months or so. When we think Dev ops, we think speed. But one of the things I love to get your perspective on we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that scene there as as needing to get, as you said, get things right but done so quickly to support essential businesses, essential workers? How have you seen that cultural shift? >>Yeah, I think you know, before, you know, test test team saw themselves of this part of the software delivery cycle. Andi, actually, now, really, our customers were expecting their quality and to deliver for our customers what they want. Quality has to be ingrained throughout the life cycle. Obviously that you know, there's lots of buzzwords like shift left. How do you do? Shift left testing. But for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle. That Dr you know, Dr Automation drive improvements. I always say that you know, you're only as good as your lowest common denominator on one thing that we're finding on our Dev Ops Journey Waas that we were you know, we would be trying thio do certain things quicker and had automated build automated tests. But if we were taking weeks to create test scripts or we were taking weeks to manly craft data, and even then when we had taken so long to do it that the coverage was quite poor and that led to lots of defects later in the lifecycle or even in in our production environment, we just couldn't afford to do that. And actually, you know, focusing on continuous testing over the last 9 to 12 months has really given us the ability Thio delivered quickly across the the whole life cycle and therefore actually go from doing a kind of semi agile kind of thing where we did you use the stories we did a few of the kind of, you know, as our ceremonies. But we weren't really deploying any quicker into production because, you know, our stakeholders were scared that we didn't have the same control that we had when we had more water for releases. And, you know, when way didn't think ourselves. So we've done a lot of work on every aspect, especially from a testing point of view, every aspect of every activity, rather than just looking at automated test, you know, whether it is actually creating the test in the first place, Whether it's doing security testing earlier in the light and performance testing. Learn the life cycle, etcetera. So, yeah, it Z It's been a riel key thing that for for C T for us to drive, develops, >>talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this, you know, there's a thing. I think people were pretty quiet. Customer experience. Gap. It reminds me of a cart, a Gilbert cartoon where, you know, we start with the requirements here on Do you know, we almost like a Chinese whisper effects and what we deliver eyes completely, completely different. So we think the testing team or the the delivery team, you know, you know, you think they've done a great job. This is what it said in the acceptance criteria, but then our customers the same Well, actually, that's not working. This isn't working, you know, on there's this kind of gap Way had a great launched this year of actual Requirement Society, one of the board common tools Onda that for the first time in in since I remember actually working within B. T, I had customers saying to may, Wow, you know, we want more of this. We want more projects, um, to have a actual requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do you actually, you know, do that have something that both the business on technical people can understand? And we've actually been working with the business using at our requirement. Designer Thio, you know, really look about what the requirements are. Tease out requirements to the hadn't even thought off and making sure that we've got high levels of test coverage. And so what we actually deliver at the end of it, not only have you been able Thio generate test more quickly, but we've got much higher test coverage and also can more smartly, you're using the kind of AI within the tour and with some of the other kind of pipeline tools actually deliver to choose the right tests on the bar, still actually doing a risk based testing approach. So that's been a great launched this year, but just the start of many kind of things that we're >>doing. But what I hear in that Glenn is a lot of positives that have come out of a very challenging situation. Uh, talk to me about it and I like that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration, perspective is you're right. We talk about that a lot critical with Dev Ops. But those challenges there you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pit it so fast? >>I mean, you talked about culture. I mean, you know, Bt is like most come countries companies. So, um, is very siloed. You know, we're still trying to work to become closer as a company. So I think there's a lot of challenges around. How do you integrate with other tools? How do you integrate with you know, the various different technologies and bt we have 58 different whitey stacks? That's not systems that stacks all of those stacks of can have, you know, hundreds of systems on we're trying to. We're gonna drive at the moment a simplified program where we're trying Thio, you know, reduce that number 2 14 stacks. And even then they'll be complexity behind the scenes that that we will be challenged. Maurin Mawr As we go forward, how do you actually hired that to our users on as an I T organization? How do we make ourselves Lena so that even when we you know, we've still got some of that legacy and we'll never fully get rid of it on that's the kind of trade off that we have to make. How do we actually deal with that and and hide that for my users a say and and and drive those programs so we can actually accelerate change. So we take, you know, reduce that kind of waste, and that kind of legacy costs out of our business. You know, the other thing is, well, beating. And I'm sure you know telecoms probably no difference to insurance or finance we've got You know, when you take the number of products that we do and then you combine them, the permutations are tens and hundreds of thousands of products. So we as a business to trying to simplify. We are trying Thio do that in a natural way and haven't trying to do agile in the proper way, you know, and really actually work it paste really deliver value. So I think what we're looking Maura, Maura, at the moment is actually, um is more value focus? Before we used to deliver changes, sometimes into production, someone had a great idea or it was a great idea nine months ago or 12 months ago. But actually, then we end up deploying it. And then we look at the the the users, you know, the usage of that product of that application or whatever it is on. It's not being used for six months, so we're getting much we haven't got, you know, because of the last 12 months, we certainly haven't got room for that kind of waste and you know, the for not really understanding the value of changes that we we are doing. So I think that's the most important thing at the moment is really taken that waste out. You know, there's lots of focus on things like flow management. What bits of the our process are actually taking too long, and we've We've started on that journey, but we've got a hell of a long way to go, you know, But that that involves looking every aspect off the kind of software delivery cycle. >>What are some? Because that that going from, what, 58 i t stocks down to 14 or whatever it's going to be go simplifying is sounds magical. Took everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we've started on a continuous testing journey, and I think that's just the start. I mean, that's really, as I say, looking at every aspect off, you know, from a Q, a point of view. It's every aspect of what we dio. But it's also looking at, you know, we're starting to branch into more like a AI ops and, you know, really, the full life cycle on. But, you know, that's just a stepping stone onto, you know, I think oughta Nomics is the way forward, right? You know all of this kind of stuff that happens um, you know, monitoring, you know, monitoring systems, what's happening in production had to be feed that back. How do you get to a point where actually we think about a change on then suddenly it's in production safely. Or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey. But if we want Thio, you know, in a world where the pace is ever increasing the demands of the team and you know, with the pressures on at the moment where with we're being asked to do things, you know more efficiently Ondas leaving as possible. We need to be, you know, thinking about every part of the process. And how do we put the kind of stepping stones in players to lead us to a more automated kind of, you know, their future? >>Do you feel that that plant outcomes are starting to align with what's delivered? Given this massive shift that you're experiencing, >>I think it's starting to, and I think you know, Azzawi. Look at more of a value based approach on. Do you know a Zeiss? A princess was a kind of flight management. I think that's that will become ever evermore important. So I think it's starting to people. Certainly realized that, you know, people teams need to work together. You know, the kind of the cousin between business and ICT, especially as we go Teoh Mawr kind of sad space solutions, low cold solutions. You know there's not such a gap anymore. Actually, some of our business partners expects to be much more tech savvy. Eso I think you know, this is what we have to kind of appreciate. What is I ts role? How do we give the capabilities become more for centers of excellence rather than actually doing Mount amount of work And for May and from a testing point of view, you know, amount, amount of testing, actually, how do we automate that? How do we actually generate that instead of created? I think that's the kind of challenge going forward. >>What are some? As we look forward, what are some of the things that you would like to see implemented or deployed in the next say, 6 to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think you know, certainly for for where we are as a company from a Q A perspective. We are. Yeah, there's certain bits that we do Well, you know, we've started creating continuous delivery. A day evokes pipelines. Um, there's still manual aspects of that. So, you know, certainly for May I I've challenged my team with saying, How do we do an automated journey? So if I, you know, I put a requirement injera or value whoever it is, that's why. Then click a button on bond, you know, with either zero touch of one touch, then put that into production and have confidence that that has been done safely on that it works. And what happens if it doesn't work? So you know, that's that's the next in the next few months, that's what our concentration is about. But it's also about decision making, you know, how do we actually understand those value judgements? And I think there's lots of the things Dev ops, ai ops, kind of always that aspects of business operations. I think it's about having the information in one place to make those kind of decisions. How does it all tied together, as I say, even still with kind of Dev ops, we've still got elements within my company where we've got lots of different organizations doing some doing similar kind of things but the walking of working in silos Still. So I think, having a eye ops Aziz becomes more and more to the fore as we go to the cloud. And that's what we need to. You know, we're still very early on in our cloud journey, you know. So we need to make sure the technologies work with Cloud as well as you kind of legacy systems. But it's about bringing that all together and having a full visible pipeline. Everybody can see and make decisions against >>you said the word confidence, which jumped out at me right away. Because absolutely, you've gotta have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops to be able to gain the confidence that they're making the right decisions for their business? >>Yeah, I mean, I think the the approach that we've taken actually is not started with technology we've actually taken human centered design a za core principle of what we dio within the i t part of BT. So by using humans tend to design. That means we talked to our customers. We understand their pain points, we map out their current processes on. But when we mapped out, those processes also understand their aspirations as well, you know, Where do they want to be in six months? You know, Do they want to be more agile and you know, or do they want Teoh? Is this apart their business that they want thio run better? We have to Then look at why that's not running well and then see what solutions are out there. We've been lucky that, you know, with our partnership with Broadcom within the P l. A. A lot of the tortures and the P l. A have directly answered some of the businesses problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which is you know, in some companies, including as they do there is that kind of, you know, almost by understanding their their pain points and then saying This is how we can solve your problem We've tended to be much more successful than trying Thio impose something and say We're here to technology that they don't quite understand doesn't really understand how it could have resonate with their problems. So I think that's the heart of it is really about, you know, getting looking at the data, looking at the processes, looking at where the kind of waste is on. Then actually then looking at the right solutions. And as I say, continuous testing is a massive for us. We've also got a good relationship with capitals looking at visual ai on. Actually, there's a common theme through that, and I mean, AI is becoming more and more prevalent, and I know yeah, sometimes what is A I and people have kind of the semantics of it. Is it true, ai or not? But yes, certainly, you know, AI and machine learning is becoming more and more prevalent in the way that we work, and it's allowing us to be much more effective, the quicker and what we do on being more accurate. You know, whether it's finding defects, running the right tests or, you know, being able to anticipate problems before they're happening in a production environment. >>Welcome. Thank you so much for giving us this sort of insight. Outlook at Dev Ops, sharing the successes that you're having taking those challenges, converting them toe opportunities and forgiving folks who might be in your shoes or maybe slightly behind advice. I'm sure they appreciate it. We appreciate your time. >>It's been an absolute pleasure, Really. Thank you for inviting me of Extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glynn Martin and Lisa Martin. You're watching the Cube?

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi, Lisa Martin here covering the Broadcom dev ops virtual forum. I'm very pleased to be joined today by a cube alumni, Jeffrey Hammond, the vice president and principal analyst serving CIO is at Forester. Jeffrey. Nice to talk with you today. >>Good morning. It's good to be here. Yeah. >>So a virtual forum, great opportunity to engage with our audiences so much has changed in the last it's an understatement, right? Or it's an overstated thing, but it's an obvious, so much has changed when we think of dev ops. One of the things that we think of is speed, you know, enabling organizations to be able to better serve customers or adapt to changing markets like we're in now, speaking of the need to adapt, talk to us about what you're seeing with respect to dev ops and agile in the age of COVID, what are things looking like? >>Yeah, I think that, um, for most organizations, we're in a, uh, a period of adjustment, uh, when we initially started, it was essentially a sprint, you know, you run as hard as you can for as fast as you can for as long as you can and you just kind of power through it. And, and that's actually what, um, the folks that get hub saw in may when they ran an analysis of how developers, uh, commit times and a level of work that they were committing and how they were working, uh, in the first couple of months of COVID was, was progressing. They found that developers, at least in the Pacific time zone were actually increasing their work volume, maybe because they didn't have two hour commutes or maybe because they were stuck away in their homes, but for whatever reason, they were doing more work. >>And it's almost like, you know, if you've ever run a marathon the first mile or two in the marathon, you feel great and you just want to run and you want to power through it and you want to go hard. And if you do that by the time you get to mile 18 or 19, you're going to be gassed. It's sucking for wind. Uh, and, and that's, I think where we're starting to hit. So as we start to, um, gear our development chops out for the reality that most of us won't be returning into an office until 2021 at the earliest and many organizations will, will be fundamentally changing, uh, their remote workforce, uh, policies. We have to make sure that the agile processes that we use and the dev ops processes and tools that we use to support these teams are essentially aligned to help developers run that marathon instead of just kind of power through. >>So, um, let me give you a couple of specifics for many organizations, they have been in an environment where they will, um, tolerate Rover remote work and what I would call remote work around the edges like developers can be remote, but product managers and, um, you know, essentially scrum masters and all the administrators that are running the, uh, uh, the SCM repositories and, and the dev ops pipelines are all in the office. And it's essentially centralized work. That's not, we are anymore. We're moving from remote workers at the edge to remote workers at the center of what we do. And so one of the implications of that is that, um, we have to think about all the activities that you need to do from a dev ops perspective or from an agile perspective, they have to be remote people. One of the things I found with some of the organizations I talked to early on was there were things that administrators had to do that required them to go into the office to reboot the SCM server as an example, or to make sure that the final approvals for production, uh, were made. >>And so the code could be moved into the production environment. And so it actually was a little bit difficult because they had to get specific approval from the HR organizations to actually be allowed to go into the office in some States. And so one of the, the results of that is that while we've traditionally said, you know, tools are important, but they're not as important as culture as structure as organization as process. I think we have to rethink that a little bit because to the extent that tools enable us to be more digitally organized and to hiring, you know, achieve higher levels of digitization in our processes and be able to support the idea of remote workers in the center. They're now on an equal footing with so many of the other levers, uh, that, that, um, uh, that organizations have at their disposal. Um, I'll give you another example for years. >>We've said that the key to success with agile at the team level is cross-functional co located teams that are working together physically co located. It's the easiest way to show agile success. We can't do that anymore. We can't be physically located at least for the foreseeable future. So, you know, how do you take the low hanging fruits of an agile transformation and apply it in, in, in, in the time of COVID? Well, I think what you have to do is that you have to look at what physical co-location has enabled in the past and understand that it's not so much the fact that we're together looking at each other across the table. It's the fact that we're able to get into a shared mindspace, uh, from, um, uh, from a measurement perspective, we can have shared purpose. We can engage in high bandwidth communications. It's the spiritual aspect of that physical co-location that is actually important. So one of the biggest things that organizations need to start to ask themselves is how do we achieve spiritual colocation with our agile teams? Because we don't have the, the ease of physical co-location available to us anymore? >>Well, the spiritual co-location is such an interesting kind of provocative phrase there, but something that probably was a challenge here, we are seven, eight months in for many organizations, as you say, going from, you know, physical workspaces, co-location being able to collaborate face to face to a, a light switch flip overnight. And this undefined period of time where all we were living with with was uncertainty, how does spiritual, what do you, when you talk about spiritual co-location in terms of collaboration and processes and technology help us unpack that, and how are you seeing organizations adopted? >>Yeah, it's, it's, um, it's a great question. And, and I think it goes to the very root of how organizations are trying to transform themselves to be more agile and to embrace dev ops. Um, if you go all the way back to the, to the original, uh, agile manifesto, you know, there were four principles that were espoused individuals and interactions over processes and tools. That's still important. Individuals and interactions are at the core of software development, processes and tools that support those individual and interact. Uh, those individuals in those interactions are more important than ever working software over comprehensive documentation. Working software is still more important, but when you are trying to onboard employees and they can't come into the office and they can't do the two day training session and kind of understand how things work and they can't just holler over the cube, uh, to ask a question, you may need to invest a little bit more in documentation to help that onboarding process be successful in a remote context, uh, customer collaboration over contract negotiation. >>Absolutely still important, but employee collaboration is equally as important if you want to be spiritually, spiritually co-located. And if you want to have a shared purpose and then, um, responding to change over following a plan. I think one of the things that's happened in a lot of organizations is we have focused so much of our dev ops effort around velocity getting faster. We need to run as fast as we can like that sprinter. Okay. You know, trying to just power through it as quickly as possible. But as we shift to, to the, to the marathon way of thinking, um, velocity is still important, but agility becomes even more important. So when you have to create an application in three weeks to do track and trace for your employees, agility is more important. Um, and then just flat out velocity. Um, and so changing some of the ways that we think about dev ops practices, um, is, is important to make sure that that agility is there for one thing, you have to defer decisions as far down the chain to the team level as possible. >>So those teams have to be empowered to make decisions because you can't have a program level meeting of six or seven teams and one large hall and say, here's the lay of the land. Here's what we're going to do here are our processes. And here are our guardrails. Those teams have to make decisions much more quickly that developers are actually developing code in smaller chunks of flow. They have to be able to take two hours here or 50 minutes there and do something useful. And so the tools that support us have to become tolerant of the reality of, of, of, of how we're working. So if they work in a way that it allows the team together to take as much autonomy as they can handle, um, to, uh, allow them to communicate in a way that, that, that delivers shared purpose and allows them to adapt and master new technologies, then they're in the zone in their spiritual, they'll get spiritually connected. I hope that makes sense. >>It does. I think we all could use some of that, but, you know, you talked about in the beginning and I've, I've talked to numerous companies during the pandemic on the cube about the productivity, or rather the number of hours of work has gone way up for many roles, you know, and, and, and times that they normally late at night on the weekends. So, but it's a cultural, it's a mind shift to your point about dev ops focused on velocity, sprints, sprints, sprints, and now we have to, so that cultural shift is not an easy one for developers. And even at this folks to flip so quickly, what have you seen in terms of the velocity at which businesses are able to get more of that balance between the velocity, the sprint and the agility? >>I think, I think at the core, this really comes down to management sensitivity. Um, when everybody was in the office, you could kind of see the mental health of development teams by, by watching how they work. You know, you call it management by walking around, right. We can't do that. Managers have to, um, to, to be more aware of what their teams are doing, because they're not going to see that, that developer doing a check-in at 9:00 PM on a Friday, uh, because that's what they had to do, uh, to meet the objectives. And, um, and, and they're going to have to, to, um, to find new ways to measure engagement and also potential burnout. Um, friend of mine once had, uh, had a great metric that he called the parking lot metric. It was helpful as the parking lot at nine. And how full was it at five? >>And that gives you an indication of how engaged your developers are. Um, what's the digital equivalent equivalent to the parking lot metric in the time of COVID it's commit stats, it's commit rates. It's, um, you know, the, uh, the turn rate, uh, that we have in our code. So we have this information, we may not be collecting it, but then the next question becomes, how do we use that information? Do we use that information to say, well, this team isn't delivering as at the same level of productivity as another team, do we weaponize that data or do we use that data to identify impedances in the process? Um, why isn't a team working effectively? Is it because they have higher levels of family obligations and they've got kids that, that are at home? Um, is it because they're working with, um, you know, hardware technology, and guess what, they, it's not easy to get the hardware technology into their home office because it's in the lab at the, uh, at the corporate office, uh, or they're trying to communicate, uh, you know, halfway around the world. >>And, uh, they're communicating with a, with an office lab that is also shut down and, and, and the bandwidth just doesn't enable the, the level of high bandwidth communications. So from a dev ops perspective, managers have to get much more sensitive to the, the exhaust that the dev ops tools are throwing off, but also how they're going to use that in a constructive way to, to prevent burnout. And then they also need to, if they're not already managing or monitoring or measuring the level of developer engagement, they have, they really need to start whether that's surveys around developer satisfaction, um, whether it's, you know, more regular social events, uh, where developers can kind of just get together and drink a beer and talk about what's going on in the project, uh, and monitoring who checks in and who doesn't, uh, they have to, to, um, work harder, I think, than they ever have before. >>Well, and you mentioned burnout, and that's something that I think we've all faced in this time at varying levels and it changes. And it's a real, there's a tension in the air, regardless of where you are. There's a challenge, as you mentioned, people having, you know, coworker, their kids as coworkers and fighting for bandwidth, because everyone is forced in this situation. I'd love to get your perspective on some businesses that are, that have done this well, this adaptation, what can you share in terms of some real-world examples that might inspire the audience? >>Yeah. Uh, I'll start with, uh, stack overflow. Uh, they recently published a piece in the journal of the ACM around some of the things that they had discovered. Um, you know, first of all, just a cultural philosophy. If one person is remote, everybody is remote. And you just think that way from an executive level, um, social spaces. One of the things that they talk about doing is leaving a video conference room open at a team level all day long, and the team members, you know, we'll go on mute, you know, so that they don't have to, that they don't necessarily have to be there with somebody else listening to them. But if they have a question, they can just pop off mute really quickly and ask the question. And if anybody else knows the answer, it's kind of like being in that virtual pod. Uh, if you, uh, if you will, um, even here at Forrester, one of the things that we've done is we've invested in social ceremonies. >>We've actually moved our to our team meetings on, on my analyst team from, from once every two weeks to weekly. And we have built more time in for social Ajay socialization, just so we can see, uh, how, how, how we're doing. Um, I think Microsoft has really made some good, uh, information available in how they've managed things like the onboarding process. I think I'm Amanda silver over there mentioned that a couple of weeks ago when, uh, uh, a presentation they did that, uh, uh, Microsoft onboarded over 150,000 people since the start of COVID, if you don't have good remote onboarding processes, that's going to be a disaster. Now they're not all developers, but if you think about it, um, everything from how you do the interviewing process, uh, to how you get people, their badges, to how they get their equipment. Um, security is a, is another issue that they called out typically, uh, it security, um, the security of, of developers machines ends at, at, at the corporate desktop. >>But, you know, since we're increasingly using our own machines, our own hardware, um, security organizations kind of have to extend their security policies to cover, uh, employee devices, and that's caused them to scramble a little bit. Uh, so, so the examples are out there. It's not a lot of, like, we have to do everything completely differently, but it's a lot of subtle changes that, that have to be made. Um, I'll give you another example. Um, one of the things that, that we are seeing is that, um, more and more organizations to deal with the challenges around agility, with respect to delivering software, embracing low-code tools. In fact, uh, we see about 50% of firms are using low-code tools right now. We predict it's going to be 75% by the end of next year. So figuring out how your dev ops processes support an organization that might be using Mendix or OutSystems, or, you know, the power platform building the front end of an application, like a track and trace application really, really quickly, but then hooking it up to your backend infrastructure. Does that happen completely outside the dev ops investments that you're making and the agile processes that you're making, or do you adapt your organization? Um, our hybrid teams now teams that not just have professional developers, but also have business users that are doing some development with a low-code tool. Those are the kinds of things that we have to be, um, willing to, um, to entertain in order to shift the focus a little bit more toward the agility side, I think >>Lot of obstacles, but also a lot of opportunities for businesses to really learn, pay attention here, pivot and grow, and hopefully some good opportunities for the developers and the business folks to just get better at what they're doing and learning to embrace spiritual co-location Jeffrey, thank you so much for joining us on the program today. Very insightful conversation. >>My pleasure. It's it's, it's an important thing. Just remember if you're going to run that marathon, break it into 26, 10 minute runs, take a walk break in between each and you'll find that you'll get there. >>Digestible components, wise advice. Jeffery Hammond. Thank you so much for joining for Jeffrey I'm Lisa Martin, you're watching Broadcom's dev ops virtual forum >>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom, >>Continuing our conversations here at Broadcom's dev ops virtual forum. Lisa Martin here, please. To welcome back to the program, Serge Lucio, the general manager of the enterprise software division at Broadcom. Hey, Serge. Welcome. Thank you. Good to be here. So I know you were just, uh, participating with the biz ops manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept, but I wanted to get your thoughts on spiritual co-location as really a necessity for biz ops to succeed in this unusual time in which we're living. What are your thoughts on spiritual colocation in terms of cultural change versus adoption of technologies? >>Yeah, it's a, it's, it's quite interesting, right? When we, when we think about the major impediments for, uh, for dev ops implementation, it's all about culture, right? And swore over the last 20 years, we've been talking about silos. We'd be talking about the paradox for these teams to when it went to align in many ways, it's not so much about these teams aligning, but about being in the same car in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me, the, this, this is really about kind of changing this culture where people start to look at a kind of OKR is instead of the key objective, um, that, that drives the entire team. Now, what it means in practice is really that's, uh, we need to change a lot of behaviors, right? It's not about the Yarki, it's not about roles. It's about, you know, who can do what and when, and, uh, you know, driving a bias towards action. It also means that we need, I mean, especially in this school times, it becomes very difficult, right? To drive kind of a kind of collaboration between these teams. And so I think there there's a significant role that especially tools can play in terms of providing this complex feedback from teams to, uh, to be in that preface spiritual qualification. >>Well, and it talked about culture being, it's something that, you know, we're so used to talking about dev ops with respect to velocity, all about speed here. But of course this time everything changed so quickly, but going from the physical spaces to everybody being remote really does take it. It's very different than you can't replicate it digitally, but there are collaboration tools that can kind of really be essential to help that cultural shift. Right? >>Yeah. So 2020, we, we touch to talk about collaboration in a very mundane way. Like, of course we can use zoom. We can all get into, into the same room. But the point when I think when Jeff says spiritual, co-location, it's really about, we all share the same objective. Do we, do we have a niece who, for instance, our pipeline, right? When you talk about dev ops, probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team, but just thinking about a pipeline, right, at the end of the day, it's all about what is the meantime to beat back to these teams. If I'm a developer and a commit code, I don't, does it take where, you know, that code to be processed through pipeline pushy? Can I get feedback if I am a finance person who is funding a product or a project, what is my meantime to beat back? >>And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of the Dora metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration, it's really a balance. How do you provide the feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this, this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when I checked in code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allegation, how can the system help me reduce my change pillar rates? Because it's, it's able to predict that some issue was introduced in the outpatient or work product. Um, so I think there's, there's a great role of technology and AI candidate Lynch to, to actually provide that new level of collaboration. >>So we'll get to AI in a second, but I'm curious, what are some of the, of the metrics you think that really matter right now is organizations are still in some form of transformation to this new almost 100% remote workforce. >>So I'll just say first, I'm not a big fan of metrics. Um, and the reason being that, you know, you can look at a change killer rate, right, or a lead time or cycle time. And those are, those are interesting metrics, right? The trend on metric is absolutely critical, but what's more important is you get to the root cause what is taught to you lean to that metric to degrade or improve or time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll give you a very mundane example. You know, we know that cycle time is heavily influenced by, um, organizational boundaries. So, you know, we talk a lot about silos, but, uh, we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are a pretty, you know, valid set metrics, but what's way more important is to understand what are the antiperspirants, what are the things that we can detect through the data that actually are affecting those metrics. And, uh, I mean, over the last 10, 20 years, we've learned a lot about kind of what are, what are the antiperspirants within our large enterprise customers. And there are plenty of them. >>What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >>So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle, where DevOps is more and more embracing value stream management value stream mapping. Um, and I think what's interesting is that in many ways the product is becoming the new silo. Uh, the notion of a product is very difficult by itself to actually define people are starting to recognize that a value stream is not its own little kind of Island. That in reality, when I define a product, this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So, so even on that, and there is clearly kind of a new sets, if you will, of anti-patterns where products are being defined as a set of OTRs, they have interdependencies and you have have a new set of silos on the operands, uh, the Abra key movement to Israel and the SRE space where, um, I think there is a cultural clash while the dev ops side is very much embracing this notion of OTRs and value stream mapping and Belgium management. >>On the other end, you have the it operations teams. We still think business services, right? For them, they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so the, the, there is there's this boundary where, um, I think, well, SRE is being put in place. And there's lots of thinking about what kind of metrics can be fined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for true operations team. >>And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You talked about, uh, AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate, maybe see eye to eye with the SRS? >>Yeah. Um, so th you're kind of too. So even for myself, as a leader of a, you know, 1500 people organization, there's a number of things I don't see right. On a daily basis. And, um, I think the, the, the, the technologies that we have at our disposal today from the AI are able to mind a lot of data and expose a lot of, uh, issues that's as leaders we may not be aware of. And some of the, some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you, when you start to understand, for instance, uh, what is the, what is the working progress right to during the sprint? Um, when you start to analyze the data you can detect, for instance, that maybe the teams are over committed, that there is too much work in progress. >>You can start to identify kind of, interdepencies either from a technology, from a people point of view, which were hidden, uh, you can start to understand maybe the change filler rates he's he is dragging. So I believe that there is a, there's a fundamental role to be played by the tools to, to expose again, these anti parents, to, to make these things visible to the teams, to be able to even compare teams. Right. One of the things that's, that's, uh, that's amazing is now we have access to tons of data, not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate, and what's working, what's not working >>Thoughts on AI and automation as, as a facilitator of spiritual co-location. >>Yeah, absolutely. Absolutely. It's um, you know, th there's, uh, the problem we all face is the unknown, right? The, the law city, but volume variety of the data, uh, everyday we don't really necessarily completely appreciate what is the impact of our actions, right? And so, um, AI can really act as a safety net that enables us to, to understand what is the impact of our actions. Um, and so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data, um, and collaborate on the data. And the actual matter, I think is, is a, is a very powerful enabler, uh, on, in that respect. I mean, I, I've seen, um, I've seen countless of times that, uh, for instance, at the SRE boundary, um, to basically show that we'll turn the quality attributes, so an incoming release, right. And exposing that to, uh, an operations person and a sorry person, and enabling that collaboration dialogue through data is a very, very powerful tool. >>Do you have any recommendations for how teams can use, you know, the SRE folks, the dev ops says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be nonproductive. >>Yeah. So to me, the th there, there's a part of the question really is when, when we talk about data, there are there different ways you can use data, right? Um, so you can, you can do a lot of an analytics, predictive analytics. So I think there is a, there's a tendency, uh, to look at, let's say a, um, a specific KPI, like a, an availability KPI, or change filler rate, and to basically do a regression analysis and projecting all these things, going to happen in the future. To me, that that's, that's a, that's a bad approach. The reason why I fundamentally think it's a better approach is because we are systems. The way we develop software is, is a, is a non-leader kind of system, right? Software development is not linear nature. And so I think there's a D this is probably the worst approach is to actually focus on metrics on the other end. >>Um, if you, if you start to actually understand at a more granular level, what har, uh, which are the things which are contributing to this, right? So if you start to understand, for instance, that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we, we have, I've actually, uh, a customer who, uh, identified that, uh, over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed, this resulted in these plant outages. So if you start to be able to basically establish causality, right, cause an effect between kind of data across the last cycle. I think, I think this is the right way to, uh, to, to use AI. And so pharma to be, I think it's way more God could have a classification problem. What are the classes of problems that do exist and affect things as opposed to analytics, predictive, which I don't think is as powerful. >>So I mentioned in the beginning of our conversation, that just came off the biz ops manifesto. You're one of the authors of that. I want to get your thoughts on dev ops and biz ops overlapping, complimenting each other, what, from a, the biz ops perspective, what does it mean to the future of dev ops? >>Yeah, so, so it's interesting, right? If you think about DevOps, um, there's no felony document, right? Can we, we can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways, there's no clear definition of what dev ops is. Uh, if you go to the dev ops Institute today, you'll see that they are specific, um, trainings for instance, on value management on SRE. And so in many ways, the problem we have as an industry is that, um, there are set practices between agile dev ops, SRE Valley should management. I told, right. And we all basically talk about the same things, right. We all talk about essentially, um, accelerating in the meantime fee to feedback, but yet we don't have the common framework to talk about that. The other key thing is that we add to wait, uh, for, uh, for jeans, Jean Kim's Lascaux, um, to, uh, to really start to get into the business aspect, right? >>And for value stream mapping to start to emerge for us to start as an industry, right. It, to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about driving these business outcomes. And so to me, these ops is really about kind of, uh, putting a lens on this critical element that it's not business and it, that we in fact need to fuse business 19 that I need needs to transform itself to recognize that it's, it's this value generator, right. It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this Oliver or kind of framework, if you will. That set the context for what is the reason, uh, for it to exist. What's part of the core values and principles that it needs to embrace to, again, change from a cost center to a value center. And then we need to start to use this as a way to start to unify some of the, again, the core practices, whether it's agile, DevOps value, stream mapping SRE. Um, so, so I think over time, my hope is that we start to optimize a lot of our practices, language, um, and, uh, and cultural elements. >>Last question surgeon, the last few seconds we have here talking about this, the relation between biz ops and dev ops, um, what do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >>So to me, the key, the key, um, challenge for, for the industry is really around. So we were seeing a very rapid shift towards kind of, uh, product to product, right. Which we don't want to do is to recreate kind of these new silos, these hard silos. Um, so that, that's one of the big changes, uh, that I think we need to be, uh, to be really careful about, um, because it is ultimately, it is about culture. It's not about, uh, it's not about, um, kind of how we segment the work, right. And, uh, any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of, um, kind of the spiritual co-location, I think it's, it's really about that too. It's really about kind of, uh, uh, focusing on the business outcomes on kind of aligning on driving engagement across the teams, but, but not for create a, kind of a new set of silos, which instead of being vertical are going to be these horizontal products >>Crazy by surge that looking at culture as kind of a way of really, uh, uh, addressing and helping to, uh, re re reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps virtual forum. >>Thank you. Thanks for your time. >>I'll be right back >>From around the globe it's the cube with digital coverage of devops virtual forum brought to you by Broadcom. >>Welcome to Broadcom's DevOps virtual forum, I'm Lisa Martin, and I'm joined by another Martin, very socially distanced from me all the way coming from Birmingham, England is Glynn Martin, the head of QA transformation at BT. Glynn, it's great to have you on the program. Thank you, Lisa. I'm looking forward to it. As we said before, we went live to Martins for the person one in one segment. So this is going to be an interesting segment guys, what we're going to do is Glynn's going to give us a really kind of deep inside out view of devops from an evolution perspective. So Glynn, let's start. Transformation is at the heart of what you do. It's obviously been a very transformative year. How have the events of this year affected the >> transformation that you are still responsible for driving? Yeah. Thank you, Lisa. I mean, yeah, it has been a difficult year. >>Um, and although working for BT, which is a global telecommunications company, um, I'm relatively resilient, I suppose, as a, an industry, um, through COVID obviously still has been affected and has got its challenges. And if anything, it's actually caused us to accelerate our transformation journey. Um, you know, we had to do some great things during this time around, um, you know, in the UK for our emergency and, um, health workers give them unlimited data and for vulnerable people to support them. And that's spent that we've had to deliver changes quickly. Um, but what we want to be able to do is deliver those kinds of changes quickly, but sustainably for everything that we do, not just because there's an emergency. Um, so we were already on the kind of journey to agile, but ever more important now that we are, we are able to do those, that kind of work, do it more quickly. >>Um, and that it works because the, the implications of it not working is, can be terrible in terms of you know, we've been supporting testing centers,  new hospitals to treat COVID patients. So we need to get it right. And then therefore the coverage of what we do, the quality of what we do and how quickly we do it really has taken on a new scale and what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that, you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously, um, deal with the fact that, you know, COVID 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less costs, but having to deliver more value quicker and  to higher quality. So yeah, certainly the finances is, um, on our minds and that's why we need flexible models, cost models that allow us to kind of do growth, but we get that growth by showing that we're delivering value. Um, especially in these times when there are financial challenges on companies. So one of the things that I want to ask you about, I'm again, looking at DevOps from the inside >>Out and the evolution that you've seen, you talked about the speed of things really accelerating in this last nine months or so. When we think dev ops, we think speed. But one of the things I'd love to get your perspective on is we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that you've seen there as, as needing to get, as you said, get things right, but done so quickly to support essential businesses, essential workers. How have you seen that cultural shift? >>Yeah, I think, you know, before test teams for themselves at this part of the software delivery cycle, um, and actually now really our customers are expecting that quality and to deliver for our customers what they want, quality has to be ingrained throughout the life cycle. Obviously, you know, there's lots of buzzwords like shift left. Um, how do we do shift left testing? Um, but for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle that drive automation, drive improvements. I always say that, you know, you're only as good as your lowest common denominator. And one thing that we were finding on our dev ops journey was that we  would be trying to do certain things quick, we had automated build, automated tests. But if we were taking a weeks to create test scripts, or we were taking weeks to manually craft data, and even then when we had taken so long to do it, that the coverage was quite poor and that led to lots of defects later on in the life cycle, or even in our production environment, we just couldn't afford to do that. >>And actually, focusing on continuous testing over the last nine to 12 months has really given us the ability to deliver quickly across the whole life cycle. And therefore actually go from doing a kind of semi agile kind of thing, where we did the user stories, we did a few of the kind of agile ceremonies, but we weren't really deploying any quicker into production because our stakeholders were scared that we didn't have the same control that we had when we had more waterfall releases. And, you know, when we didn't think of ourselves. So we've done a lot of work on every aspect, um, especially from a testing point of view, every aspect of every activity, rather than just looking at automated tests, you know, whether it is actually creating the test in the first place, whether it's doing security testing earlier in the lot and performance testing in the life cycle, et cetera. So, yeah,  it's been a real key thing that for CT, for us to drive DevOps, >>Talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this. Um, you know, there's a thing that I think people will probably call it a customer experience gap, and it reminds me of a Gilbert cartoon, where we start with the requirements here and you're almost like a Chinese whisper effects and what we deliver is completely different. So we think the testing team or the delivery teams, um, know in our teeth has done a great job. This is what it said in the acceptance criteria, but then our customers are saying, well, actually that's not working this isn't working and there's this kind of gap. Um, we had a great launch this year of agile requirements, it's one of the Broadcom tools. And that was the first time in, ever since I remember actually working within BT, I had customers saying to me, wow, you know, we want more of this. >>We want more projects to have extra requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do we actually, you know, do that and have something that both the business and technical people can understand. And we've actually been working with the business , using agile requirements designer to really look at what the requirements are, tease out requirements we hadn't even thought of and making sure that we've got high levels of test coverage. And what we actually deliver at the end of it, not only have we been able to generate tests more quickly, but we've got much higher test coverage and also can more smartly, using the kind of AI within the tool and then some of the other kinds of pipeline tools, actually deliver to choose the right tasks, and actually doing a risk based testing approach. So that's been a great launch this year, but just the start of many kinds of things that we're doing >>Well, what I hear in that, Glynn is a lot of positives that have come out of a very challenging situation. Talk to me about it. And I liked that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration perspective you're right, we talk about that a lot critical with devops. But those challenges there, you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pivot so fast? >>I mean, you talked about culture. You know, BT is like most companies  So it's very siloed. You know we're still trying to work to become closer as a company. So I think there's a lot of challenges around how would you integrate with other tools? How would you integrate with the various different technologies. And BT, we have 58 different IT stacks. That's not systems, that's stacks, all of those stacks can have hundreds of systems. And we're trying to, we've got a drive at the moment, a simplified program where we're trying to you know, reduce that number to 14 stacks. And even then there'll be complexity behind the scenes that we will be challenged more and more as we go forward. How do we actually highlight that to our users? And as an it organization, how do we make ourselves leaner, so that even when we've still got some of that legacy, and we'll never fully get rid of it and that's the kind of trade off that we have to make, how do we actually deal with that and hide that from our users and drive those programs, so we can, as I say, accelerate change,  reduce that kind of waste and that kind of legacy costs out of our business. You know, the other thing as well, I'm sure telecoms is probably no different to insurance or finance. When you take the number of products that we do, and then you combine them, the permutations are tens and hundreds of thousands of products. So we, as a business are trying to simplify, we are trying to do that in an agile way. >>And haven't tried to do agile in the proper way and really actually work at pace, really deliver value. So I think what we're looking more and more at the moment is actually  more value focused. Before we used to deliver changes sometimes into production. Someone had a great idea, or it was a great idea nine months ago or 12 months ago, but actually then we ended up deploying it and then we'd look at the users, the usage of that product or that application or whatever it is, and it's not being used for six months. So we haven't got, you know, the cost of the last 12 months. We certainly haven't gotten room for that kind of waste and, you know, for not really understanding the value of changes that we are doing. So I think that's the most important thing of the moment, it's really taking that waste out. You know, there's lots of focus on things like flow management, what bits of our process are actually taking too long. And we've started on that journey, but we've got a hell of a long way to go. But that involves looking at every aspect of the software delivery cycle. >> Going from, what 58 IT stacks down to 14 or whatever it's going to be, simplifying sounds magical to everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we were started on a continuous testing journey, and I think that's just the start. I mean as I say, looking at every aspect of, you know, from a QA point of view is every aspect of what we do. And it's also looking at, you know, we've started to branch into more like AI, uh, AI ops and, you know, really the full life cycle. Um, and you know, that's just a stepping stone to, you know, I think autonomics is the way forward, right. You know, all of this kind of stuff that happens, um, you know, monitoring, uh, you know, watching the systems what's happening in production, how do we feed that back? How'd you get to a point where actually we think about change and then suddenly it's in production safely, or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey, but if we want to, you know, in a world where the pace is in ever-increasing and the demands for the team, and, you know, with the pressures on, at the moment where we're being asked to do things, uh, you know, more efficiently and as lean as possible, we need to be thinking about every part of the process and how we put the kind of stepping stones in place to lead us to a more automated kind of, um, you know, um, the future. >>Do you feel that that planned outcomes are starting to align with what's delivered, given this massive shift that you're experiencing? >>I think it's starting to, and I think, you know, as I say, as we look at more of a value based approach, um, and, um, you know, as I say, print, this was a kind of flow management. I think that that will become ever, uh, ever more important. So, um, I think it starting to people certainly realize that, you know, teams need to work together, you know, the kind of the cousin between business and it, especially as we go to more kind of SAS based solutions, low code solutions, you know, there's not such a gap anymore, actually, some of our business partners that expense to be much more tech savvy. Um, so I think, you know, this is what we have to kind of appreciate what is its role, how do we give the capabilities, um, become more of a centers of excellence rather than actually doing mounds amounts of work. And for me, and from a testing point of view, you know, mounds and mounds of testing, actually, how do we automate that? How do we actually generate that instead of, um, create it? I think that's the kind of challenge going forward. >>What are some, as we look forward, what are some of the things that you would like to see implemented or deployed in the next, say six to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think, um, you know, certainly for, for where we are as a company from a QA perspective, we are, um, you let's start in bits that we do well, you know, we've started creating, um, continuous delivery and DevOps pipelines. Um, there's still manual aspects of that. So, you know, certainly for me, I I've challenged my team with saying how do we do an automated journey? So if I put a requirement in JIRA or rally or wherever it is and why then click a button and, you know, with either zero touch for one such, then put that into production and have confidence that, that has been done safely and that it works and what happens if it doesn't work. So, you know, that's, that's the next, um, the next few months, that's what our concentration, um, is, is about. But it's also about decision-making, you know, how do you actually understand those value judgments? >>And I think there's lots of the things dev ops, AI ops, kind of that always ask aspects of business operations. I think it's about having the information in one place to make those kinds of decisions. How does it all try and tie it together? As I say, even still with kind of dev ops, we've still got elements within my company where we've got lots of different organizations doing some, doing similar kinds of things, but they're all kind of working in silos. So I think having AI ops as it comes more and more to the fore as we go to cloud, and that's what we need to, you know, we're still very early on in our cloud journey, you know, so we need to make sure the technologies work with cloud as well as you can have, um, legacy systems, but it's about bringing that all together and having a full, visible pipeline, um, that everybody can see and make decisions. >>You said the word confidence, which jumped out at me right away, because absolutely you've got to have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question then for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops, to be able to gain the confidence that they're making the right decisions for their business? >>I think the, the, the, the, the approach that we've taken actually is not started with technology. Um, we've actually taken a human centered design, uh, as a core principle of what we do, um, within the it part of BT. So by using human centered design, that means we talk to our customers, we understand their pain points, we map out their current processes. Um, and then when we mapped out what this process does, it also understand their aspirations as well, you know? Um, and where do they want to be in six months? You know, do they want it to be, um, more agile and, you know, or do they want to, you know, is, is this a part of their business that they want to do one better? We actually then looked at why that's not running well, and then see what, what solutions are out there. >>We've been lucky that, you know, with our partnership, with Broadcom within the payer line, lots of the tools and the PLA have directly answered some of the business's problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which in, in, uh, you know, in some companies include not as they do there is that kind of, you know, almost by understanding their, their pain points and then starting, this is how we can solve your problem. Um, is we've, we've tended to be much more successful than trying to impose something and say, well, here's the technology that they don't quite understand. It doesn't really understand how it kind of resonates with their problems. So I think that's the heart of it. It's really about, you know, getting, looking at the data, looking at the processes, looking at where the kind of waste is. >>And then actually then looking at the right solutions. Then, as I say, continuous testing is massive for us. We've also got a good relationship with Apple towards looking at visual AI. And actually there's a common theme through that. And I mean, AI is becoming more and more prevalent. And I know, you know, sometimes what is AI and people have kind of this semantics of, is it true AI or not, but it's certainly, you know, AI machine learning is becoming more and more prevalent in the way that we work. And it's allowing us to be much more effective, be quicker in what we do and be more accurate. And, you know, whether it's finding defects running the right tests or, um, you know, being able to anticipate problems before they're happening in a production environment. >>Well, thank you so much for giving us this sort of insight outlook at dev ops sharing the successes that you're having, taking those challenges, converting them to opportunities and forgiving folks who might be in your shoes, or maybe slightly behind advice enter. They appreciate it. We appreciate your time. >>Well, it's been an absolute pleasure, really. Thank you for inviting me. I have a extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glenn Martin. I'm Lisa Martin. You're watching the cube >>Driving revenue today means getting better, more valuable software features into the hands of your customers. If you don't do it quickly, your competitors as well, but going faster without quality creates risks that can damage your brand destroy customer loyalty and cost millions to fix dev ops from Broadcom is a complete solution for balancing speed and risk, allowing you to accelerate the flow of value while minimizing the risk and severity of critical issues with Broadcom quality becomes integrated across the entire DevOps pipeline from planning to production, actionable insights, including our unique readiness score, provide a three 60 degree view of software quality giving you visibility into potential issues before they become disasters. Dev ops leaders can manage these risks with tools like Canary deployments tested on a small subset of users, or immediately roll back to limit the impact of defects for subsequent cycles. Dev ops from Broadcom makes innovation improvement easier with integrated planning and continuous testing tools that accelerate the flow of value product requirements are used to automatically generate tests to ensure complete quality coverage and tests are easily updated. >>As requirements change developers can perform unit testing without ever leaving their preferred environment, improving efficiency and productivity for the ultimate in shift left testing the platform also integrates virtual services and test data on demand. Eliminating two common roadblocks to fast and complete continuous testing. When software is ready for the CIC CD pipeline, only DevOps from Broadcom uses AI to prioritize the most critical and relevant tests dramatically improving feedback speed with no decrease in quality. This release is ready to go wherever you are in your DevOps journey. Broadcom helps maximize innovation velocity while managing risk. So you can deploy ideas into production faster and release with more confidence from around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of dev ops and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high-performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago. So it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate physical co-location. Okay. That makes sense. >>It does. I shouldn't have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with jump. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other understanding, Oh, individuals teams, uh, contributed to kind of a business outcome, what is our shared goal or shared vision? What's what is it we're trying to achieve collectively and keeping it kind of aligned to that? Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organization, there's been specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of reverted back to, okay, what is my database all the time? What is my cycle time? >>Right. And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens or these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, favorites experience was to actually weakness at one of a large financial institution. Um, you know, Tuesday Golder's unquote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, test execution results, you know, Coverity coverage, um, official liabilities and all the all ran. It could have a direction level links. And that's when you start to put these things in context and represent that to you in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and, and understand have they joined our company to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And w I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it can, it says simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos. We were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very dev, uh, different environments, depending on what area of BT you're in, whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can, um, drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. >>And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in their kind of testing world at the moment. But, you know, I could talk about things like continuous delivery direct to when a ball comes tools and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum it up, autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter? Jeff, >>I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we look at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test, uh, departments, a group, the fourth level that I think is really important, uh, in, in the current, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric helpful was the parking lot at nine. And how full was it at five o'clock. I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customers. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big impact. And, and, and also how do you measure quality kind of following onto the advice that Jeff provided? >>That's some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker, uh, called me Kirsten a couple of months ago. It taught very much around how important flow management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes long? So I think that's a very important thing for us. It's even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, you know, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how, how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up our teams, training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are, um, delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, Jeff's point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. >>And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about our, these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood is that, you know, when you have a bad cycle time or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, if you look at, for instance, things like lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. >>And so I think that there is, you know, the only the metrics there is, I think a lot of work that we need to do in terms of classifying, descend type patterns, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as, as a key pattern, the industry or anti-spatter. Um, but waterscrumfall right is a key one, right? And you will detect that through kind of a defect arrival rates. That's where that looks like an S-curve. And so I think it's beyond kind of the, the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on this. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff will go back to you. Okay. >>I would say look for the opportunities that this disruption presents. And there are a couple that I see, first of all, uh, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks to be on the lookout for now? I, I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but low code also. Um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the, the life cycle, um, where, when I heard the word scrum fall, it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment we need to get beyond that. We need >>To be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production and go straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately we all need some peanut butter and chocolate. All right. It starts to take us home. That's what's that nugget you think everyone needs to have on their agendas? >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about the lost city, but it's all about sweet. We still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of a spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value into aligning everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.