>>And welcome to this cube conversation here in Palo Alto, California. I'm John furrier, host of the cube. We have a great guest coming in remotely from LA Christian Hernandez developer experienced lead at code fresh code fresh IO. Recently they were on our feature at a startup showcase series, season two episode one cloud data innovations, open source innovations, all good stuff, Christian. Thanks for coming on this cube conversation. >>Thank you. Thank you, John. Thank you for having me on, >>You know, I'm I was really impressed with code fresh. My met with the founders on here on the cube because GI ops AI, everything's something ops devs dev sec ops. You've got AI ops. You've got now GI ops, essentially operationalizing the software future is here and software's eating the world is, was written many years ago, but it's open source is now all. So all things software's open source and that's kind of a done deal. It's only getting better and better. Mainstream companies are contributing. You guys are on this wave of, of this open source tsunami and you got cloud scale. Automation's right there, machine learning, all this stuff is now the next gen of, of, of code, right? So you, your code fresh and your title is developer experience lead. What does that mean right now? What does it mean to be a developer experience lead? Like you make sure people having a good experience. Are you developing you figuring out the product? What does that mean? >>Yeah. That's and it's also part of the, the whole Debre explosion that's happening right now. I believe it's, you know, everyone's always asking, well, what, you know, what is developer advocate? What does that mean developer experience? What does that mean? So, so you, you kind of hit the nail on the head a little bit up there in, in the beginning, is that the, the experience of the developer when using a particular platform, right? Especially the code flash platform. That is my responsibility there at code fresh to enable, to enable end users, to enable partners, to enable, you know, anyone that wants to use the code fresh platform for their C I C D and get ops square flows. So that's, that's really my, my corner of the world is to make sure their experience is great. So that's, it's really what, what I'm here to do >>At food fresh. You know, one of the things I can say of my career, you've been kind of become a historian over time. When I was a developer back in the old days, it was simply you compiled stuff, you did QA on it. You packaged it out. You wanted out the door and you know, that was a workflow right now with the cloud. I was talking with your founders, you got new abstraction layers. Cloud has changed again again, open source. So newer things are coming, right? Like, like, like Kubernetes for instance is a great example that came out of the open source kind of the innovations. But that, and Hadoop, we were mentioning before he came on camera from a storage standpoint, kind of didn't make it because it was just too hard. Right. And it made the developer's job harder. And then it made the developer's requirements to be specialized. >>So you had kind of two problems. You had hard to use a lot of friction and then it required certain expertise when the developers just want to code. Right. So, so you have now the motion of, with GI ops, you guys are in the middle of kinda this idea of frictionless based software delivery with the cloud. So what's different now, can you talk about that specific point because no one wants to be, do hard work and have to redo things. Yeah. Shift left and all that good stuff. What's hard now, what do you guys solve? What's the, what's the friction that you're taking out what's to become frictionless. >>Yeah. Yeah. And you, you, you mentioned a very interesting point about how, you know, things that are coming out almost makes it seem harder nowadays to develop an application. You used to have it to where, you know, kind of a, sort of a waterfall sort of workflow where, you know, you develop your code, you know, you compile it. Right. You know, I guess back in the day, Java was king. I think Java still is, has a, is a large footprint out there where you would just compile it, deploy it. If it works, it works. Alright cool. And you have it and you kind of just move it along in its process. Whereas I think the, the whole idea of, I think Netflix came out with like the, the fail often fail fast release often, you know, the whole Atlassian C I C D thing, agile thing came into play. >>Where now it's, it's a little bit more complex to get your code out there delivered to get your code from one environment to the other environment, especially with the, the Avan of Kubernetes and cloud native architecture, where you can deploy and have this imutable infrastructure where you can just deploy and automate so quickly. So often that there needs to be some sort of new process now into place where to have a new process, like GI ops to where it'll, it it's frictionless, meaning that it's, it, it makes it that process a little easier makes that little, that comp that complex process of deploying onto like a cloud native architecture easier. So that way, as you said before, returning the developers to back to what they care about, mot, the most is just code. I just want to code. >>Yeah. You know, the other thing, cool thing, Christian, I wanna bring up and we'll get into some of the specifics around Argo specifically CD is that the community is responding as a kind of, it takes a village kind of mindset. People are getting into this just saying, Hey, if we can get our act together around some de facto workflows and de facto capabilities, everyone wins. It's a rising tide, floats all boats, kind of concept. CNCF certainly has been a big part of that. Even seen some of the big hyper scales getting behind it. But you guys are part of the founding members of the open get ups working group, Amazon Azure, GitHub, red hat Weaveworks and then a ton of contributors. Okay. So this is kind of cool. This means that there's like people behind this thing. Look, we gotta get here faster. What happened at co con this year? You guys had some news around Argo and you had some news around the hosted solution. Can you take a minute to explain two things, one the open community vibe, and then two, what you guys announced at Coon in Spain. >>Yeah. Yeah. So as far as open get ups, that was, you know, as you said before, code fresh was part of that, that founding committee. Right. Of, of group of people trying to figure out, define what get ups is. Right. We're trying to bring it beyond the, you know, the, the hype word, right beyond just like a marketing term to where we actually define what it actually is, because it is actually something that's out there that people are doing. Right. A lot of people, you know, remember that the, the Chick-fil-A story where it's like, they, they are completely doing, you know, this get ops thing, we're just now wanting, putting definition around it. So that was just amazing to see out at there in, in Cuban. And, but like you said, in QAN, we, you know, we're, we're, we're taking some of that, that acceleration that we see in the community to, and we, we announce our, our hosted get ops offering. >>Right. So hosted get ops is something that our customers have been asking for for a while. Many times when, you know, someone wants to use something like Argo CD, the, in, they install it on their cluster, they get up and running. And, but with, with all that comes like the feed and care of that platform, and, you know, not only just keeping the lights on, but also management security, you know, general maintenance, you know, all the things that, that come along with managing a system. And on top of that comes like the scale aspect of it. Right. And so with scale, so a lot of people go with like a hub and spoke others, go with like a fleet design in, in either case, right. There's, there's a challenge for the feet and care of it. Right. And so with code fresh coast of get ups, we take that management headache away. >>Right? So we, we take the, the, the management of, of Argo CD, the management of, of all of that, and kind of just offer Argo CD as a surface, right. Which offers, you know, allows users to, you know, let us take care of all the, of the get offs, runtime. And so they can concentrate on, you know, their application deployments. Right. And you also get things like Dora metrics, right. Integrated with the platform, you have the ability to integrate multiple CI providers, you know, like get hub actions or whatever, existing Jenkins pipelines. And really that, that code fresh platform becomes like your get ops platform becomes like, you know, your, your central view of the world of, of your, you know, get ups processes. >>Yeah. I mean, that whole single source of truth concept is really kind of needed. I gotta ask you though, with the popularity of the Argo CD on get ups internally, right. That's been clear, right. Kubernetes, the way that's going, it's accelerating fast. People want simple it's scaling, you got automation built in all that good stuff. What was the driver behind the hosted get up solution? Was it customer needs? Was it efficiency all the above? What was specifically and, and why would someone want to have the hosted versus say internal? >>Yeah. So it's, it was really driven by, you know, customer need been something that the customers have been asking for. And it's also been something that, you know, you, you, you have a process of developing an application to, you know, you know, a fleet of clusters in a traditional, you know, I keep saying traditional, get outs practice as if get outs are so old. And, you know, in, you know, when, when, when people first start out, they'll start, you know, installing Argo city on all these clusters and trying to manage that at scale it's, it's, it, it seemed like there was, you know, it it'd be nice if we can just like, be able to consume this as a service. So we don't have to like, worry about, you know, you know, best practices. We don't have to worry about security. We don't just, all of that is taken care of and managed by us at code fresh. So this is like something that, you know, has been asked for and, and something that, you know, we believe will accelerate, you know, developers into actually developing their, their applications. They don't have to worry about managing >>The platform. So just getting this right. Hosted, managed service by you guys on this one, >>Correct? Yes. >>Okay. Got it. All right. So let me, let me get in the Argo real quick, just to kind of just level set for the folks that are, are leaning into this and then kicking the tires. Where are we with Argo? What, why was it so popular? What did it do specifically? Did it just make it easier for developers to manage and monitor Kubernetes, keep 'em updated? What was the specific value behind Argo? Where, where, where did it come from and why is it so popular? >>Yeah, so Argo the Argo project, which is made up of, of a few tools, usually when people say Argo, they meet, they they're talking about Argo CD, but there's also Argo workflows, Argo events, Argo notifications. And, and like I said before, CD with that, and that is something that was developed internally at Intuit. Right? So for those of who don't know, Intuit is the company behind turbo tax. So for those, those of us in the us, we, we know, you know, we know that season all too well, the tax season. And so that was a tool that was developed internally. >>And by the way, Intuit we've done many years. They're very huge cloud adopters. They've been on that train from the day one. They've been, they've been driving a lot of cloud scale too. Sorry >>To interrupt. Yeah. And, and, and yeah, no, and, and, and also, you know, they, they were always open source first, right. So they've always had, you know, they developed something internally. They always had the, the intention of opensourcing it. And so it was really a tool that was born internally, and it was a tool that helped them, you know, get stuff done with Kubernetes. And that's kind of like the tagline they use for, for the Argo project is you need to get stuff done. They wanted their developers to focus less on deploying the application and more right. More than on writing the application itself. And so the, and so the Argo project is a suite of tools essentially that helps deploy onto Kubernetes, you know, using get ups as that, you know, that cornerstone in design, right in the design philosophy, it's so popular because of the ease of use and developer friendliness aspect of it. It's, it's, it's, it's meant to be simple right. In and simple in a, in a good sense of getting up and running, which attracted, you know, developers from, you know, all around the world. You know, other companies like red hat got into it as well. BlackRock also is, is a, is a big contributor, thousands of other independent contributors as well to the Argo project. >>Yeah. Christian, if you bring up a good point and I'm gonna go on a little tangent here, but I wanna get your reaction to something that Dave ante and I, and our cube team has been kind of riffing on lately. You mentioned, you know, Netflix earlier, you mentioned Intuit. There's a kind of a story that's been developing and, and with traction and momentum and trajectory over the past, say 10 years, the companies that went on the cloud, like Netflix into it, snowflake, snowflake, not so much now, but in terms of open source, they're all contributing lift. They're all contributing back to open source, but they're not cloud providers. Right. So you're seeing that kind of first generation, I's a massive contribution to open source. So open source been around for a while, remember the early days, and we'd all participate on projects, but now you have real companies building IP going open source first because they're on a hyperscale cloud, but they're not the cloud themselves. They took advantage of that. So there's kind of this cycle of flywheel of cloud to open source, not from the vendors themselves like Amazon, which services or Azure, but the people who rode their CapEx and built on that scale, feeding into the open source. And then coming back, this is kind of an interesting dynamic. What's your reaction to that? Do you see that? Yeah. Super cloud kind of vibe there. >>Yeah. Yeah. Well, and, and also it, it, I think it's, it's a, it's indicative that, you know, open source is not only, you know, a way to develop, you know, applications, a way to engineer, you know, your project, but also kind of like a strategic advantage in, in, in such a way. Right. You know, you, you see, you see companies like, like, like even like Microsoft has been going into, you know, open source, right. They they've been going to open source first. They made a, a huge pivot to, you know, using open source as, you know, like, like a, like a strategic direction for, for the company. And I think that goes back to, you know, a little bit for my roots, you know, I, I, I always, I always talk about, you know, I always talk about red hat, right. I always talk about, you know, I was, I was, I was in red hat previously and, you know, you know, red hat being, you know, the first billion dollar open source company. >>Right. I, we always joke is like, well, you know, internally, like we know you were a billion dollar company that sold free software. How, you know, how, how does that happen? But it's, it's, it's really, you know, built into the, built into being able to tap into those expert resources. Yeah. You know, people love using software. People love the software they love using, and they wanna improve it. Companies are now just getting out of their way. Yeah. You know, companies now, essentially, it's just like, let's just get out of the way. Let's let people work on, you know, what they wanna work on. They love the software. They wanna improve it. Let's let them, >>It's interesting. A lot of people love the clouds have all this power. If you think about what we are just riffing on and what you just said, the economics and the organic self-governing has always been the open source way where commercial value is enabled. If you play ball, right. Like, oh, red hat, for instance. And now you're seeing the community kind of be that arbiter of the cloud. So, Hey, if everyone can create value on say AWS or Azure, bring it to open source, everyone benefits across all clouds hope eventually. So the choice aspect comes in. So this community angle is huge. And I think it's changing a lot for the better. And I think this is where we're seeing a lot of that growth. And you guys have been the middle level with the Argo project and get ups specifically in that, in that sector. How have you seen that growth? What some dynamics have you seen power dynamics, organic? Is it governed well, whats some of the, the successes, what are some of the challenges? Can you share your thoughts on the community's growth around get ops and Argo project? >>Yeah, yeah. Yeah. So I've been, you know, part of some of these communities, right? Like the, the open, get, get ops community, the Argos community pretty much from the beginning and, and seeing it developed from an idea to, you know, having all these contributors, having, you know, the, the, the buzzword come out of it, you know, the get ups and it be that being the, you know, having it, you know, all over the, you know, social media, all over LinkedIn, all over all, all these, all these different channels, you know, I I've seen things like get ops con, right. So, you know, being part of the, get ops open, get ops community, you know, one of the things we did was we did get ops con it started as a meetup, you know, couple years ago. And now, you know, it was a, you know, we had an actual event at Cuan in Los Angeles. >>You know, we had like, you know, about 50 people there, but then, you know, Cuan in Valencia this past Cuan we had over 200 people, it was a second largest co-located events in, at Cuan. So that just, just seeing that community and, you know, from a personal standpoint, you know, be being part of that, that the, the community being the, the event chair, right. Yeah. Being, being one of the co-chairs was a, was a moment of pride for me being able to stand up there and just seeing a sea of people was like, wow, we just started with a handful of people at a meetup. And now, you know, we're actually having conferences and, and, and speaking of conference, like the Argo community as well, we put in, you know, we put on a virtual only event on Argo con last year. We're gonna do it in person today. You know, this year. >>Do you have a date on that? Do you have a date on that Argo con 22? >>Two? Yeah, yeah, yeah. Argo con September 19th, 2022. So, you know, mark your calendars, it it's, you know, it's a multi-day event, you know, it's, it's part of something else that I've seen in the community where, you know, first we're talk talking about these meetups. Now we're doing multi-day events. We're, you know, in talks of the open, get ups, you know, get ups can also make that a multi-day event. There's just so many talks in so many people that want to be involved in network that, you know, we're saying, well, we're gonna need more days because there's just so many people coming to these events, you know, in, in, you know, seeing these communities grow, not just from like the engineering standpoint, but also from the end user standpoint, but also from the people that are actually doing these things. And, you know, seeing some of these use cases, seeing some of the success, seeing some of the failures, right? Like people love listening to those talks about postmortems, I think are part of my favorite talks as well. So seeing that community grow is, is, you know, on a personal level, it's, it's a point >>It's like CSI for software developers. You want to curious about >>Exactly >>What happened. You know, you know, it's interesting, you mentioned about the, the multiple events at Coon. You know, the vibe that's going on is a very festival vibe, right? You have organic groups coming together. I remember when they had just started doing the day zero programs. Now you have like, almost like multiple stages of content at these events. It feels like, like a Coachella vibe or some sort of like festival vibe, like a lot of things going on and you, and if you pick your kind of area, but you can move around, I find that the kind of the format de Azure I think is going well these days. What do you think about that? >>Yeah, yeah. No, for sure. It's and, and, and I love that that analogy of Coachella, it does feel like, you know, it's, there's something for everyone and you can find what you like, and you'll find a little, you know, a little group, right. A little click of, of, of people that's probably the wrong term to use, but you know, you, you find, you know, you, you know, like-minded people and, you know, passionate about the same thing, right? Like the security guys, they, you know, you see them all clump together, right? Like you see like the, the developer C I CD get ops guys, we all kind of clump together and start talking, you know, about everything that we're doing. And it's, that's, that's, I think that's really something special that coupon, you know, some, you know, it's gotten so big that it's almost impossible to fit everything in a, in a week, because unless there's just so much to do. And there's so much that that interests, you know, someone, but it's >>A code, a code party is what we call it. It's a code party. Yeah. >>It's, it's a code party for sure. For >>Sure. Nerd nerd Fest on, on steroids. Hey, I gotta get, I wanna wrap this up and give you the final word, Christian. Thanks for coming on. Great insight, great conversation. There's a huge, you guys are in the middle of a hot area, obviously large scale data growth. Kubernetes is scaling beautifully and making it easier at managed services. What people want machine learning's kicking in and, and you get automation building in all favoring, the developer and C I CD pipeline and all that good stuff. People want to learn more. Can you take a minute to put the plug in for code fresh on the certification? How do I get involved? Where are you? Is there levels if I want to jump in and get trained and get fluent on code fresh, can you share commentary and, and, and what the status is? >>Yeah, yeah, for sure. So code fresh is offering a free certification, right? For get ups or Argo CD and get ops. The first of it's kind for Argo CD, first of it's kind for get ops is you can actually go get certified with Argo CD and get ops. You know, we there level one is out right now. You can go take that code, fresh.io/certification. It's out there, sign up, you know, you, you don't, you don't need to pay anything, right. It's, it's something it's a, of a free course. You could take level two is coming soon. Right? So level two is coming soon in the next few months, I believe I don't wanna quote a specific day, but soon because I, but soon I, it it's soon, soon as in, as in months. Right? So, you know, we're, we're counting that down where you can not only level one cert level certification, but a level, two more advanced certification for those who have been using Argo for a while, they can still, you know, take that and be, you know, be able to get, you know, another level of certification for that. So also, you know, Argo con will be there. We're, we're part of the programming committee for Argo con, right? This is a community driven event, but, you know, code fresh is a proud diamond sponsor. So we'll be there. >>Where's it located up to us except for eptember 19th multiday or one day >>It's a, it's a multi-day event. So Argo con from 19, 19 20 and 21 in a mountain view. So it'll be in mountain view in the bay area. So for those of you who are local, you can just drive in. Great. >>I'm write that down. I'll plug it. I'll put in the show notes. >>Awesome. Awesome. Yeah. And you will be there so you can talk to me, you can talk to anyone else at code, fresh talking about Argo CD, you know, find, find out more about hosted, get ups code, fresh.io. You know, you can find us in the Argo project, open, get ups community, you know, we're, we're, we're deep in the community for both Argo and get ups. So, you know, you can find us there as well. >>Well, let's do a follow up in when you're in town, so's only a couple months away and getting through the summer, it's already, I can't believe events are back. So it's really great to see face to face in the community. And there was responding. I mean, co con in October, I think that was kind of on the, that was a tough call and then get to see your own in Spain. I couldn't make it. Unfortunately, I had got COVID came down with it, but our team was there. Open sources, booming continues to go. The next level, new power dynamics are developing in a great way. Christian. Thanks for coming on, sharing your insights as the developer experience lead at code fresh. Thanks so much. >>Thank you, John. I appreciate it. >>Okay. This is a cube conversation. I'm John feer, host of the cube. Thanks for watching.

(bright upbeat music) >> Hey everyone, welcome to theCUBE's AWS startup showcase. This is season two of the startup showcase, episode one. I'm your host, Lisa Martin. Pleased to be welcoming back one of our alumni, Alexis Richardson, the founder >> Hey. >> and CEO of Weaveworks. Alexis, welcome back to the program. >> Thank you so much, Lisa, I'm really happy to be here. Good to see you again. >> Likewise. So it's been a while since we've had Weaveworks on the program. Give the audience an overview of Weaveworks. You were founded in 2014, pioneering getopts, automating Kubernetes across all industries, but help us understand, unpack that a bit. >> Well, so my previous role was at Pivotal, where I was head of application platform and I was responsible for Spring and Vfabric, and some pieces of Cloud Foundry. And you may remember back in those days, everybody wanted to build like a Heroku, but for the enterprise. And so they were asking, how can we build more cloud services? And my team was involved in building out cloud services, but we were running into trouble with the technology that we had. And then when containers appeared, we thought this is the technology for us to roll out cloud services. So with some of my team, we decided to start a new company, Weaveworks, really intending to focus on developers. Because these new containers were pretty cool, but they were really complex operational centric tools, and enterprise developers need simplicity. That's what we'd learned from things like Spring. They want simplicity, productivity, velocity, all of that stuff, they don't want operational complexity. So Weaveworks' mission is to make applications easy for developers with containers. >> Talk to me about how you've accomplished that over the last seven years, and some of the things that you're doing to facilitate a DevOps practice within organizations across any industry? >> Yeah, well, our story is pretty interesting because of course in 2014, all of this was incredibly new. You couldn't even take two containers and put them together into a single application. So forget about enterprise. What we did was we built a network, which gave the company its name, Weave. But then we spent several years building out more and more pieces of the stack. We decided that we should go to market commercially because we're an open source company with a commercial SaaS. And we thought we would be like new Relic, that there'll be lots of customers in the cloud. And, therefore, they would need monitoring and management. And Weave started writing a SaaS based on Kubernetes, which was what we chose as our platform, back in the day, very, very, very early. We were one of the very first companies to start running Kubernetes in production other than Google. And so what we learned was customers didn't want to have management and monitoring for applications in the cloud, based on Kubernetes. Because they were all still struggling to get Docker working, to get basic Kubernetes clusters set up. And they kept saying to us "this is great, we love your tool, but we really need simpler things right now." So what we had done was we'd learned how to operate Kubernetes. And we discovered that we were doing it in this specific way, a way that meant that we could be reliable, we could set things up remotely, we could move things between zones. And so we called this approach getopts. So we've named the practice of getopts, which is really DevOps for Kubernetes. We decided that it was exciting after we had an outage and made a very quick recovery. Told people about it and they said, "well, we can't even Kubernetes started, let alone recover it from a crash." So we started evangelizing getopts and saying to people that we knew how to set up and run Kubernetes as operators for developers of apps, based on this experience. And people said, "well, why don't you help us do that?" So we pivoted the company away from a SaaS business, doing management, and straight back into enterprise software, providing a solution for people to run Kubernetes stacks, deploy applications, detect drifts, and operate them at scale. And we've never looked back. And since then we've built, very successfully, a big business out of telco customers, banks, car companies, really global two thousands. Starting from that open source base, continuing to respect that, but always keeping in mind helping developers build applications at scale. >> So in terms of that pivot that you've made, it sounds like you made that in conjunction with developers across industries to really understand what the right direction is here. What's the approach, what's their appetite? Talk to me about a customer example or two that really you think articulate the value and the right decision that that pivot was and how you're helping customers to really further their DevOps practice. >> Well, one of our first customers was actually Fidelity in this new world. Fidelity has a very advanced technology organization, a very forward thinking CTO, who I seem to recall is, or CEO, who I think is female. Really is into technology as a source of, you know, velocity and business strength. And we were brought to Fidelity by our partner, Amazon. And they said, "look, Fidelity have been using your open source tools, they want to run on Kubernetes, the early EKS service on AWS, but they need help, because what they want is a shared application platform that people can use across Fidelity to deploy and manage apps." So the idea Fidelity had was they're going to split their IT into a platform team, that was going to provide this platform, and a bunch of app teams that were going to write business apps like risk management, other financial processing. Paths, basically. And we came in to help Fidelity. And what we did was help Fidelity rollout, using getopts, a Amazon wide application platform. We also helped them to build, this was very early days for us post pivot, we really helped them to build an add on layer. So you could take any Kubernetes cluster and add other components to it, and then you'd have your platform right there. And the whole stack would be managed by getopts, which nobody had done before. Nobody who'd come up with a way of managing the whole stack, so you could start and stop stacks wherever you wanted, at will, correctly. I mean, if you talk to people about what's hard in IT, they'll tell you shutting down Kubernetes is hard, 'cause I know I'm never going to know how to start it again. So being able to start and stop things, move them around is really crucial. What Fidelity also wanted, which made I think the whole thing even more exciting, was to duplicate this environment on Azure and actually also on-premise later on. So where Fidelity are today is the whole Fidelity platform runs on Microsoft and on Amazon and on-premise, using three different implementations of Kubernetes. But using this platform technology and getopts that we helped Fidelity rollout. And if you want to know a bit about the story, type FIDEKS, F I D E K S into Google and you'll find a video of me three or four years ago on stage at Cube Con talking with a Fidelity chief architect about this story. It's pretty exciting and these are early days for these new Kubernetes platforms. >> Early days, but so transformative. And I can't imagine the events of the last few years without having this capability and this technology to facilitate such pivots and transformation where we would all be. I want to kind of dig into some use cases, 'cause one of the things that you just mentioned with the Fidelity example got me thinking use case of hybrid, multi-cloud, but also continuous app development. Talk to me about some of the key use cases that you work with customers on. >> Well you just named two. So hybrid and multi-cloud is absolutely critical, and also sovereign, which is when you're actually offline and you only update your cloud periodically. That's one of the major use cases for us. And what customers want there is they want consistency. They want a single operating model, across all of these different locations, so that all of their teams can get trained on one set of technologies and then move from place to place. They're not looking for magic, where apps move with the sun or any of that stuff. They just want to know they can base everything on a single, homogeneous skillset and have scale across their teams. Maybe tens of thousands of developers, all who know how to do the same thing. That's a really important use case. You also mentioned continuous delivery. That's probably the second really critical use case for us. People say, "I've got Kubernetes set up now, and I have Jenkins." At JP Morgan once told me they had 40,000 Jenkins servers, or something like that, you know, Jenkins at scale. And they're like, "okay, how do I push changes from Jenkins into the cloud?" So getopts provides a bridge between the world of CI and the runtime of Kubernetes. So one group of our customers is help me to put that middle piece of CD that gets you CI, CD, to Kubernetes, that's a classic. And then what they're looking for is an increase in velocity. And what we typically see is people go from deploying once every six months to deploying once a week, to deploying once a day, to deploying several times a day. And then they split things up into teams and suddenly, wow, that vision of microservices has come and everybody's excited 'cause IT velocity has gone up by two X. Another really >> So, >> Sorry, carry on. >> Go ahead, I was just going to say in terms of IT velocity it sounds like that's a major business outcome that you're enabling for, whether it's teleco, financial services, or whatnot. That velocity is, as you just described, is rapidly accelerating. >> Yeah, if you go to our website, you'll find a bunch of these use cases. And one that I really like is NatWest mettle, which is another financial example. They're not all financial by the way. But there's some metrics in there. We're getting people up to two X productivity, which at scale is huge, really makes a difference. Also, meantime to recovery. If you know the metric space, you'll know these are all DORA metrics. And DORA, which was acquired by Google a couple of years ago, is a really fantastic analyst in the space that came up with a bunch of ways of thinking about how to measure your performance as a business and IT organization. Recovery time and things like this that you really need to focus on if you're in this world. >> Well, from an IT velocity perspective, if I translate that to business outcomes, especially given the dynamics in the market over the last two years, this is transformative and probably helped a lot of organizations to pivot multiple times during the last couple of years. To get to that survival mode and into that thriving mode, enabling organizations to meet customer demand that was changing faster, et cetera. That's a really big imperative that this technology can deliver to the business. >> Yeah, I mean, that's been huge for us. So when the pandemic first began, obviously, we had some road bumps and there were some challenges, but what we found out very quickly was that people were moving into digital much faster. And we've been mostly enabling them, not just in finance, as I said, but also, car companies, utilities, et cetera. The other one, of course, is modern operations. So, everyone's excited about the potential for automation. If I have thousands and thousands of developers and thousands of applications, do I need thousands of operations staff? And the answer is, with Kubernetes in this new era, you can reduce your operational loads. So that actually very few people are needed to keep systems up, to do basic monitoring, to do redeployments and so on, which are all boring infrastructure tasks that no developer wants to do. If we can automate all of that, we can modernize the whole IT space. And that's what I think the promise of Kubernetes that we're also seeing as well. So applications speed first and then operational competence second. >> So you guys had a launch, here we are in early calendar year 2022, you guys had a launch just about six or eight weeks ago in November of 2021, where you were launching announcing the GA of Weave getopts enterprise, which is a licensed product building on the free open source Weave getopts core. Talk to me about that and what the significance of that is. >> Well, this is an enterprise solution that helps customers build these critical use cases, like shared service platform or secure DevOps or multi-cloud, using getopts, which gives them higher security, lower costs of management, and better operations, and higher velocity. And all of it is taking all the best practices that we've learned starting from those days of running our own Kubernetes stack and then through those early customers like Fidelity into the modern era where we have an at-scale platform for these people. And the crucial properties are it provides you with a platform, it provides you with trusted delivery, and it provides you with what we call release orchestration, which is when you deploy things at scale into production, using tools like canaries and other modern practices. So, all of it is enabling what we call the cloud native enterprise, application delivery, modern operations. >> So what's the upgrade path for customers that are using the free open-source tier to the enterprise package, what does that look like? >> The good news is it's an add on. So, I have been in the industry a while and I strongly believe it's really important that if you have an open source product, you shouldn't ask people to delete it or uninstall it to install your enterprise product, unless you really, really, really have to. And I'm not trying to be picky here. Maybe there are cases where it's important, but actually in our case, it's very simple. If you're already using one of our upstream tools, like Flux, for example, then going from Flux to Weave getopts enterprise is an add-on installation. So you don't have to change or take out what you're doing. You might be using Flux without knowing it. You may not be aware of this, but it's also insight as your AKS and ARC, it's inside the Amazon EKS anywhere bundle. It's available on Alibaba, VMware have used it in cartographer and Tanzu application platform. And even Red Hat use it too in some cases. So you may be using it already, from one of the big vendors who are partners of ours, as a precursor to buying Weave getopts enterprise. So, you know, don't be scared. Get in touch is what I would say to people. >> Get in touch. And of course, folks can go to weave.works to learn more about that. And, also we want to watch the Weave.works space, 'cause you have some news coming out relatively soon that sounds pretty exciting, Alexis. >> Well, I mentioned trusted delivery. And I think one of the things with that is no CIO wants to go faster, unless they also have the safety wheels on, let's face it. And the big question we get asked is "I love this getopts stuff, but how can I bring my team with me? How can I introduce change?" I have all of these approvals mechanisms in place, can I move into the world of getopts? And the answer is yes, yes you can because we now support policy engines as baked into our enterprise product. Now, if you don't know what policy is, it's really a way of applying rules to what you're seeing in IT. And you can detect whether something passes or fails conditions, which means that we can detect if something bad is about to happen in a deployment and stop it from happening, this is really critical. It also goes hand in hand with things like supply chain and security, which I'm sure we read about in the news far too much. >> Yeah, pretty much daily supply chain and security >> Pretty much daily. >> is one of those things that we're all in every generation concerned about. Well, Alexis, it's been a pleasure having you back on the program, talking to us about what's new at Weaveworks, the direction that you're going, how you're helping organizations across industries really advance their DevOps practice. And we will check weave.works in the next couple of weeks for more on that news that you started to break a little bit with us today. We appreciate your time, Alexis. >> Thank you very much, indeed, take care. >> Likewise. For Alexis Richardson, I'm Lisa Martin. Keep it right here on theCUBE, your leader in hybrid tech event coverage. (bright music) (music fades)

>>From around the globe. It's the queue with digital coverage of AWS reinvent 2020 sponsored by Intel, AWS and our community partners. >>Welcome back to the cubes. Live coverage for ADFS reinvent 2020 I'm John Ford with the cube, your host. We are the cube virtual. We're not there in person this year. We're remote with the pandemic and we're here for the keynote analysis for Verner Vogels, and we've got some great analysts on and friends of the cube cube alumni is Rob Hirschfeld is the founder and CEO of Rakin a pioneer in the dev ops space, as well as early on on the bare metal, getting on the whole on-premise he's seen the vision and I can tell you, I've talked to him many times over the years. He's been on the same track. He's on the right wave frog. Great to have you on. I'm going to have to start Veatch, come on. Y'all come on as well, but great to see you. Thanks, pleasure to be here. Um, so the keynote with Verna was, you know, he's like takes you on a journey, you know, and, and virtual is actually a little bit different vibe, but I thought he did an exceptional job of stage layout and some of the virtual stage craft. Um, but what I really enjoyed the most was really this next level, thinking around systems thinking, right, which is my favorite topic, because, you know, we've been saying, going back 10 years, the cloud is just, here's a computer, right. It's operating system. And so, um, this is the big thing. This is, what's your reaction to the keynote. >>Wow. So I think you're right. This is one of the challenges with what Amazon has been building is it's, you know, it is a lock box, it's a service. So you don't, you don't get to see behind the scenes. You don't really get to know how they run these services. And what, what I see happening out of all of those pieces is they've really come back and said, we need to help people operate this platform. And, and that shouldn't be surprising to anyone. Right? Last couple of years, they've been rolling out service, service service, all these new things. This talk was really different for Verner's con normal ones, because he wasn't talking about whizzbang new technologies. Um, he was really talking about operations, um, you know, died in the wool. How do we make the system easier to use? How do we expose things? What assistance can we have in, in building applications? Uh, in some cases it felt like, uh, an application performance monitoring or management APM talk from five or even 10 years ago, um, canaries, um, you know, Canary deployments, chaos engineering, observability, uh, sort of bread and butter, operational things. >>We have Savi Joel, who's a influencer cloud computing Xtrordinair dev ops guru. Uh, we don't need dev ops guru from Amazon. We got Sarpy and prop here. So it'd be great to see you. Um, you guys had a watch party. Um, tell me what the reaction was, um, with, of the influencers in the cloud or ADI out there that were looking at Vernon's announcement, because it does attract a tech crowd. What was your take and what was the conversation like? >>Yeah, we kinda geeked out. Um, we had a watch party and we were commenting back and forth, like when we were watching it. I think that the general consensus is that the complexity of AWS stack itself is, is increasing. Right. And they have been focused on developers a lot, I think a lot longer than they needed to be a little bit. I think, uh, now they need to focus on the operations. Like we, we are, we all love dev ops talks and it's very fancy and it's very modern way of building software. But if you think deep down that, like once we developed software traditionally and, and also going forward, I think we need to have that separation. Once you develop something in production, it's, it's, it's operating right. Once you build a car, you're operating car, you're not building car all the time. Right? >>So same with the software. Once you build a system, it should have some stability where you're running it, operating it for, for a while, at least before you touch it or refactoring all that stuff. So I think like building and operating at the same time, it's very good for companies like Amazon, AWS, especially, uh, and, and Google and, and, and Facebook and all those folks who are building technology because they are purely high-tech companies, but not for GM Ford Chrysler or Kaiser Permanente, which is healthcare or a school district. The, they, they need, need to operate that stuff once it's built. So I think, uh, the operationalization of cloud, uh, well, I think take focus going forward a lot more than it has and absorbable Deanna, on a funny note, I said, observability is one of those things. I, now these days, like, like, you know, and the beauty pageants that every contestant say is like, whatever question you asked, is it Dora and the answer and say at the end world peace, right? >>And that's a world peace term, which is the absorbability. Like you can talk about all the tech stuff and all that stuff. And at the end you say observability and you'll be fine. So, um, what I'm making is like observability is, and was very important. And when I was talking today about like how we can enable the building of absorbability into this new paradigm, which is a microservices, like where you pass a service ID, uh, all across all the functions from beginning to the end. Right. And so, so you can trace stuff. So I think he was talking, uh, at that level. Yeah. >>Let me, let's take an observer Billy real quick. I have a couple of other points. I want to get your opinions on. He said, quote, this three, enabling major enabling technologies, powering observability metrics, logging and tracing here. We know that it would, that is of course, but he didn't take a position. If you look at all the startups out there that are sitting there, the next observability, there's at least six that I know of. I mean, that are saying, and then you got ones that are kind of come in. I think signal effects was one. I liked, like I got bought by Splunk and then is observability, um, a feature, um, or is it a company? I mean, this is something that kind of gets talked about, right? I mean, it's, I mean, is it really something you can build a business on or is it a white space? That's a feature that gets pulled in what'd you guys react to that? >>So this is a platform conversation and, and, you know, one of the things that we've been having conversations around recently is this idea of platforms. And, and, you know, I've been doing a lot of work on infrastructure as code and distributed infrastructure and how people want infrastructure to be more code, like, which is very much what, what Verna was, was saying, right? How do we bring development process capabilities into our infrastructure operations? Um, and these are platform challenges. W what you're asking about from, uh, observability is perspective is if I'm running my code in a platform, if I'm running my infrastructure as a platform, I actually need to understand what that platform is doing and how it's making actions. Um, but today we haven't really built the platforms to be very transparent to the users. And observability becomes this necessary component to fix all the platforms that we have, whether they're Kubernetes or AWS, or, you know, even going back to VMware or bare metal, if you can't see what's going on, then you're operating in the blind. And that is an increasingly big problem. As we get more and more sophisticated infrastructure, right? Amazon's outage was based on systems can being very connected together, and we keep connecting systems together. And so we have to be able to diagnose and troubleshoot when those connections break or for using containers or Lambdas. The code that's running is ephemeral. It's only around for short periods of time. And if something's going wrong in it, it's incredibly hard to fix it, >>You know? And, and also he, you know, he reiterated his whole notion of log everything, right? He kept on banging on the drum on that one, like log everything, which is actually a good practice. You got to log everything. Why wouldn't you, >>I mean, how you do, but they don't make it easy. Right? Amazon has not made it easy to cross, cross, and, uh, connect all the data across all of those platforms. Right? People think of Amazon as one thing, but you know, the people who are using it understand it's actually a collection of services. And some of those are not particularly that tied together. So figuring out something that's going on across, across all of your service bundles, and this isn't an Amazon problem, this is an industry challenge. Especially as we go towards microservices, I have to be able to figure out what happened, even if I used 10 services, >>Horizontal, scalability argument. Sorry. Do you want to get your thoughts on this? So the observability, uh, he also mentioned theory kind of couched it before he went into the talk about systems theory. I'm like, okay. Let's, I mean, I love systems, and I think that's going to be the big wake up call here for the next 10 years. That's a systems mindset. And I think, you know, um, Rob's right. It's a platform conversation. When you're thinking about an operating system or a system, it has consequences when things change, but he talked about controllability versus, uh, observability and kinda T that teed up the, well, you can control systems controls, or you can have observability, uh, what's he getting at in all of this? What's he trying to say, keep, you know, is it a cover story? Is it this, is it a feature? What was the, what was the burner getting at with all this? >>Uh, I, I, I believe they, they understand that, that, uh, that all these services are very sort of micro in nature from Amazon itself. Right. And then they are not tied together as Rob said earlier. And they, he addressed that. He, uh, he, uh, announced that service. I don't know the name of that right now of problem ahead that we will gather all the data from all the different places. And then you can take a look at all the data coming from different services at this at one place where you have the service ID passed on to all the servers services. You have to do that. It's a discipline as a software developer, you have to sort of adhere to even in traditional world, like, like, you know, like how you do logging and monitoring and tracing, um, it's, it's your creativity at play, right? >>So that's what software is like, if you can pass on, I was treating what they gave an example of Citrix, uh, when, when, when you are using like tons of applications with George stream to your desktop, through Citrix, they had app ID concept, right? So you can trace what you're using and all that stuff, and you can trace the usage and all that stuff, and they can, they can map that log to that application, to that user. So you need that. So I think he w he was talking about, I think that's what he's getting too. Like we have to, we have to sort of rethink how we write software in this new Microsoft, uh, sort of a paradigm, which I believe it, it's a beautiful thing. Uh, as long as we can manage it, because Microsoft is, are spread across like, um, small and a smaller piece of software is everywhere, right? So the state, how do we keep the state intact? How do we, um, sort of trace things? Uh, it becomes a huge problem if we don't do it right? So it it's, um, it's a little, this is some learning curve for most of the developers out there. So 60 dash 70% >>Rob was bringing this up, get into this whole crash. And what is it kind of breakdown? Because, you know, there's a point where you don't have the Nirvana of true horizontal scalability, where you might have microservices that need to traverse boundaries or systems, boundaries, where, or silos. So to Rob's point earlier, if you don't see it, you can't measure it or you can't get through it. How do you wire services across boundaries? Is that containers, is that, I mean, how does this all work? How do you guys see that working? I just see a train wreck there. >>It's, it's a really hard problem. And I don't think we should underestimate it because everything we toast talked about sounds great. If you're in a single AWS region, we're talking about distributed infrastructure, right? If you think about what we've been seeing, even more generally about, you know, edge sites, uh, colo on prem, you know, in cloud multi-region cloud, all these things are actually taking this one concept and you're like, Oh, I just want to store all the log data. Now, you're not going to store all your log data in one central location anymore. That in itself, as a distributed infrastructure problem, where I have to be able to troubleshoot what's going on, you know, and know that the logs are going to the right place and capture the data, that's really important. Um, and one of the innovations in this that I think is going to impact the industry over the next couple of years is the addition of more artificial intelligence and machine learning, into understanding operations patterns and practices. >>And I think that that's a really significant industry trend where Amazon has a distinct advantage because it's their systems and it's captive. They can analyze and collect a lot of data across very many customers and learn from those things and program systems that learn from those things. Um, and so the way you're going to keep up with this is not by logging more and more data, but by doing exactly what we're talking through, which was how do I analyze the patterns with machine learning so that I can get predictive analysis so that I can understand something that looks wrong and then put people on checking it before it goes wrong. >>All right, I gotta, I gotta bring up something controversial. I can't hold back any longer. Um, you know, Mark Zuckerberg said many, many years ago, all the old people, they can do startups, they're too old and you gotta be young and hungry. You gotta do that stuff. If we're talking systems theory, uh, automated meta reasoning, evolvable systems, resilience, distributed computing, isn't that us old guys that have actually have systems experience. I mean, if you're under the age of 30, you probably don't even know what a system is. Um, and, or co coded to the level of systems that we use to code. And I'm putting my quote old man kind of theory, only kidding, by the way on the 30. But my point is there is a generation of us that had done computer science in the, in the eighties and seventies, late seventies, maybe eighties and nineties, it's all it was, was systems. It was a systems world. Now, when you have a software world, the aperture is increasing in terms of software, are the younger generation of developers system thinkers, or have we lost that art, uh, or is it doesn't matter? What do you guys think? >>I, I think systems thinking comes with age. I mean, that's, that's sort of how I think, I mean, like I take the systems thinking a greater sort of, >>Um, world, like state as a system country, as a system and everything is a system, your body's a system family system, so it's the same way. And then what impacts the system when you operated internal things, which happened within the system and external, right. And we usually don't talk about the economics and geopolitics. There's a lot of the technology. Sometimes we do, like we have, I think we need to talk more about that, the data sovereignty and all that stuff. But, but even within the system, I think the younger people appreciate it less because they don't have the, they don't see, um, software taught like that in the universities. And, and, and, and by these micro micro universities now online trainings and stuff like sweaty, like, okay, you learn this thing and you're good at it saying, no, no, it's not like that. So you've got to understand the basics and how the systems operate. >>Uh, I'll give you an example. So like we were doing the, the, the client server in early nineties, and then gradually we moved more towards like having ESB enterprise services, bus where you pass a state, uh, from one object to another, and we can bring in the heterogeneous, uh, languages. This thing is written in Java. This is in.net. This is in Python. And then you can pass it through that. Uh, you're gonna make a state for, right. And that, that was contained environment. Like ESBs were contained environment. We were, I, I wrote software for ESPs myself at commerce one. And so like, we, what we need today is the ESP equallant in the cloud. We don't have that. >>Rob, is there a reverse ageism developers? I mean, if you're young, you might not have systems. What do you think? I, I don't agree with that. I actually think that the nature of the systems that we're programming forces people into more distributed infrastructure thinking the platforms we have today are much better than they were, you know, 20 years ago, 30 years ago, um, in the sense that I can do distributed infrastructure programming without thinking about it very much anymore, but you know, people know, they know how to use cloud. They know how to use a big platform. They know how to break things into microservices. I, I think that these are inherent skills that people need to think about that you're you're right. There is a challenge in that, you know, you get very used to the platform doing the work for you, and that you need to break through it, but that's an experiential thing, right? >>The more experienced developers are going to have to understand what the platforms do. Just like, you know, we used to have to understand how registers worked inside of a CPU, something I haven't worried about for a long, long time. So I, I don't think it's that big of a problem. Um, from, from that perspective, I do think that the thing that's really hard is collaboration. And so, you know, it's, it's hard people to people it's hard inside of a platform. It's hard when you're an Amazon size and you've been rolling out services all over the place and now have to figure out how to fit them all together. Um, and that to me is, is a design problem. And it's more about being patient and letting things, uh, mature. If anything might take away from this keynote is, you know, everybody asked Amazon to take a breath and work on usability and, and cross cross services synchronizations rather than, than adding more services into the mix. And that's, >>That's a good point. I mean, again, I bring up the conversation because it's kind of the elephant in the room and I make it being controversial to make a point there. So our view, because, you know, I interviewed Judy Estrin who helped found the internet with Vince Cerf. She's well-known for her contributions for the TCP IP protocol. Andy Besta Stein. Who's the, who's the Rembrandt of motherboards. But as Pat Gelsinger, CEO of VMware, I would say both said to me on the cube that without systems thinking, you don't understand consequences of when things change. And we start thinking about this microservices conversation, you start to hear a little bit of that pattern emerging, where those systems, uh, designs matter. And then you have, on the other hand, you have this modern application framework where serverless takes over. So, you know, Rob back to your infrastructure as code, it really isn't an either, or they're not mutually exclusive. You're going to have a set of nerds and geeks engineering systems to make them better and easier and scalable. And then you're going to have application developers that need to just make it work. So you start to see the formation of kind of the, I won't say swim lanes, but I mean, what do you guys think about that? Because you know, Judy and, um, Andy better sign up. They're kind of right. Uh, >>Th th the enemy here, and we're seeing this over and over again is complexity. And, and the challenge has been, and serverless is like, those people like, Oh, I don't have to worry about servers anymore because I'm dealing with serverless, which is not true. What you're doing is you're not worrying about infrastructure as much, but you, the complexity, especially in a serverless infrastructure where you're pulling, you know, events from all sorts of things, and you have one, one action, one piece of code, you know, triggering a whole bunch of other pieces of code in a decoupled way. We are, we are bringing so much complexity into these systems, um, that they're very hard to conceive of. Um, and AIML is not gonna not gonna address that. Um, I think one of the things that was wonderful about the setting, uh, in the sugar factory and at all of that, you know, sort of very mechanical viewpoint, you know, when you're actually connecting all things together, you can see it. A lot of what we've been building today is almost impossible to observe. And so the complexity price that we're paying in infrastructure is going up exponentially and we can't sustain infrastructures like that. We have to start leveling that in, right? >>Your point on the keynote, by the way, great call out on, on the, on the setting. I thought that was very clever. So what do you think about this? Because as enterprises go through this transformation, one of the big conversations is the solution architecture, the architecture of, um, how you lay all this out. It's complexity involved. Now you've got on premise system, you've got cloud, you've got edge, which you're hearing more and more local processing, disconnected systems, managing it at the edge with visualization. We're going to hear more about that, uh, with Dirk, when he comes on the queue, but you know, just in general as a practitioner out there, what, what's, what's your, what do you see people getting their arms around, around this, this keynote? What do they, what's your thoughts? >>Yeah, I, I think, uh, the, the pattern I see emerging is like, or in the whole industry, regardless, like if you put, when does your sign is that like, we will write less and less software in-house I believe that SAS will emerge. Uh, and it has to, I mean, that is the solution to kill the complexity. I believe, like we always talk about software all the time and we, we try to put this in the one band, like it's, everybody's dining, same kind of software, and they have, I'm going to complexity and they have the end years and all that stuff. That's not true. Right. If you are Facebook, you're writing totally different kind of software that needs to scale differently. You needs a lot of cash and all that stuff, right. Gash like this and cash. Well, I ain't both gases, but when you are a mid size enterprise out there in the middle, like fly over America, what, uh, my friend Wayne says, like, we need to think about those people too. >>Like, how do they drive software? What kind of software do they write? Like how many components they have in there? Like they have three tiers of four tiers. So I think they're a little more simpler software for internal use. We have to distinguish these applications. I always talk about this, like the systems of record systems of differentiation, the system of innovation. And I think cloud will do great. And the newer breed of applications, because you're doing a lot of, a lot of experimentation. You're doing a lot of DevOps. You have two pizza teams and all that stuff, which is good stuff we talk about, well, when you go to systems of record, you need stability. You need, you need some things which is operational. You don't want to touch it again, once it's in production. Right? And so the, in between that, that thing is, I think that's, that's where the complexity lies the systems are, which are in between those systems of record and system or innovation, which are very new Greenfield. That, that's what I think that's where we need to focus, uh, our, um, platform development, um, platform as a service development sort of, uh, dollars, if you will, as an industry, I think Amazon is doing that right. And, and Azura is doing that right to a certain extent too. I, I, I, I worry a little bit about, uh, uh, Google because they're more tilted towards the data science, uh, sort of side of things right now. >>Well, Microsoft has the most visibility into kind of the legacy world, but Rob, you're shaking your head there. Um, on his comment, >>You know, I, I, you know, I, I watched the complexity of all these systems and, and, you know, I'm not sure that sass suffocation of everything that we're doing is leading to less is pushing the complexity behind a curtain so that you, you, you can ignore the man behind the curtain. Um, but at the end of the day, you know what we're really driving towards. And I think Amazon is accelerating this. The cloud is accelerating. This is a new set of standard operating processes and procedures based on automation, based on API APIs, based on platforms, uh, that ultimately, I think people could own and could come back to how we want to operate it. When I look at what we w we were just shown with the keynote, you know, it was an, is things that application performance management and monitoring do. It's, it's not really Amazon specific stuff. There's no magic beans that Amazon is growing operational knowledge, you know, in Amazon, greenhouses that only they know how to consume. This is actually pretty block and tackle stuff. Yeah. And most people don't need to operate it at that type of scale to be successful. >>It's a great point. I mean, let's, let's pick up on that for the last couple of minutes we have left. Cause I think that's a great, great double-down because you're thinking about the mantra, Hey, everything is a service, you know, that's great for business model. You know, you hand it over to the techies. They go, wait a minute. What does that actually mean? It's harder. But when I talk to people out there and you hear people talking about everything is a service or sanctification, I do agree. I think you're putting complexity behind the curtain, but it's kind of the depends answer. So if you're going to have everything as a service, the common thesis is it has to have support automation everywhere. You got to automate things to make things sassiphy specified, which means you need five nines, like factory type environments. They're not true factories, but Rob, to your point, if you're going to make something a SAS, it better be Bulletproof. Because if you're, if you're automating something, it better be automated, right? You can measure things all you want, but if it's not automated, like a, like a, >>And you have no idea what's going on behind the curtains with some of these, these things, right. Especially, you know, I know our business and you know, our customers' businesses, they're, they're reliant on more and more services and you have no idea, you know, the persistence that service, if they're going to break an API, if they're going to change things, a lot of the stuff that Amazon is adding here defensively is because they're constantly changing the wheels on the bus. Um, and that is not bad operational practice. You should be resilient to that. You should have processes that are able to be constantly updated and CICB pipelines and, you know, continuous deployments, you shouldn't expect to, to, you know, fossilize your it environment in Amber, and then hope it doesn't have to change for 10 years. But at the same time, we'll work control your house. >>That's angle about better dev ops hypothetical, like a factory, almost metaphor. Do you care if the cars are being shipped down the assembly line and the output works and the output, if you have self-healing and you have these kinds of mechanisms, you know, you could have do care. The services are being terminated and stood up and reformed as long as the factory works. Right? So again, it's a complexity level of how much it, or you want to bite off and chew or make work. So to me, if it's automated, it's simple, did it work or not? And then the cost of work to be, what's your, what's your angle on this? Yeah. >>I believe if you believe in systems thinking, right. You have to believe in, um, um, the concept of, um, um, Oh gosh, I'm losing over minor. Um, abstraction. Right? So abstraction is your friend in software. Abstraction is your friend anyways, right? That's how we, humans pieces actually make a lot more progress than any other sort of living things here in this world. So that's why we are smart. We can abstract complexity behind the curtains, right? We, we can, we can keep improving, like from the, the, you know, wooden cart to the car, to the, to the plane, to the other, like, we, we, we have this, like when, when we see we are flying these airplanes, like 90% of the time they're on autopilot, like that's >>Hi, hiding my attractions is, is about evolution. Evolvable software term. He said, it's true. All right, guys, we have one minute left. Um, let's close this out real quick. Each of you give a closing statement on what you thought of the keynote and Verner's talk prop, we'll start with you. >>Uh, you know, as always, it's a perf keynote, uh, very different this year because it was so operationally focused and using the platform and, and helping people run their, their, off their applications and software better. And I think it's an interesting turn that we've been waiting for for Amazon, uh, to look at, you know, helping people use their own platform more. Um, so, uh, refreshing change and I think really powerful and well delivered. I really did like the setting >>Great shopping. And when we found, I found out today, that's Teresa Carlson is now running training and certification. So I'm expecting that to be highly awesomely accelerated a success there. Sorry, what's your take real quick on burners talk, walk away. Keynote thoughts. >>I, I, I think it was what I expected it to be like, he focused on the more like a software architecture kind of discussion. And he focused this time a little more on the ops side and the dev side, which I think they, they are pivoting a little bit, um, because they, they want to sell more AWS stuff to us, uh, to the existing enterprises. So I think, um, that was, um, good. Uh, I wish at the end, he said, not only like, go, go build, but also go build and operate. So can, you know, they all say, go build, build, build, but like, who's going to operate this stuff. Right. So I think, um, uh, I will see a little shift, I think, going forward, but we were talking earlier, uh, during or watch party that I think, uh, going forward, uh, AWS will open start open sourcing the commoditized version of their cloud, which have been commoditized by other vendors and gradually they will open source it so they can keep the hold onto the enterprises. I think that's what my take is. That's my prediction is >>Awesome and want, I'll make sure I'm at your watch party next time. Sorry. I missed it. Nobody's taking notes. Try and prepare. Sorry, Rob. Thanks for coming on and sharing awesome insight and expertise to experts in cloud and dev ops. I know them. And can firstly vouch for their awesomeness? Thanks for coming on. I think Verner can verify what I thought already was reporting Amazon everywhere. And if you connect the dots, this idea of reasoning, are we going to have smarter cloud? That's the next conversation? I'm John for your host of the cube here, trying to get smarter with Aus coverage. Thanks to Robin. Sarvi becoming on. Thanks for watching.

>>from around the globe. It's the Cube with digital coverage of AWS reinvent 2020 sponsored by Intel AWS and our community partners. Welcome to the cubes Coverage of AWS reinvent 2020. The digital version I'm Lisa Martin and I'm joined by a couple of guests from Unisys. Please welcome unprompted high BP and Cloud CTO on income. Great to have you on the program. Thanks for joining me today. >>Great to be here leader >>and a new pre or a new Ram Raj, VP of Cloud Services. A new welcome. Great to have you on a swell. Great >>to be here in this virtual AWS being that great. >>Thank you. Very socially Distance We're following all the guidelines here. A new Let's start with you. I'd love to get just kind of Ah, you know, a vision of the AWS Unisys partnership. I know you guys are advanced consulting partner MSP. Tell me about that partnership. >>Absolutely. Lisa, we see our clients on a cloud journey which we accelerate with Unisys Cloud Services and AWS partnership is a big piece of that again. Way thorough. We have bean rated in aws MSP partner Come out very, very highly from those msb audited our, uh and we're investing in multiple competencies across the boat as well. So and we work very closely with AWS in terms off innovating in sharing our platform cloud 44 world map In looking at what our customers looking around the corner, what services could be co developed. So we're looking at some potential I o T engagements to jointly with AWS is, well, eso you're always co inventing and it's a great partnership with a W s >>excellent. And you let's stick with you Following on hybrid Cloud Journey you mentioned the Cloud Forte platform. I wanted to understand what that platform is, how your co developing that with AWS and how your customers are benefiting >>absolutely s. Um, every year Unisys does a cloud barometer study across thousands of our clients and and we got some interesting takeaways from that. Essentially two thirds of her clients that have started this cloud journey believe they don't really realize the benefits out of that and up thio 53% off. The the respondents said they needed some help with cloud security. And this is where I believe that Unisys Cloud Services has a strong viewpoint and can find their AWS, um, journey, no matter where, what challenges they're facing, whether it's budgetary challenges on optimizing AWS and whether it's getting I t operations right when you move your applications to AWS. Um, and is it is it getting the that I have seen cops models established? So no matter where clients are in the A journey, we look to accelerate with our set of solutions and services, and we're very proud about the fact that we respond very me to make sure our clients can innovate and achieve the business outcomes that they need. For example, with California Stink City, we were able to work with them on the AWS. John Pretty set up a native other lake and analytics on top of it so we could actually predict and influence graduation rates with students. Our scores are higher than any off are coming because of the outcomes that we deliver for our clients. And it's really about business outcomes and 40 platform, which helps us drive those outcomes. I mean, probably do you want to add on without cloud 40 platform? >>Sure, I know eso, as as I knew was saying cloud for the platform provides AH set off capabilities that allows us to create an offer highly differentiated services with Unisys Pipe and, as was mentioned earlier, our cloud solutions are are able to help customers no matter where they are in their car, in their cloud journeys, whether it's ah Greenfield opportunity, where they where the customers are intending to move to the cloud, or if it's a brownfield opportunity where they already have adopted the cloud and are looking to manage and operate and optimize their deployments. Cloud Forte Platform and our Cloud Solutions are able to provide, uh, customized solution for that customer context to really deliver the solution that addresses some of the pain points that you talked about. The keeping points really relate to security to get secured. It also relates to cost optimization and then optimizing the cloud purse, a cloud deployment hybrid cloud deployment of the key requirement. So our cloud 40 platform health drives the key use cases. The key pain points that our customers are looking for through a combination off accelerators, the number of cloud photo accelerators that enable customers to rapidly prove it provisioned customers and to rapidly migrate to the cloud with God rails so that they're the secure, their compliant. And then we've got the the Cloud Cloud 40 Cloud management platform for ensuring provisioning onda management and operations, along with cost optimization capabilities and the eyelid operations. So it's a comprehensive suite off services and solutions that addresses the key business outcomes. There are customers are are looking for >>outcomes. Focused is absolutely critical, especially these days. I knew I wanted to go back to you for a second. You talked about the Unisys Barometer study, and I like the name of that. When was that done? And I'm just wondering if there are certain things that you saw this year from a customer. Cloud journey. Need perspective because of the pandemic that have really influenced that barometer >>Wait Question. Hey said and development is study. The last version of it was done late last year, and we're still waiting on the ones from this year. So, but we're starting to see some of the trends that were influenced by the pandemic. We saw rush to cloud when the pandemic hit because business adopt to to remote workers to do more digital selling and then seeing our CEO is kind of struggle with optimizing and maximizing the results off their cloud. Spend right, So So that's a unique challenge that that we're seeing based on our tryingto interaction. So the rush to the cloud and the ask for more spend optimization and in terms of spend optimization, that's an interesting facet because, uh, it cuts through my multiple angles. It's it's cuts through having the platforms around, being able to dio right predictions on where you spend is going, and then it also it's across collaborative effort. Finn ops. As we see it, we call it as a synopsis of is that we bring to our clients it's passing with multiple organizations, including finance, to sometimes figure out. Where will this business be? Where should you spend be? What should be the reserved instance buys right. So combining cloud knowledge with financial knowledge and organizational and business knowledge. And that's the service that we bring to our clients with our phenoms services. At least a great question about how how is I kind of making the current business climate affecting our operating models? Um, like we said, there's increased ask for Finn ops is an increase. Ask for security ops because security threats have only amplified. And then the entire cloud ups model. I think hybrid cloud operations its's prompted us to rethink a lot off. How do we do? I t operations and and we're investing a lot in terms of automation and then underpinning that by ai led operation. So, um, you talked about the client management platform making sure we've got the best automation and processes which are repeatable around all the way from just doing provisioning to data operations to optimization. Just making all of that robust and repeatable um, is such a value. Add to clients because then they can see SOS can sleep at night knowing that everything is taken care off and, uh, the CIA, the CEOs can be rest assured that hey, they're not going to get that AWS bill that's going to make them hit the roof. So making sure we've got the right checks and balances and approval flow is all a part of our child management platform. And at that point, I know you really passionate AI and the role that it plays in operations and the entire cloud management platform and cloud for day platform So your thoughts in the poem? >>Yes, sir. No, thank you. But so yeah, yeah, I led operations is really part off the bigger question and the pain point that customers are faced with, which is I've reached the cloud. Now, how do I optimized and get benefits from the cloud on the benefits is around. You know, uh, utility for on demand access to resource is, uh, this cost optimization potential and the security, uh, cloud security potential that, if not managed properly, can really blow up in the face. And unfortunately, you know that in the case on the AI ops led Operation Side, that's again a huge foretell area where Unisys Investor is investing a lot off a lot off i p and creating a lot of differentiation. And the objective there is to ask Customers adopt cloud for day as they adopt Unisys Cloud services. They're able to take advantage off cost optimization capabilities, which essentially looks at historical usage on predicts future usage, based on a number off a I artificial intelligence and machine learning technologies that that is able to give you predictions that otherwise very hard to hard to get and, uh, in the cloud environment because of the sheer velocity volume and variety of the data. Doing that in a manual fashion is very, very hard. So automated machine learning driven approach is very productive is very effective on, you know, some of the outcomes that we've achieved is is just amazing. We've been able to save up to 25% off infrastructure costs through the island operations. About 40% off infrastructure incidents have bean reduced due to root cause analysis. Eso onda up to 35% off meantime, to resolution improvements in time. So huge customer benefits driven by e I led operations. The I am a approaches to following the problem. >>Let me see him If I could stick with you for a second big numbers that you just talked about and we talked a few minutes ago about outcomes. It's all about outcomes right now with this rush to cloud as as a new set. And we talked about this on the Cuba all the time. We've seen that the last eight months there is an acceleration of this digital transformation. I'm just curious una come from your perspective as the VP and CTO cloud how are you? What are some of the things that you advise customers to do if they need to rush to the cloud 21 just, you know, move their business quickly and not have the stay on life support. What are some of the things that you advise them to do when they're in this? Maybe a few months ago, when they were in the beginning of this? >>Yeah, that's that's a very interesting question, and lot off our clients are faced with that question as they either they're already in the cloud or the deciding to migrate to the cloud on the whole journey. Customer journeys for either stepping on the cloud or managing and operating the optimizing the cloud deployments is very key. So if you look at the market research that's out there and what we hear from our customers, the key challenges are really, really around. How do I migrate to the cloud without facing a lot of bottlenecks and challenges, and how do I overcome them? So that's the keeping pain point and again cloud for the advisory services and the cloud services that we offer allows customers to take up uh, toe work with us, and we work with the customer to ensure that they're able to do that on and then rapidly migrating to the cloud, managing and operating their operations. The hybrid cloud operations in optimized fashion is a huge challenge. How do they migrate? How do they migrate with security and compliance not being compromised once they're in the cloud, ensuring cloud security is and compliance is is maintained. Ensuring that the cost structure is is optimized so that they're not being mawr wants to move to the cloud compared to on premises and and then taking advantage of the whole cloud. Deployments to ensure you're looking at data are nothing the data to derive meaningful business outcomes. So if the entire end to end customer journey that needs to be looked at optimized. And that's where Unisys comes in with a cloud for the platform where we work with the customers to enhance the journeys. And in this case I want to mention CSU, which is, uh, the California State University, where the approach Unisys to really work with them to deliver uh, cloud services by enhancing the the objective was to enhance the student learning experience to enable adoption off off the technology by the students but also to achieve better performance, better adoption cost savings on we were able to deliver about 30% better performance help realize about 30 33% savings on 40% plus growth in adoption. On this was for about half a million student bodies. The 50,000 plus faculty staff spread across 23 campuses. So deploying, optimizing on and managing the infrastructure is something that Unisys does. Does that. And this is an example of that. I know you want to add anything to that. >>Absolutely Any Permanente's really well and, >>uh, >>it Z also securing, making sure securities with the >>journey >>it Z O Keefe or hybrid cloud. Um, uh, at least I'm sure you're aware of the Unisys tagline is securing tomorrow. So who better s so we really, really take that really, really seriously in terms of making sure we seek clients cloud journeys, and >>you >>probably heard the statistic from her. About 80% off cloud breaches are due to mis configuration, and this could have bean prevented. And and it doesn't. There's an element of the human angle in there. You believe strongly that can automate using our platform. So we've got 2000 plus security policies, which makes sure which again enables our clients to be compliant as well. So no matter what compliant standards, we've got several off our clients, for example, in the financial sector that are hosted on AWS and that we managed and they have to, especially the US They have to comply with Y de f s, the New York Department of Financial Services and making sure that they compliant with all the standards out there, which is next plus plus in this case. So that's part of what we do and enabling those journeys and then just keeping up with the rate of change like on different was talking about the variety and velocity of the data and and the rate of change of the applications out there, especially as businesses react to the pandemic and have to cope with the changing business paradigms out there. They have to be quick. Um, so we've got a drugmaker, one of the most premium drug makers in the US, who is who is against it on AWS, and, uh, they're racing for the cure and they are always looking at How do they get drugs quicker to the market? And that means accelerating applications. And we know that based on research by the Dora study, that if you adopt develops paradigms, you can accelerate 200 times faster than if you didn't. But then you have to underpin backward security as well. So really helping this adopt deaths are cops in all their deployments to AWS so that they can really race for the cure. That's the kind of business outcomes that we really, uh, are really, really proud to drive for our clients. >>Excellent on a pound. Let's wrap this up with you. We've just got about 30 seconds left sticking on the security front. It's such a huge topic right now. It has been for a long time, but even more so during these unprecedented times when you're talking with customers, what makes Unisys unique from a security perspective? >>So first thing is to understand what it takes to solve the hybrid cloud security problem. Like you said earlier, that's the biggest pain point that we hear from customers from our clients on. It's all over the market research all the breaches that have happened, like the zoom breach that happened that compromised about half a million, you know, user log ins. And then there was also the the Marriott breach, where about half a billion users names and credential for legal legal. So it Zaveri easy for customers, potential customers to become like a headline. And our our job really are the companies to make sure that they're not the next capital one or the next Marriott, uh, showing up in the newspaper. So we kind of look at their customer deployments situation on. We put together a comprehensive into an hybrid cloud solution, hybrid cloud security and compliance solution that includes look, securing their cloud infrastructure, their cloud workloads in terms of applications that they might have secured, and also to look at securing their applications, which may or may not be running on the cloud. So we kind of take a very holistic approach, using our homegrown solutions and partner solutions to create a comprehensive, robust hybrid cloud solution that really fits the customer context and and so we we are essentially a trusted adviser for our for our clients to create the solution, which again, at the cloud 40 ashore, which is a cloud security posture management solution. We have a cloud worker protection solution on then stealth, which is a full stack security solution if combined together with the other cloud Forte platform components on. We wrap this up in a matter of security services offering that allows US customers to have complete peace of mind as we take care off assessment remediation monitoring on, then continues Posture, posture, management. I know. Do you want to add anything to that? >>If I'm think in terms of closing, I think like you covered it well, we've got platform competence and services that run the gamut off the off the life cycle from migrations to two transformations. And one thing that I think in terms of outcomes of these, uh, when the service built around it have really helped us. Dr is, um is kind of responding especially to our public sector clients, very passionate about enabling cloud journeys for our public sector clients. And we'll take the example of Georgia Technology s So this is the G t A. Is the technology agency for all services are 14 of the agencies in Georgia and many of these public sector agencies had to quickly adopt cloud to deal with the report workers. Whether it was v D I whether it was chatbots on cloud, um, it was it was, ah, brand new world out there, the new normal. And it was just using the cloud management platform that anyone was refering to. We were able to kind of take them from taking three months. Plus to be able to provision workloads Thio thio less than 30 minutes to provision workloads. And this is this is across hybrid cloud. So and this is >>a big outcome, especially in this time where things were changing so quickly. Well, I wish we had more time, guys because I could tell you have a lot more that you can share. You're just gonna have to come back. And I like that. The tagline securing tomorrow. Adding on to what Anu Pump said So your customers don't become the next headline. I think they would all appreciate that. Thank you both. So much for joining me on the Cube today and sharing what's the latest with Unisys. We appreciate your time. Thank you. Thank >>you for having us >>aren't my pleasure for my guests. I'm Lisa Martin. And you're watching the Cube? Yeah,

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi, Lisa Martin here covering the Broadcom dev ops virtual forum. I'm very pleased to be joined today by a cube alumni, Jeffrey Hammond, the vice president and principal analyst serving CIO is at Forester. Jeffrey. Nice to talk with you today. >>Good morning. It's good to be here. Yeah. >>So a virtual forum, great opportunity to engage with our audiences so much has changed in the last it's an understatement, right? Or it's an overstated thing, but it's an obvious, so much has changed when we think of dev ops. One of the things that we think of is speed, you know, enabling organizations to be able to better serve customers or adapt to changing markets like we're in now, speaking of the need to adapt, talk to us about what you're seeing with respect to dev ops and agile in the age of COVID, what are things looking like? >>Yeah, I think that, um, for most organizations, we're in a, uh, a period of adjustment, uh, when we initially started, it was essentially a sprint, you know, you run as hard as you can for as fast as you can for as long as you can and you just kind of power through it. And, and that's actually what, um, the folks that get hub saw in may when they ran an analysis of how developers, uh, commit times and a level of work that they were committing and how they were working, uh, in the first couple of months of COVID was, was progressing. They found that developers, at least in the Pacific time zone were actually increasing their work volume, maybe because they didn't have two hour commutes or maybe because they were stuck away in their homes, but for whatever reason, they were doing more work. >>And it's almost like, you know, if you've ever run a marathon the first mile or two in the marathon, you feel great and you just want to run and you want to power through it and you want to go hard. And if you do that by the time you get to mile 18 or 19, you're going to be gassed. It's sucking for wind. Uh, and, and that's, I think where we're starting to hit. So as we start to, um, gear our development chops out for the reality that most of us won't be returning into an office until 2021 at the earliest and many organizations will, will be fundamentally changing, uh, their remote workforce, uh, policies. We have to make sure that the agile processes that we use and the dev ops processes and tools that we use to support these teams are essentially aligned to help developers run that marathon instead of just kind of power through. >>So, um, let me give you a couple of specifics for many organizations, they have been in an environment where they will, um, tolerate Rover remote work and what I would call remote work around the edges like developers can be remote, but product managers and, um, you know, essentially scrum masters and all the administrators that are running the, uh, uh, the SCM repositories and, and the dev ops pipelines are all in the office. And it's essentially centralized work. That's not, we are anymore. We're moving from remote workers at the edge to remote workers at the center of what we do. And so one of the implications of that is that, um, we have to think about all the activities that you need to do from a dev ops perspective or from an agile perspective, they have to be remote people. One of the things I found with some of the organizations I talked to early on was there were things that administrators had to do that required them to go into the office to reboot the SCM server as an example, or to make sure that the final approvals for production, uh, were made. >>And so the code could be moved into the production environment. And so it actually was a little bit difficult because they had to get specific approval from the HR organizations to actually be allowed to go into the office in some States. And so one of the, the results of that is that while we've traditionally said, you know, tools are important, but they're not as important as culture as structure as organization as process. I think we have to rethink that a little bit because to the extent that tools enable us to be more digitally organized and to hiring, you know, achieve higher levels of digitization in our processes and be able to support the idea of remote workers in the center. They're now on an equal footing with so many of the other levers, uh, that, that, um, uh, that organizations have at their disposal. Um, I'll give you another example for years. >>We've said that the key to success with agile at the team level is cross-functional co located teams that are working together physically co located. It's the easiest way to show agile success. We can't do that anymore. We can't be physically located at least for the foreseeable future. So, you know, how do you take the low hanging fruits of an agile transformation and apply it in, in, in, in the time of COVID? Well, I think what you have to do is that you have to look at what physical co-location has enabled in the past and understand that it's not so much the fact that we're together looking at each other across the table. It's the fact that we're able to get into a shared mindspace, uh, from, um, uh, from a measurement perspective, we can have shared purpose. We can engage in high bandwidth communications. It's the spiritual aspect of that physical co-location that is actually important. So one of the biggest things that organizations need to start to ask themselves is how do we achieve spiritual colocation with our agile teams? Because we don't have the, the ease of physical co-location available to us anymore? >>Well, the spiritual co-location is such an interesting kind of provocative phrase there, but something that probably was a challenge here, we are seven, eight months in for many organizations, as you say, going from, you know, physical workspaces, co-location being able to collaborate face to face to a, a light switch flip overnight. And this undefined period of time where all we were living with with was uncertainty, how does spiritual, what do you, when you talk about spiritual co-location in terms of collaboration and processes and technology help us unpack that, and how are you seeing organizations adopted? >>Yeah, it's, it's, um, it's a great question. And, and I think it goes to the very root of how organizations are trying to transform themselves to be more agile and to embrace dev ops. Um, if you go all the way back to the, to the original, uh, agile manifesto, you know, there were four principles that were espoused individuals and interactions over processes and tools. That's still important. Individuals and interactions are at the core of software development, processes and tools that support those individual and interact. Uh, those individuals in those interactions are more important than ever working software over comprehensive documentation. Working software is still more important, but when you are trying to onboard employees and they can't come into the office and they can't do the two day training session and kind of understand how things work and they can't just holler over the cube, uh, to ask a question, you may need to invest a little bit more in documentation to help that onboarding process be successful in a remote context, uh, customer collaboration over contract negotiation. >>Absolutely still important, but employee collaboration is equally as important if you want to be spiritually, spiritually co-located. And if you want to have a shared purpose and then, um, responding to change over following a plan. I think one of the things that's happened in a lot of organizations is we have focused so much of our dev ops effort around velocity getting faster. We need to run as fast as we can like that sprinter. Okay. You know, trying to just power through it as quickly as possible. But as we shift to, to the, to the marathon way of thinking, um, velocity is still important, but agility becomes even more important. So when you have to create an application in three weeks to do track and trace for your employees, agility is more important. Um, and then just flat out velocity. Um, and so changing some of the ways that we think about dev ops practices, um, is, is important to make sure that that agility is there for one thing, you have to defer decisions as far down the chain to the team level as possible. >>So those teams have to be empowered to make decisions because you can't have a program level meeting of six or seven teams and one large hall and say, here's the lay of the land. Here's what we're going to do here are our processes. And here are our guardrails. Those teams have to make decisions much more quickly that developers are actually developing code in smaller chunks of flow. They have to be able to take two hours here or 50 minutes there and do something useful. And so the tools that support us have to become tolerant of the reality of, of, of, of how we're working. So if they work in a way that it allows the team together to take as much autonomy as they can handle, um, to, uh, allow them to communicate in a way that, that, that delivers shared purpose and allows them to adapt and master new technologies, then they're in the zone in their spiritual, they'll get spiritually connected. I hope that makes sense. >>It does. I think we all could use some of that, but, you know, you talked about in the beginning and I've, I've talked to numerous companies during the pandemic on the cube about the productivity, or rather the number of hours of work has gone way up for many roles, you know, and, and, and times that they normally late at night on the weekends. So, but it's a cultural, it's a mind shift to your point about dev ops focused on velocity, sprints, sprints, sprints, and now we have to, so that cultural shift is not an easy one for developers. And even at this folks to flip so quickly, what have you seen in terms of the velocity at which businesses are able to get more of that balance between the velocity, the sprint and the agility? >>I think, I think at the core, this really comes down to management sensitivity. Um, when everybody was in the office, you could kind of see the mental health of development teams by, by watching how they work. You know, you call it management by walking around, right. We can't do that. Managers have to, um, to, to be more aware of what their teams are doing, because they're not going to see that, that developer doing a check-in at 9:00 PM on a Friday, uh, because that's what they had to do, uh, to meet the objectives. And, um, and, and they're going to have to, to, um, to find new ways to measure engagement and also potential burnout. Um, friend of mine once had, uh, had a great metric that he called the parking lot metric. It was helpful as the parking lot at nine. And how full was it at five? >>And that gives you an indication of how engaged your developers are. Um, what's the digital equivalent equivalent to the parking lot metric in the time of COVID it's commit stats, it's commit rates. It's, um, you know, the, uh, the turn rate, uh, that we have in our code. So we have this information, we may not be collecting it, but then the next question becomes, how do we use that information? Do we use that information to say, well, this team isn't delivering as at the same level of productivity as another team, do we weaponize that data or do we use that data to identify impedances in the process? Um, why isn't a team working effectively? Is it because they have higher levels of family obligations and they've got kids that, that are at home? Um, is it because they're working with, um, you know, hardware technology, and guess what, they, it's not easy to get the hardware technology into their home office because it's in the lab at the, uh, at the corporate office, uh, or they're trying to communicate, uh, you know, halfway around the world. >>And, uh, they're communicating with a, with an office lab that is also shut down and, and, and the bandwidth just doesn't enable the, the level of high bandwidth communications. So from a dev ops perspective, managers have to get much more sensitive to the, the exhaust that the dev ops tools are throwing off, but also how they're going to use that in a constructive way to, to prevent burnout. And then they also need to, if they're not already managing or monitoring or measuring the level of developer engagement, they have, they really need to start whether that's surveys around developer satisfaction, um, whether it's, you know, more regular social events, uh, where developers can kind of just get together and drink a beer and talk about what's going on in the project, uh, and monitoring who checks in and who doesn't, uh, they have to, to, um, work harder, I think, than they ever have before. >>Well, and you mentioned burnout, and that's something that I think we've all faced in this time at varying levels and it changes. And it's a real, there's a tension in the air, regardless of where you are. There's a challenge, as you mentioned, people having, you know, coworker, their kids as coworkers and fighting for bandwidth, because everyone is forced in this situation. I'd love to get your perspective on some businesses that are, that have done this well, this adaptation, what can you share in terms of some real-world examples that might inspire the audience? >>Yeah. Uh, I'll start with, uh, stack overflow. Uh, they recently published a piece in the journal of the ACM around some of the things that they had discovered. Um, you know, first of all, just a cultural philosophy. If one person is remote, everybody is remote. And you just think that way from an executive level, um, social spaces. One of the things that they talk about doing is leaving a video conference room open at a team level all day long, and the team members, you know, we'll go on mute, you know, so that they don't have to, that they don't necessarily have to be there with somebody else listening to them. But if they have a question, they can just pop off mute really quickly and ask the question. And if anybody else knows the answer, it's kind of like being in that virtual pod. Uh, if you, uh, if you will, um, even here at Forrester, one of the things that we've done is we've invested in social ceremonies. >>We've actually moved our to our team meetings on, on my analyst team from, from once every two weeks to weekly. And we have built more time in for social Ajay socialization, just so we can see, uh, how, how, how we're doing. Um, I think Microsoft has really made some good, uh, information available in how they've managed things like the onboarding process. I think I'm Amanda silver over there mentioned that a couple of weeks ago when, uh, uh, a presentation they did that, uh, uh, Microsoft onboarded over 150,000 people since the start of COVID, if you don't have good remote onboarding processes, that's going to be a disaster. Now they're not all developers, but if you think about it, um, everything from how you do the interviewing process, uh, to how you get people, their badges, to how they get their equipment. Um, security is a, is another issue that they called out typically, uh, it security, um, the security of, of developers machines ends at, at, at the corporate desktop. >>But, you know, since we're increasingly using our own machines, our own hardware, um, security organizations kind of have to extend their security policies to cover, uh, employee devices, and that's caused them to scramble a little bit. Uh, so, so the examples are out there. It's not a lot of, like, we have to do everything completely differently, but it's a lot of subtle changes that, that have to be made. Um, I'll give you another example. Um, one of the things that, that we are seeing is that, um, more and more organizations to deal with the challenges around agility, with respect to delivering software, embracing low-code tools. In fact, uh, we see about 50% of firms are using low-code tools right now. We predict it's going to be 75% by the end of next year. So figuring out how your dev ops processes support an organization that might be using Mendix or OutSystems, or, you know, the power platform building the front end of an application, like a track and trace application really, really quickly, but then hooking it up to your backend infrastructure. Does that happen completely outside the dev ops investments that you're making and the agile processes that you're making, or do you adapt your organization? Um, our hybrid teams now teams that not just have professional developers, but also have business users that are doing some development with a low-code tool. Those are the kinds of things that we have to be, um, willing to, um, to entertain in order to shift the focus a little bit more toward the agility side, I think >>Lot of obstacles, but also a lot of opportunities for businesses to really learn, pay attention here, pivot and grow, and hopefully some good opportunities for the developers and the business folks to just get better at what they're doing and learning to embrace spiritual co-location Jeffrey, thank you so much for joining us on the program today. Very insightful conversation. >>My pleasure. It's it's, it's an important thing. Just remember if you're going to run that marathon, break it into 26, 10 minute runs, take a walk break in between each and you'll find that you'll get there. >>Digestible components, wise advice. Jeffery Hammond. Thank you so much for joining for Jeffrey I'm Lisa Martin, you're watching Broadcom's dev ops virtual forum >>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom, >>Continuing our conversations here at Broadcom's dev ops virtual forum. Lisa Martin here, please. To welcome back to the program, Serge Lucio, the general manager of the enterprise software division at Broadcom. Hey, Serge. Welcome. Thank you. Good to be here. So I know you were just, uh, participating with the biz ops manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept, but I wanted to get your thoughts on spiritual co-location as really a necessity for biz ops to succeed in this unusual time in which we're living. What are your thoughts on spiritual colocation in terms of cultural change versus adoption of technologies? >>Yeah, it's a, it's, it's quite interesting, right? When we, when we think about the major impediments for, uh, for dev ops implementation, it's all about culture, right? And swore over the last 20 years, we've been talking about silos. We'd be talking about the paradox for these teams to when it went to align in many ways, it's not so much about these teams aligning, but about being in the same car in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me, the, this, this is really about kind of changing this culture where people start to look at a kind of OKR is instead of the key objective, um, that, that drives the entire team. Now, what it means in practice is really that's, uh, we need to change a lot of behaviors, right? It's not about the Yarki, it's not about roles. It's about, you know, who can do what and when, and, uh, you know, driving a bias towards action. It also means that we need, I mean, especially in this school times, it becomes very difficult, right? To drive kind of a kind of collaboration between these teams. And so I think there there's a significant role that especially tools can play in terms of providing this complex feedback from teams to, uh, to be in that preface spiritual qualification. >>Well, and it talked about culture being, it's something that, you know, we're so used to talking about dev ops with respect to velocity, all about speed here. But of course this time everything changed so quickly, but going from the physical spaces to everybody being remote really does take it. It's very different than you can't replicate it digitally, but there are collaboration tools that can kind of really be essential to help that cultural shift. Right? >>Yeah. So 2020, we, we touch to talk about collaboration in a very mundane way. Like, of course we can use zoom. We can all get into, into the same room. But the point when I think when Jeff says spiritual, co-location, it's really about, we all share the same objective. Do we, do we have a niece who, for instance, our pipeline, right? When you talk about dev ops, probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team, but just thinking about a pipeline, right, at the end of the day, it's all about what is the meantime to beat back to these teams. If I'm a developer and a commit code, I don't, does it take where, you know, that code to be processed through pipeline pushy? Can I get feedback if I am a finance person who is funding a product or a project, what is my meantime to beat back? >>And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of the Dora metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration, it's really a balance. How do you provide the feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this, this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when I checked in code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allegation, how can the system help me reduce my change pillar rates? Because it's, it's able to predict that some issue was introduced in the outpatient or work product. Um, so I think there's, there's a great role of technology and AI candidate Lynch to, to actually provide that new level of collaboration. >>So we'll get to AI in a second, but I'm curious, what are some of the, of the metrics you think that really matter right now is organizations are still in some form of transformation to this new almost 100% remote workforce. >>So I'll just say first, I'm not a big fan of metrics. Um, and the reason being that, you know, you can look at a change killer rate, right, or a lead time or cycle time. And those are, those are interesting metrics, right? The trend on metric is absolutely critical, but what's more important is you get to the root cause what is taught to you lean to that metric to degrade or improve or time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll give you a very mundane example. You know, we know that cycle time is heavily influenced by, um, organizational boundaries. So, you know, we talk a lot about silos, but, uh, we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are a pretty, you know, valid set metrics, but what's way more important is to understand what are the antiperspirants, what are the things that we can detect through the data that actually are affecting those metrics. And, uh, I mean, over the last 10, 20 years, we've learned a lot about kind of what are, what are the antiperspirants within our large enterprise customers. And there are plenty of them. >>What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >>So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle, where DevOps is more and more embracing value stream management value stream mapping. Um, and I think what's interesting is that in many ways the product is becoming the new silo. Uh, the notion of a product is very difficult by itself to actually define people are starting to recognize that a value stream is not its own little kind of Island. That in reality, when I define a product, this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So, so even on that, and there is clearly kind of a new sets, if you will, of anti-patterns where products are being defined as a set of OTRs, they have interdependencies and you have have a new set of silos on the operands, uh, the Abra key movement to Israel and the SRE space where, um, I think there is a cultural clash while the dev ops side is very much embracing this notion of OTRs and value stream mapping and Belgium management. >>On the other end, you have the it operations teams. We still think business services, right? For them, they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so the, the, there is there's this boundary where, um, I think, well, SRE is being put in place. And there's lots of thinking about what kind of metrics can be fined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for true operations team. >>And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You talked about, uh, AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate, maybe see eye to eye with the SRS? >>Yeah. Um, so th you're kind of too. So even for myself, as a leader of a, you know, 1500 people organization, there's a number of things I don't see right. On a daily basis. And, um, I think the, the, the, the technologies that we have at our disposal today from the AI are able to mind a lot of data and expose a lot of, uh, issues that's as leaders we may not be aware of. And some of the, some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you, when you start to understand, for instance, uh, what is the, what is the working progress right to during the sprint? Um, when you start to analyze the data you can detect, for instance, that maybe the teams are over committed, that there is too much work in progress. >>You can start to identify kind of, interdepencies either from a technology, from a people point of view, which were hidden, uh, you can start to understand maybe the change filler rates he's he is dragging. So I believe that there is a, there's a fundamental role to be played by the tools to, to expose again, these anti parents, to, to make these things visible to the teams, to be able to even compare teams. Right. One of the things that's, that's, uh, that's amazing is now we have access to tons of data, not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate, and what's working, what's not working >>Thoughts on AI and automation as, as a facilitator of spiritual co-location. >>Yeah, absolutely. Absolutely. It's um, you know, th there's, uh, the problem we all face is the unknown, right? The, the law city, but volume variety of the data, uh, everyday we don't really necessarily completely appreciate what is the impact of our actions, right? And so, um, AI can really act as a safety net that enables us to, to understand what is the impact of our actions. Um, and so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data, um, and collaborate on the data. And the actual matter, I think is, is a, is a very powerful enabler, uh, on, in that respect. I mean, I, I've seen, um, I've seen countless of times that, uh, for instance, at the SRE boundary, um, to basically show that we'll turn the quality attributes, so an incoming release, right. And exposing that to, uh, an operations person and a sorry person, and enabling that collaboration dialogue through data is a very, very powerful tool. >>Do you have any recommendations for how teams can use, you know, the SRE folks, the dev ops says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be nonproductive. >>Yeah. So to me, the th there, there's a part of the question really is when, when we talk about data, there are there different ways you can use data, right? Um, so you can, you can do a lot of an analytics, predictive analytics. So I think there is a, there's a tendency, uh, to look at, let's say a, um, a specific KPI, like a, an availability KPI, or change filler rate, and to basically do a regression analysis and projecting all these things, going to happen in the future. To me, that that's, that's a, that's a bad approach. The reason why I fundamentally think it's a better approach is because we are systems. The way we develop software is, is a, is a non-leader kind of system, right? Software development is not linear nature. And so I think there's a D this is probably the worst approach is to actually focus on metrics on the other end. >>Um, if you, if you start to actually understand at a more granular level, what har, uh, which are the things which are contributing to this, right? So if you start to understand, for instance, that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we, we have, I've actually, uh, a customer who, uh, identified that, uh, over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed, this resulted in these plant outages. So if you start to be able to basically establish causality, right, cause an effect between kind of data across the last cycle. I think, I think this is the right way to, uh, to, to use AI. And so pharma to be, I think it's way more God could have a classification problem. What are the classes of problems that do exist and affect things as opposed to analytics, predictive, which I don't think is as powerful. >>So I mentioned in the beginning of our conversation, that just came off the biz ops manifesto. You're one of the authors of that. I want to get your thoughts on dev ops and biz ops overlapping, complimenting each other, what, from a, the biz ops perspective, what does it mean to the future of dev ops? >>Yeah, so, so it's interesting, right? If you think about DevOps, um, there's no felony document, right? Can we, we can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways, there's no clear definition of what dev ops is. Uh, if you go to the dev ops Institute today, you'll see that they are specific, um, trainings for instance, on value management on SRE. And so in many ways, the problem we have as an industry is that, um, there are set practices between agile dev ops, SRE Valley should management. I told, right. And we all basically talk about the same things, right. We all talk about essentially, um, accelerating in the meantime fee to feedback, but yet we don't have the common framework to talk about that. The other key thing is that we add to wait, uh, for, uh, for jeans, Jean Kim's Lascaux, um, to, uh, to really start to get into the business aspect, right? >>And for value stream mapping to start to emerge for us to start as an industry, right. It, to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about driving these business outcomes. And so to me, these ops is really about kind of, uh, putting a lens on this critical element that it's not business and it, that we in fact need to fuse business 19 that I need needs to transform itself to recognize that it's, it's this value generator, right. It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this Oliver or kind of framework, if you will. That set the context for what is the reason, uh, for it to exist. What's part of the core values and principles that it needs to embrace to, again, change from a cost center to a value center. And then we need to start to use this as a way to start to unify some of the, again, the core practices, whether it's agile, DevOps value, stream mapping SRE. Um, so, so I think over time, my hope is that we start to optimize a lot of our practices, language, um, and, uh, and cultural elements. >>Last question surgeon, the last few seconds we have here talking about this, the relation between biz ops and dev ops, um, what do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >>So to me, the key, the key, um, challenge for, for the industry is really around. So we were seeing a very rapid shift towards kind of, uh, product to product, right. Which we don't want to do is to recreate kind of these new silos, these hard silos. Um, so that, that's one of the big changes, uh, that I think we need to be, uh, to be really careful about, um, because it is ultimately, it is about culture. It's not about, uh, it's not about, um, kind of how we segment the work, right. And, uh, any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of, um, kind of the spiritual co-location, I think it's, it's really about that too. It's really about kind of, uh, uh, focusing on the business outcomes on kind of aligning on driving engagement across the teams, but, but not for create a, kind of a new set of silos, which instead of being vertical are going to be these horizontal products >>Crazy by surge that looking at culture as kind of a way of really, uh, uh, addressing and helping to, uh, re re reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps virtual forum. >>Thank you. Thanks for your time. >>I'll be right back >>From around the globe it's the cube with digital coverage of devops virtual forum brought to you by Broadcom. >>Welcome to Broadcom's DevOps virtual forum, I'm Lisa Martin, and I'm joined by another Martin, very socially distanced from me all the way coming from Birmingham, England is Glynn Martin, the head of QA transformation at BT. Glynn, it's great to have you on the program. Thank you, Lisa. I'm looking forward to it. As we said before, we went live to Martins for the person one in one segment. So this is going to be an interesting segment guys, what we're going to do is Glynn's going to give us a really kind of deep inside out view of devops from an evolution perspective. So Glynn, let's start. Transformation is at the heart of what you do. It's obviously been a very transformative year. How have the events of this year affected the >> transformation that you are still responsible for driving? Yeah. Thank you, Lisa. I mean, yeah, it has been a difficult year. >>Um, and although working for BT, which is a global telecommunications company, um, I'm relatively resilient, I suppose, as a, an industry, um, through COVID obviously still has been affected and has got its challenges. And if anything, it's actually caused us to accelerate our transformation journey. Um, you know, we had to do some great things during this time around, um, you know, in the UK for our emergency and, um, health workers give them unlimited data and for vulnerable people to support them. And that's spent that we've had to deliver changes quickly. Um, but what we want to be able to do is deliver those kinds of changes quickly, but sustainably for everything that we do, not just because there's an emergency. Um, so we were already on the kind of journey to agile, but ever more important now that we are, we are able to do those, that kind of work, do it more quickly. >>Um, and that it works because the, the implications of it not working is, can be terrible in terms of you know, we've been supporting testing centers,  new hospitals to treat COVID patients. So we need to get it right. And then therefore the coverage of what we do, the quality of what we do and how quickly we do it really has taken on a new scale and what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that, you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously, um, deal with the fact that, you know, COVID 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less costs, but having to deliver more value quicker and  to higher quality. So yeah, certainly the finances is, um, on our minds and that's why we need flexible models, cost models that allow us to kind of do growth, but we get that growth by showing that we're delivering value. Um, especially in these times when there are financial challenges on companies. So one of the things that I want to ask you about, I'm again, looking at DevOps from the inside >>Out and the evolution that you've seen, you talked about the speed of things really accelerating in this last nine months or so. When we think dev ops, we think speed. But one of the things I'd love to get your perspective on is we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that you've seen there as, as needing to get, as you said, get things right, but done so quickly to support essential businesses, essential workers. How have you seen that cultural shift? >>Yeah, I think, you know, before test teams for themselves at this part of the software delivery cycle, um, and actually now really our customers are expecting that quality and to deliver for our customers what they want, quality has to be ingrained throughout the life cycle. Obviously, you know, there's lots of buzzwords like shift left. Um, how do we do shift left testing? Um, but for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle that drive automation, drive improvements. I always say that, you know, you're only as good as your lowest common denominator. And one thing that we were finding on our dev ops journey was that we  would be trying to do certain things quick, we had automated build, automated tests. But if we were taking a weeks to create test scripts, or we were taking weeks to manually craft data, and even then when we had taken so long to do it, that the coverage was quite poor and that led to lots of defects later on in the life cycle, or even in our production environment, we just couldn't afford to do that. >>And actually, focusing on continuous testing over the last nine to 12 months has really given us the ability to deliver quickly across the whole life cycle. And therefore actually go from doing a kind of semi agile kind of thing, where we did the user stories, we did a few of the kind of agile ceremonies, but we weren't really deploying any quicker into production because our stakeholders were scared that we didn't have the same control that we had when we had more waterfall releases. And, you know, when we didn't think of ourselves. So we've done a lot of work on every aspect, um, especially from a testing point of view, every aspect of every activity, rather than just looking at automated tests, you know, whether it is actually creating the test in the first place, whether it's doing security testing earlier in the lot and performance testing in the life cycle, et cetera. So, yeah,  it's been a real key thing that for CT, for us to drive DevOps, >>Talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this. Um, you know, there's a thing that I think people will probably call it a customer experience gap, and it reminds me of a Gilbert cartoon, where we start with the requirements here and you're almost like a Chinese whisper effects and what we deliver is completely different. So we think the testing team or the delivery teams, um, know in our teeth has done a great job. This is what it said in the acceptance criteria, but then our customers are saying, well, actually that's not working this isn't working and there's this kind of gap. Um, we had a great launch this year of agile requirements, it's one of the Broadcom tools. And that was the first time in, ever since I remember actually working within BT, I had customers saying to me, wow, you know, we want more of this. >>We want more projects to have extra requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do we actually, you know, do that and have something that both the business and technical people can understand. And we've actually been working with the business , using agile requirements designer to really look at what the requirements are, tease out requirements we hadn't even thought of and making sure that we've got high levels of test coverage. And what we actually deliver at the end of it, not only have we been able to generate tests more quickly, but we've got much higher test coverage and also can more smartly, using the kind of AI within the tool and then some of the other kinds of pipeline tools, actually deliver to choose the right tasks, and actually doing a risk based testing approach. So that's been a great launch this year, but just the start of many kinds of things that we're doing >>Well, what I hear in that, Glynn is a lot of positives that have come out of a very challenging situation. Talk to me about it. And I liked that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration perspective you're right, we talk about that a lot critical with devops. But those challenges there, you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pivot so fast? >>I mean, you talked about culture. You know, BT is like most companies  So it's very siloed. You know we're still trying to work to become closer as a company. So I think there's a lot of challenges around how would you integrate with other tools? How would you integrate with the various different technologies. And BT, we have 58 different IT stacks. That's not systems, that's stacks, all of those stacks can have hundreds of systems. And we're trying to, we've got a drive at the moment, a simplified program where we're trying to you know, reduce that number to 14 stacks. And even then there'll be complexity behind the scenes that we will be challenged more and more as we go forward. How do we actually highlight that to our users? And as an it organization, how do we make ourselves leaner, so that even when we've still got some of that legacy, and we'll never fully get rid of it and that's the kind of trade off that we have to make, how do we actually deal with that and hide that from our users and drive those programs, so we can, as I say, accelerate change,  reduce that kind of waste and that kind of legacy costs out of our business. You know, the other thing as well, I'm sure telecoms is probably no different to insurance or finance. When you take the number of products that we do, and then you combine them, the permutations are tens and hundreds of thousands of products. So we, as a business are trying to simplify, we are trying to do that in an agile way. >>And haven't tried to do agile in the proper way and really actually work at pace, really deliver value. So I think what we're looking more and more at the moment is actually  more value focused. Before we used to deliver changes sometimes into production. Someone had a great idea, or it was a great idea nine months ago or 12 months ago, but actually then we ended up deploying it and then we'd look at the users, the usage of that product or that application or whatever it is, and it's not being used for six months. So we haven't got, you know, the cost of the last 12 months. We certainly haven't gotten room for that kind of waste and, you know, for not really understanding the value of changes that we are doing. So I think that's the most important thing of the moment, it's really taking that waste out. You know, there's lots of focus on things like flow management, what bits of our process are actually taking too long. And we've started on that journey, but we've got a hell of a long way to go. But that involves looking at every aspect of the software delivery cycle. >> Going from, what 58 IT stacks down to 14 or whatever it's going to be, simplifying sounds magical to everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we were started on a continuous testing journey, and I think that's just the start. I mean as I say, looking at every aspect of, you know, from a QA point of view is every aspect of what we do. And it's also looking at, you know, we've started to branch into more like AI, uh, AI ops and, you know, really the full life cycle. Um, and you know, that's just a stepping stone to, you know, I think autonomics is the way forward, right. You know, all of this kind of stuff that happens, um, you know, monitoring, uh, you know, watching the systems what's happening in production, how do we feed that back? How'd you get to a point where actually we think about change and then suddenly it's in production safely, or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey, but if we want to, you know, in a world where the pace is in ever-increasing and the demands for the team, and, you know, with the pressures on, at the moment where we're being asked to do things, uh, you know, more efficiently and as lean as possible, we need to be thinking about every part of the process and how we put the kind of stepping stones in place to lead us to a more automated kind of, um, you know, um, the future. >>Do you feel that that planned outcomes are starting to align with what's delivered, given this massive shift that you're experiencing? >>I think it's starting to, and I think, you know, as I say, as we look at more of a value based approach, um, and, um, you know, as I say, print, this was a kind of flow management. I think that that will become ever, uh, ever more important. So, um, I think it starting to people certainly realize that, you know, teams need to work together, you know, the kind of the cousin between business and it, especially as we go to more kind of SAS based solutions, low code solutions, you know, there's not such a gap anymore, actually, some of our business partners that expense to be much more tech savvy. Um, so I think, you know, this is what we have to kind of appreciate what is its role, how do we give the capabilities, um, become more of a centers of excellence rather than actually doing mounds amounts of work. And for me, and from a testing point of view, you know, mounds and mounds of testing, actually, how do we automate that? How do we actually generate that instead of, um, create it? I think that's the kind of challenge going forward. >>What are some, as we look forward, what are some of the things that you would like to see implemented or deployed in the next, say six to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think, um, you know, certainly for, for where we are as a company from a QA perspective, we are, um, you let's start in bits that we do well, you know, we've started creating, um, continuous delivery and DevOps pipelines. Um, there's still manual aspects of that. So, you know, certainly for me, I I've challenged my team with saying how do we do an automated journey? So if I put a requirement in JIRA or rally or wherever it is and why then click a button and, you know, with either zero touch for one such, then put that into production and have confidence that, that has been done safely and that it works and what happens if it doesn't work. So, you know, that's, that's the next, um, the next few months, that's what our concentration, um, is, is about. But it's also about decision-making, you know, how do you actually understand those value judgments? >>And I think there's lots of the things dev ops, AI ops, kind of that always ask aspects of business operations. I think it's about having the information in one place to make those kinds of decisions. How does it all try and tie it together? As I say, even still with kind of dev ops, we've still got elements within my company where we've got lots of different organizations doing some, doing similar kinds of things, but they're all kind of working in silos. So I think having AI ops as it comes more and more to the fore as we go to cloud, and that's what we need to, you know, we're still very early on in our cloud journey, you know, so we need to make sure the technologies work with cloud as well as you can have, um, legacy systems, but it's about bringing that all together and having a full, visible pipeline, um, that everybody can see and make decisions. >>You said the word confidence, which jumped out at me right away, because absolutely you've got to have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question then for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops, to be able to gain the confidence that they're making the right decisions for their business? >>I think the, the, the, the, the approach that we've taken actually is not started with technology. Um, we've actually taken a human centered design, uh, as a core principle of what we do, um, within the it part of BT. So by using human centered design, that means we talk to our customers, we understand their pain points, we map out their current processes. Um, and then when we mapped out what this process does, it also understand their aspirations as well, you know? Um, and where do they want to be in six months? You know, do they want it to be, um, more agile and, you know, or do they want to, you know, is, is this a part of their business that they want to do one better? We actually then looked at why that's not running well, and then see what, what solutions are out there. >>We've been lucky that, you know, with our partnership, with Broadcom within the payer line, lots of the tools and the PLA have directly answered some of the business's problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which in, in, uh, you know, in some companies include not as they do there is that kind of, you know, almost by understanding their, their pain points and then starting, this is how we can solve your problem. Um, is we've, we've tended to be much more successful than trying to impose something and say, well, here's the technology that they don't quite understand. It doesn't really understand how it kind of resonates with their problems. So I think that's the heart of it. It's really about, you know, getting, looking at the data, looking at the processes, looking at where the kind of waste is. >>And then actually then looking at the right solutions. Then, as I say, continuous testing is massive for us. We've also got a good relationship with Apple towards looking at visual AI. And actually there's a common theme through that. And I mean, AI is becoming more and more prevalent. And I know, you know, sometimes what is AI and people have kind of this semantics of, is it true AI or not, but it's certainly, you know, AI machine learning is becoming more and more prevalent in the way that we work. And it's allowing us to be much more effective, be quicker in what we do and be more accurate. And, you know, whether it's finding defects running the right tests or, um, you know, being able to anticipate problems before they're happening in a production environment. >>Well, thank you so much for giving us this sort of insight outlook at dev ops sharing the successes that you're having, taking those challenges, converting them to opportunities and forgiving folks who might be in your shoes, or maybe slightly behind advice enter. They appreciate it. We appreciate your time. >>Well, it's been an absolute pleasure, really. Thank you for inviting me. I have a extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glenn Martin. I'm Lisa Martin. You're watching the cube >>Driving revenue today means getting better, more valuable software features into the hands of your customers. If you don't do it quickly, your competitors as well, but going faster without quality creates risks that can damage your brand destroy customer loyalty and cost millions to fix dev ops from Broadcom is a complete solution for balancing speed and risk, allowing you to accelerate the flow of value while minimizing the risk and severity of critical issues with Broadcom quality becomes integrated across the entire DevOps pipeline from planning to production, actionable insights, including our unique readiness score, provide a three 60 degree view of software quality giving you visibility into potential issues before they become disasters. Dev ops leaders can manage these risks with tools like Canary deployments tested on a small subset of users, or immediately roll back to limit the impact of defects for subsequent cycles. Dev ops from Broadcom makes innovation improvement easier with integrated planning and continuous testing tools that accelerate the flow of value product requirements are used to automatically generate tests to ensure complete quality coverage and tests are easily updated. >>As requirements change developers can perform unit testing without ever leaving their preferred environment, improving efficiency and productivity for the ultimate in shift left testing the platform also integrates virtual services and test data on demand. Eliminating two common roadblocks to fast and complete continuous testing. When software is ready for the CIC CD pipeline, only DevOps from Broadcom uses AI to prioritize the most critical and relevant tests dramatically improving feedback speed with no decrease in quality. This release is ready to go wherever you are in your DevOps journey. Broadcom helps maximize innovation velocity while managing risk. So you can deploy ideas into production faster and release with more confidence from around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of dev ops and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high-performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago. So it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate physical co-location. Okay. That makes sense. >>It does. I shouldn't have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with jump. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other understanding, Oh, individuals teams, uh, contributed to kind of a business outcome, what is our shared goal or shared vision? What's what is it we're trying to achieve collectively and keeping it kind of aligned to that? Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organization, there's been specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of reverted back to, okay, what is my database all the time? What is my cycle time? >>Right. And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens or these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, favorites experience was to actually weakness at one of a large financial institution. Um, you know, Tuesday Golder's unquote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, test execution results, you know, Coverity coverage, um, official liabilities and all the all ran. It could have a direction level links. And that's when you start to put these things in context and represent that to you in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and, and understand have they joined our company to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And w I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it can, it says simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos. We were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very dev, uh, different environments, depending on what area of BT you're in, whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can, um, drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. >>And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in their kind of testing world at the moment. But, you know, I could talk about things like continuous delivery direct to when a ball comes tools and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum it up, autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter? Jeff, >>I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we look at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test, uh, departments, a group, the fourth level that I think is really important, uh, in, in the current, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric helpful was the parking lot at nine. And how full was it at five o'clock. I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customers. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big impact. And, and, and also how do you measure quality kind of following onto the advice that Jeff provided? >>That's some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker, uh, called me Kirsten a couple of months ago. It taught very much around how important flow management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes long? So I think that's a very important thing for us. It's even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, you know, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how, how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up our teams, training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are, um, delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, Jeff's point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. >>And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about our, these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood is that, you know, when you have a bad cycle time or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, if you look at, for instance, things like lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. >>And so I think that there is, you know, the only the metrics there is, I think a lot of work that we need to do in terms of classifying, descend type patterns, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as, as a key pattern, the industry or anti-spatter. Um, but waterscrumfall right is a key one, right? And you will detect that through kind of a defect arrival rates. That's where that looks like an S-curve. And so I think it's beyond kind of the, the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on this. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff will go back to you. Okay. >>I would say look for the opportunities that this disruption presents. And there are a couple that I see, first of all, uh, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks to be on the lookout for now? I, I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but low code also. Um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the, the life cycle, um, where, when I heard the word scrum fall, it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment we need to get beyond that. We need >>To be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production and go straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately we all need some peanut butter and chocolate. All right. It starts to take us home. That's what's that nugget you think everyone needs to have on their agendas? >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about the lost city, but it's all about sweet. We still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of a spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value into aligning everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>>LA from Las Vegas. It's the cube covering AWS reinvent 2019 brought to you by Amazon web services and they don't play along with its ecosystem partners. >>Well, welcome back live in Las Vegas. We're here on the cube. Continue our coverage here of day two of AWS. Raven 2019 in fact, it took me to the last interview on the second day to be paired up with my guy. Still many minutes to what happened is this is the first interview we've done this way. >>John, you know, I've not been out playing golf >>well and I wouldn't mind if I was, it'd be all right Brandon. You know Brandon, you play golf. Brandon Young? I do. I play college golf so, and I have a, you can't see them, but I have some trousers that might match there and prove that I have done a few times. Paint shirt would be, he would very proud granted to VP of alliances to get lab. And where'd you play college golf by the way. I split some time in Oklahoma and down at rice down in Houston. Oh you, yes. Wow. Be a sooner. How back that has some pretty good golfers there. They do. Um, let's first off, let's talk about, um, VP of alliances sure. And get like what do you do? So what does that encompass? What's that all about? Covers a bunch of pieces. Uh, covers all of the big key partnerships with us. >>So that's going to be obviously Amazon, other big cloud providers, a lot of strategic technology partnerships and then all your system integrators, man service providers, resellers, um, and then functionally anything else that comes in. So also we're bringing the open source space. So lead a lot of our open source engagement, uh, uh, in as well. What kind of customer base we're talking about here? I mean for, for you guys, sorry, cause it's pretty significant. It's, um, so in the space we've got roughly to two to 3 million users that use get lab and count on it for building, deploying and securing their code. Uh, and somewhere between a hundred thousand and 200,000 companies, uh, that get loud is, uh, is being used. Now. >>Brennan, you're not dealing with get lab. You're also on the board for the Linux foundation. And you know, we're, we're getting close to 2020. So I even, I saw some people looking back at where open source has come in the last decade. And you know, get, of course is one of the predominant drivers the proliferation of opensource. So maybe tell us a little bit about, you know, what your customers come to. Uh, w you know, why, why get lab is so critical to what, >>sure. Yeah. Because if we look at history, it kind of makes naturally in get lab we're getting, so that was where our, our base was, uh, when we started in 2012, 2013. Um, as it's evolved, so in get continues to be that core piece you need. So whether you're doing get ops infrastructure is code application development, you've got to have state, you've got to store your issues, you've got to take care of that. That's just one Oh one in software development or infrastructure management. Um, so that's got to where we started. And then, you know, a couple of years later, we picked up and did a bunch of stuff in the CIC space. Initially we had them separate, uh, and customers kept saying, God, these might work well together and to the Linux world has always been single tool, very sharp, very narrow. Uh, so we held off on that for a long time. >>Um, finally said, Oh, we're going to give it a go, shift them together. And that's kind of led to where we are now, which is we think of, you know, get lab as a single tool for the entire dev ops life cycle. And that makes it easy for someone to get started to build it, secure it, ship it, all of that from idea to production in the shortest possible time. And so that's kind of how it evolved. And yeah, we've grown up with the open source world ever since. And um, it's an awesome place. All right, so you've got the alliances and we're here at the biggest cloud show there. So help us connect the dots. Get lab AWS. Yeah. Perfect. So if we kind of look back and we go, ah, look at the keynote, right? So Andy talked a whole bunch, front keynote, Goldman Sachs, big talk with Verizon, a lot around the services, new stuff with arm new chips, new, um, a lot of new databases. >>Um, all of that rolled out. Those are services as Amazon looked at it. Our goal, our job is to get those customers onto the Amazon services. We're the tool that helps them develop and deploy those applications. Goldman, huge customer, Verizon, huge customer. So the majority of the keynotes you'd get lab to get to Amazon. So we're that tool that does the application security deployment and um, you know, lets those devs really take advantage of the great services that Amazon delivers. You know, you talk about security is it, is it, um, and obviously it's increased in terms of its importance. We recognize we've, we've seen how vulnerable apps can be and, and these invasion points, is that being reflected in budgets? Are we seeing that? Are people making these kinds of investments or is there still some lip service being paid to it and maybe they need a little more money where their mouth is. >>There's not a shortage of dollars, so I'll be be real straight forward. That is for us, the big growth area is uh, application security in a pipeline. The notion of shift left, um, and it's been, it's actually one of the easier conversations because the CSOs really want to make sure that every piece of code is tested, be it static code, dynamic code, license scanning, all the above. Um, the way they've had to do that and traditionally done it is at the end of a pipeline and they make every dev on happy because they throw it all the way back to the front with the dev. And then I was like, Oh, thank you so much. I did that two weeks ago and now I have to go, why didn't we do it on the front side instead of the back side? You kill the most important thing, which is cycle time, right? >>Cycle time is time from idea to Chimp. So by shifting it left, there's plenty of money and the CSOs love it because just want you to spend it. It's where they spend it. Right. And so now they get all the code tested. The devs love it because they get feedback instead of the CSO saying this is broken. The two old, the second they hit command a couple minutes later, Oh it's broken. They go fix it, make another commit. They're going to move way faster much. Um, so that's really what we get at and yeah, but no short in dollars, the security still the windows, the spend happens, you're saying right on the front side instead of the back shop and try and get full coverage. So a lot of times otherwise if you're trying to do security after someone's developed it, you're not sure. Like are you getting every code, all a piece of code that was developed? Are you getting just a lot of it as you talked about web apps, a lot of it is the focus. Oh the web apps. Cause that's the front end. But intrusion, once it passed the front end, it's a soft interior. You've got to do every single piece of code has to be tested. >>Yeah. It's Brandon. So you know what I've heard, especially from, I mean, you know, my peers in the security industry, you know, security needs to be considered the entire way. Security is everyone's job chair's responsibility. I need to think about it. But the other thing that really has changed for people is you talk about CIC. D I need to move fast. Well hold on. The security team's got to review everything. One of the core principles of dev ops is you want to bake it in the process, you need to get them involved. And then there's DevSecOps which pulls all of these pieces together. So tell, tell us how those trends are going and that, you know, speed and security actually go together not opposed. >>Oh yeah. And because, and it's how you measure the, the speed. Cause I think sometimes the question is all back to what is it from it. It's, it's a life cycle. And if that's what you're measuring, being able to do the security earlier is so much faster because you're not having to iterate, um, later. But, um, it's continues to increase. Devs are getting more and more say that's not gonna change anytime soon. Um, empowering those devs to own the security, uh, empowering those devs through the pipeline to be able to deploy into Lambda, into far gate. They love that. And if you could give that and give the security, the visibility, the dashboarding, the understanding of what just went in, um, what code they're using, what the licenses are, that visibility is huge and that allows you to move fast cause it's trust. >>I mean actually, uh, I love the researchers at Dora, you know, do the annual survey, uh, on dev ops and they said, actually if you are a company that tends to deploy less often, it tends to take you much longer to recover and you're not geared to be able to do it. Uh, you know, my background networking and you think about, you know, security is one of those things like, well wait, I want to keep my things stable and not changing for a while, but that means you're less and less secure cause I need to be on the latest patch. I need to be able to update things there. So, uh, you know, CIC D I think leads to should lead to greater security. Do you have some stats around that for your customer as to, you know, how they measure that? >>We have some pretty good velocity. Um, so Goldman went with us and this is real public is they, they started with us and went from about a two week release cycle down to tens, 20 a hundred times a day. Um, and that, I mean that's a company that does a great job in dev, um, but can also be like smaller companies like wag labs that we talked with earlier and they same kind of thing. They went often from a week down to they were doing, they typically do 20 to 30 deployments a day. And again, it just makes you break the pieces smaller, less likely that you're going to introduce dependencies that break something and all that process builds on each other as the door is stuff. If you haven't read, you've read it obviously, but if the users haven't great place to get started and understand how this works. >>Has testing changed or is testing changing in terms of when you establish the criteria, what you're looking for in terms of I guess you have a lot of new capabilities so you've got to change, I assumed your criteria up front do have a little proper, a little more accurate evaluation is that environment it's changed somewhat. I mean testing in application testing it is pretty specific to every comfy. So tools continue to get better. Um, ways of review have gotten a lot better. So, uh, there's now a lot of capabilities that at the point that you're going to go into deployment, one of the harder pieces is doing, um, your user acceptance testing is like, God, am I going to see the same thing that a user will? Right. And a lot of these have gotten to a point like we have a one click at the end of the deploy, a review app. >>Anyone in the company can look at exactly rebuild everything you're going to bought to deploy. So there's some tools that make it faster. Um, but in terms of what your load balancing in terms of your user acceptance testing, a lot of those principles continue to be pretty girl. Uh, one of the big things we heard from Andy Jassy is talking about transformation and he said you can't just do it incrementally and you need, you know, clear leadership and commitment. We want to hear how, you know, you're hearing about this from your customers. How is get live helping customers along those transformation journeys. Sure. Um, so totally agree that, I mean, it's a cultural piece, uh, without question. I think there's a couple of places, there's the obviously the tool piece and just getting everyone on the same page. And we, we all know this intuitively is we've seen what w when you go from a word doc to a Google doc and everyone can edit the same time, that's transformation goes, you know what everyone's working on, uh, and you're not duplicating effort. >>And that, that's really in many ways that's what get lab is doing is just helping the front end. I, you know, product manager know exactly what's going on in the infrastructure side and you communicate in a similar language. Um, the other piece of that we are working a lot in is because, um, get lamb operates an extremely open culture. So we publish how we run the company in a handbook that's 2,500 pages. We're always updating it. So, uh, we do reviews every time we release, we release every single month for the last 120 months in a row. We go through, here's what the release is going to be. It's on YouTube. Everyone can see it when things go wrong, we publish it. So we have an outage, we will, we have live broadcast, how we get back out from an outage and we publish all of it for someone to understand. >>And so one of the other things, there's a lot of our customers are getting started on that journey. There's one thing for a deck that says, here's what you do for your transformation for your company. That's another thing when you can literally jump in on Monday morning under the get lab call and watch, get lab go through a post-mortem of when we had a small outage. Oh that's what a no blame looks like. Okay, now I understand that, Hey, what, what didn't we release that we could have done better? And those are processes that you can have it on a piece of paper, but it's a different thing when you can walk through that with the company. And it's even better when you're watching the company that's doing the same product, the same tool that you're using. So I mean that's a, that's a cultural decision. >>Yes. I mean it's gotta be right. Yeah. I love the no blame. Right. Cause you're saying instead of finger pointing, great or castigating, you know, we're, we're going to learn from this. And how do you think, what impact does that have on a customer when they see you in real time solving your problems? They know that. They know that if they have a question for us, that we both take it seriously and that we're going to do it in a way that they know when it's going to be resolved. And that doesn't mean that we always deliver at the same time that a customer asks. But that level of transparency breeds both trust. And it also helps a customer quantify what do they want, helps us huge amount of communication because they know what we're prioritizing and they understand why. And that isn't something that is typical to come, but it's always typically very hard unless you're broadcast everything like we do to know, well, why are they making that decision? >>Um, and so that's one of the real big reasons that our customers work with us. That's where we get 10,000 plus additional contributors to get lab as an open source project. And that helps massively of course. So the velocity is because there's no difference between a get labber or the thousand get lappers in 64 countries or any one of the 10,000 contributors or our biggest competitors that regularly make contributions to, uh, our, um, our landscape. So we have a landscape that's, how does dev ops work? Who does stuff well? Hey, have no shame if they delivered something better. I want to know that I make that commit. We will share it with the world that we are not good at that and you are better at it and you know what? We'll get better. Right. It's a winning formula. It's good. It's been working really well. I appreciate the time brand. A good saying. You can love the slacks. Wish we could show them of course. But next time, thanks for having us. All right. You're watching Carvery Cherif AWS reinvent 2019 on the queue.

>> Live from San Francisco, it's theCUBE, covering Google Cloud Next '19. Brought to you by Google Cloud and its ecosystem partners. >> Hi everyone, welcome back to the third day of live coverage of theCUBE here in San Francisco for Google Cloud Next 2019. I'm John Furrier, the host of theCUBE, my co-host Stu Miniman, Stu, good to see you this morning. >> Great to see you, John. >> Got a very special guest here, George Kurian, the CEO of NetApp, not to be confused with Thomas Kurian, his twin brother, who's the CEO of Google Cloud, George, it's great to see you. >> Good morning. >> Thanks for stopping by. >> Thank you for having me. >> So, when you've been walking through the hallways, you getting like, a lot of looks and some selfies, um, people want to take a selfie with you, thinking you're Thomas, the famous? >> Quite a few, quite a few. >> So what's it like? >> Oh, it's exciting to see all of the innovation here and the real commitment that Google's made to building out an enterprise platform. We've been working with them for many years, and, uh, we're excited at all of the potential new opportunities that creates, alongside Google's customers and ours. >> Yeah, George, it's got to be interesting, it's almost a little bit of a mirror image, is Google is looking to get deeper into the enterprise, and of course we've been documenting NetApp for many years now, has moved beyond just being an enterprise company, you've been moving to the cloud, maybe, just go back, tell us a little bit- some of the lessons you've learned, and, you know, what you're seeing happen, dynamics in the space. >> I think customers, Thomas said, you know, many of the core tenets that we see which is customers want to operate in a hybrid, multi-cloud world. They want to have cloud technology integrated into their data centers and conversely their applications be portable with a common programming model. I think it's come a long way. You know, I think our technologies now available natively in the Google Cloud, I think the programming model with microservices and containers, and with Kubernetes as an orchestration layer, truly allows, you know, this kind of hybrid world to operate, and I think our opportunity there is to help our customers use data properly across all of these landscapes, understand where it is, you know, orchestrate new applications as well as traditional, uh, so that they can progress the business. And so it's, you know I tell you coming to these conferences over the last three to five years, you can see the pace of change really start to accelerate. I'm interested in what you guys think about. >> Well, one of the things we've been commenting on theCUBE in our opening segments is kind of looking at how the transformation of Google Cloud from Google large-scale, they know data, they know tech, into becoming an enterprise, so, a lot of window dressing around the event, you know, digital transformation, all the right words, but they got the technology. And, you know, one of the things I'd love to get your perspective of because it's not- it's no secret that the Kurian brothers, yourself and your brother, Thomas, are, have great tech chops, also have tons of enterprise experiences, you specifically have been involved in a lot of ecosystems. That's been a big topic here. Can Google really get an ecosystem up and running, I mean, they participated in the CNCF with Cloud Native, but, as an organization, this is something that you're very familiar with, uh, at NetApp, you've been in many ecosystems, you've seen the formula. How, how, how should that evolve, because it's changing, the service's base, I think you're part of the Console Google Cloud from what I've been reporting here. What's the ecosystem formula for, for this new cloud world? >> You know, I'll tell you that, uh, enterprises expect their providers to work together, that's always been the expectation, and, uh, we've had to coexist with even our competitors for a long period of time. I think the core ideas there are to keep the customer at the center of the discussion and figure out how to best solve their problems, regardless of whether it is having to coexist with someone else, right? I think what's been interesting to me is, Linux has really become sort of the core underpinning of the cloud, and Linux was an open-source technology that, in the early years, IBM backed and sponsored. I think containers together with uh, you know, what Google's doing to sponsor it, has really become the opportunity to create the next, kind of layer of, you know, common development model, programming model, common orchestration. I think there's that promise, I think, uh, it's got to be realized. >> George, you, uh, you talked about, uh, the change that we see in the industry, and, you know, we know enterprise is not like, oh, let's just redo everything we were doing, whether I'm a five year old or a, you know, hundred and fifty year old company, I have things that I need to look at, and, I mean, the applications are really tough. It'd be wonderful if I just had a clean sheet of paper, and I can make it all serverless or containerize all my pieces there. Um, the message I heard a lot this week is, you know, meeting customers where they are. It's not just, Google we know has great tech, and smart people, maybe a little too smart sometimes, but, you know, I'd love to hear your viewpoint is, you know, those enterprise customers, are they catching up to the pace of innovation faster and making more change, or, you know, is it still one of these things that we're going to measure in decades as to how long it takes to move things. >> I think it, you know, I see it in a couple of, uh, ways. One is which industry are you in, and the impact of, you know, transformation to your industry. I think if you are in a highly digitally-oriented industry, like media and entertainment, you've got to transform quickly because the whole industry's getting transformed, right? I think conversely, if you're in an industry where digitization helps your workforce be more productive, I think you can take more time. What we see also is, in the places that, uh, are common, for example, in how you evolve your customer experience and how you interact with customers, we see virtually every company needing to transform, right? So I see that, you know, this is a long transformation, it's not going to happen overnight. I think that customers will pragmatically choose to, you know, either refactor existing applications or bill Net New, on a case-by-case, business-process by business-process basis. And that's why we see hybrid, sort of being the de facto operating model. >> George, I want to get your thoughts on multi-cloud and hybrid, obviously the modern application renaissance and revolutions kind of happening, whether you call it a renaissance or revolution, applications are exploding. That's clear. Multi-cloud and hybrid-cloud are key architectural shifts. I'm writing a story right now about the Department of Defense big contract that was awarded to, or to, shorthand, Microsoft and AWS. But, one of the things that have people arguing is that it should be multi-cloud! Now, the Department of Defense is an example, and this is public sector, but also enterprises have the same makeup, they have hundreds of cloud projects. Hundreds! And the Department of Defense is five hundred cloud projects. So there's not one cloud, that's not Amazon. So, this is a world where workloads and cloud selection and the parts of the architecture have to support multiple clouds. Can you explain that, kind of, what that means to customers? Because people get often confused coming from the old way. I'm buying IBM, I'm buying Oracle, I'm buying Google Cloud, and we're done. No, it's really not that case. Can you, kind of, can you react to that? >> Most enterprises that we speak to have hundreds of applications, everything from, you know, mainframe-based core business processing, to highly digital, you know, mobile-based, customer interaction applications. I think they have, sort of, a portfolio approach to manage those, where they say, hey, some of those are going to stay on premise, some of those are going to stay in a private cloud, and then I've got this palette of, you know, choices around whether I choose software as a service or infrastructure or platform as a service. And I think that when you look at a, you know, a reasonably large company like ours, we run about five hundred applications in the company. There's no single palette, right? You've got to have these inter-operate, I think from a governance standpoint from how you integrate the data across these landscapes, and from how you ensure compliance, security, and so on. And I, so I think, you know whenever a company tries to say that I can do everything, I think that's a little bit facetious, to be honest. >> And so, the reality is, multiple workloads, multiple cloud projects will happen, multiple vendors, but in a new way. Workload driven, with the data, obviously the data's critical, storage is key. Um, Stu, you want to- >> Yeah, so, you know, I think back to the storage world, storage was always a fragmented marketplace, and I had my application silos that I, that did this. Now, what have we learned from multi-cloud, that would, from multi-vendor, as we go into multi-cloud and, how can we allow customers to really unlock that value of data, because if it all stays fragmented in silos, it's a lot harder to be able to actually leverage it, use it, for all the, you know, AI, ML, or data, uh, value. >> Absolutely, I think, you know, one of the long-term theses we've had is that the world gradually moves from system-centric or process-centric to a data-centric world where the core asset that you're operating on is not the value of an individual business process, but the integration across your business processes, right? And so, this is why we think in a hybrid world, you need something like a data fabric to stitch together all of these landscapes. Those landscapes need to increasingly be stitched together in real time because of the speed of decision making or the use of, you know, real time analytics or real time business deci-, you know, processing. And so that's why we've integrated our technology into multiple landscapes, right, both traditional, but increasingly containerized or cloud-based, cloud-native applications. And I think that's again a multi-year journey. I think IT has to transform, IT architectures has to transform, and frankly businesses need to as well. They need to think about data as a property of the whole business, rather than a for function or a department. >> So just to click on that, double click on that for a second because, what you're saying is, a data fabric allows for multiple data to move around the workloads. So what you're saying is, if you want to take it- well, I'm saying- want to take advantage of machine learning and AI, the data has to be addressable in real time. Meaning, you don't have time to go fetch it from a database that may or may not be available at any given time, so making data addressable, horizontally scalable, for whatever workload, at any given time from retail to personalization, or whatever, right? >> Absolutely, right, so for example, if you look at the way a, um, AI or an ML data pipeline works, there's a period in the pipeline which is about training and feature engineering where you're trying to develop the model the right way. And then you're going to let the model run, but the model's going to be reacting to real time data input and constantly making transformations to how the business reacts. I think that data input needs to be fed in from all of the business processes that support the business, right, rather than a, hey I'm going to create an artifact that's, uh, static artifact that's trained once and then you're going to run the business. So that's why we think you've got to operate the hybrid world as an integrated world at the data layer. >> Yeah, George, one of the interest, there's a study, uh, that Google put out that they had acquired a group, DORA that looks at high performing environments, uh, and you know, what differentiates kind of, the, you know, the leaders of the pack. You talk to a lot of companies, and I'm sure you must, you know, have some, you know, opinions on this. Tell us, what, what is separating, you know, the leaders in the end user space, as to, uh, you know, from, from those that are, that are following. >> I think that, uh, the leaders, you know, are, have the capability to transform themselves, and transformation, you know, people talk about digital transformation. I think the most important part of that is actually the transformation part, and it's organizing people to allow experimentation, learning from experimentation, to celebrate failure, I think that's hard for big companies to do, right? Because you're set up to ensure that you're managing the risk of not failing, on the other hand, I think, in a world where there's a new game being created, you got to be able to allow the organization to try different things and it's okay to fail. >> And the speed pressure, too, to go faster, certainly with cloud, everything's accelerated from time to market, time to value, technology development. >> Absolutely. And I think that is also one of the fundamental changes going on in the industry. We were at the end of a paradigm where there were horizontal slices of expertise, which is really the ultimate optimization of an existing paradigm. The new paradigm isn't exactly clear, so, you know, to move faster, IT is creating vertically integrated squads. You look at, you know, Google's creation of a site reliability engineer, it's really a way to accelerate the creation of digital services and optimize the infrastructure associated with it, so. It's a time of change, I think, you know, our view is you got to lean into it, and, uh, you've got to trust the fact that the skills and the cultural values that you've brought are going to help you innovate into the future, not necessarily just the products and the ways that you've done them. And so that's why we think culture is a massively important part of these transformations. >> We're here with George Kurian, CEO of NetApp, not to be confused with Thomas Kurian, CEO of Google who's also walking around the floor, show floor, talking to customers. George, thanks for coming on and sharing your insight, you guys are awesome, the twins are super smart, running two big companies, thanks for spending time. Share us personal story, with George, I mean Thomas hasn't come on yet, he's too busy, we'll get him later on theCUBE, but share a story about him, what's he like, who wins the arm wrestling matches, share a- what's he like, tell a personal story. >> I think he's shy, uh I think we're both really, we realize how lucky we are, you know, we grew up in places where people, you know, some of us had sort of unmerited grace, you know, the blessings of being born to extraordinary, good families and parents, and so we're always cognizant of that. It's amazing that two guys in India, who had never seen a computer till we left India to come to the United States, now have the opportunity to be a big part of the computer industry, so we're just really grateful, and God's been good to us. >> Well congratulations, love the tech chops, value and culture, big deal right now, thanks for spending the time sharing the insights, appreciate it. >> Thank you for having me. >> George Kurian here on theCUBE with John Furrier, myself, and Stu Miniman, more CUBE coverage after this short break. (upbeat music)

>> Hey, welcome back here everybody. It's Jeff Frick here with theCUBE. We're at PagerDuty Summit. It's in San Francisco at Pier 27. It's a new facility, we've never been here. It's pretty unique. It's right between the Bay bridge and Pier 39. Beautiful day out on the water and it's all about DevOps here at PagerDuty. And I'm going to tease Jen later if people even know what a pager is at this town. So we are excited to have Nicole Forsgren She's a founder at CEO and chief scientist of DevOps research and assessment. I had to read it, it's a big mouthful but it goes buy DORA for sure. Nicole, welcome to see you. Good to see you. >> Thanks so much. It's good to be here. >> Alright so you are the DevOps expert. You got a really interesting past. Did some research on the LinkedIn profile industry. Academe industry, Academe and now you're out helping people. >> Yes, bounce around a bit. It's all about the pivot right? >> Absolutely. >> Out here doing DevOps. >> Absolutely, absolutely so you do an annual report on the state of DevOps. So where are we? DevOps has been being talked about for a long, long time. How much is reality? How far are we on this journey? What are you seeing? >> Right so it's really interesting you point that out right, because for years everyone's been like DevOps. What is it? Does it matter? And so DORA and by the way, DORA is myself. Jess Humble, Jame Kim. We just brought on Sue Chow. But the core founders, we've partnered up with the team at Puppet, and for the last several years. We've put out the state of DevOps report. To kind of help define at least from a research standpoint and from our standpoint. What it is? What are the key contributors to really drive value and does it drive value? It's for years and I'll talk about this later this afternoon on my closing keynote. For years and when I say years, I mean decades of academic rigorous, pure review research. Technology didn't matter. Like it didn't matter at all. It just never delivered value to organizations. But then we started seeing patterns and really interesting patterns and companies saying no. We're seeing results, we're delivering value. We're delivering outcomes. Core essential outcomes for end users and customers in the business. And so we got together and say okay, let's really take a look at this in a really important way. >> Right, now how far we've come right. 'Cause now most companies are technology companies. They just happen to warp their technology around a particular product or a particular service. >> Yeah, exactly. >> And now most leading the technology in terms of a vehicle to drive value and to drive transformation. So DevOps is also very wrapped up in this whole concept of digital transformation. That's all anybody wants to talk about. It's in every earnings call, so how closely are the two related and how do you see, 'cause DevOps got a little bit more history in terms of the buzz of transformation. Are people applying DevOps concept beyond strictly development and operations? >> So, there's a lot to unpack there. So like you said, it's really, really involved. Although it has some kind of a buzz word, right? Some people love it, some people embrace it, some people never want to hear it. So it's really all about what's important to the company in delivering value. But it's core is really about taking important methodologies and practices to deliver value and it's about using technology and automation, in conjunction with core values and practices and processes that we've adopted from the lane and agile movements. >> Jeff: Right, right. And having a really good healthy culture that's about more than just DevOps. Right like you said. DevOps, QA, Info Sec. The business marrying all of that, pulling all of it together, working in conjunction in the right kind of ways to deliver value. To deliver key outcomes to help us pivot, move fast, learn, have fast feedback. So that we can do what we need to do for the company, for the business, because like you said, it's so many companies right now, really are technology organizations that happened to be wrapped around in some particular industry. >> Jeff: Right, right. >> Capital One is a financial institution. Really they are a technology organization that happens to do finance and deliver finance really, really well for their customers. So many other companies are doing retail but it's driven by technology. Right or they do insurance and it's driven by technology or they're a healthcare organizations that really can't do what they do unless they have technology to really drive it. >> Right, right. The financials institutions are interesting because if you talk to like my kids. If they've ever been inside of an actual bank and then and how often do they go to the atm? So not even atm, so the way that people more and more interact with the company is through digital mediums. >> Right. >> But I'm curious to get you're input on the big question that we always ask people is how do I get started. Right, what is the easy paths to success? How do I get some early success so I can build on that success? What's interesting is you have a very unique approach to solve that question as oppose to what I think or based on what I'm really good at, I think we should start here. >> Yes, we really do-- >> Do you guys have different-- >> And this is really why DORA exist and this is what we do. So myself Jess Humble, Jean Kim. This explains the genesis of DORA. So we have a couple different things so the mission of DORA is to help companies get better through science and proven methods. Ans so we have a couple of different things we do. The first is that state of DevOps report that we put together at Puppet. And those are all open sourced and so if you want some ideas of what really statistically drives improvement, go find those. They're open source, they're totally free. We've tried so many resources because we don't want companies to fail. We've all lived through that awful dot com mess. We've seen companies fail. Go find those resources. Now your question though, where should I start? If I'm a company, what should I do? We've all go into conferences myself, Jean, Jess and we've had companies come up and say well where should I start? And the answer is always, it depends. The answer is always it depends because I can't tell you absent context, absent data, absent information. If I don't know about someone's detail information. I can't tell you and so what we also have is we offer an assessment where I can collect data from the doers. Right there's this fantastic report from Forester. It's called the dangerous disconnect and that's such a great title because if you ask executives. They drastically over estimate technology and DevOps maturity in organizations. So you shouldn't be, I mean I love-- >> Over estimate. >> Of course they do. I mean because we need to be really, really optimistic about where our organizations are going. >> Right, right. >> Those are our roles as executives. And so that's appropriate but in certain conditions that's appropriate. But where it's not appropriate is when you're setting detail strategy for your organizations. And so what we do is we offer an assessment where using these strong scientifically based measure that we have prepared and refined over now, four years of rigorous academic research. We can go with a 15 minute survey, collect data from everyone in organization that like I said are the doers. DevOps, TestOps, QA, InfoSec including vendors, contractors, consultants to people that are in the weeds every single day. I can measure you. I can benchmark you against the industry. I've got over 23,000 data points around the world. All industries, all company sizes. And then, where should they start? I can algorithmically tell you what your bottle neck is, what your constraint is. Where you should start to accelerate your performance. >> Based on my data? >> Based on your data. >> Based on your algorithms and based on your population data from this huge data set >> Yes, and with the companies that we're working with right now, they're seeing amazing results. They're calling it out-sized results. So a really great example we have was with Capital One. They did the assessment across over a dozen lines of business. And by focusing on two core capabilities out of over 20. We focus them on the right two capabilities. They saw a 20X improvement in deploy frequency in only two months with zero increase in internet. >> 20% improvement-- >> 20X >> 20X? >> 20X >> In two months. >> 20 times. >> Wow. >> So it's that ability to measure consistently see visibility throughout that software engineering life cycle. So we also had feedback from customer like Verizon. That that visibility, that consistency of measurement was also a really huge value add. >> Jeff: Right, right. >> Measurements hard. >> Well it's interesting, I saw some of your videos and some of your prior key notes and stuff and talking about, everyone says data is in the world. But the data without context, the data without the right algorithms, and you talk about a bunch data dirty things and data problems. Data itself is not the new oil. So I wanted to get to your report 'cause that's kind of your bench mark. That's your big stake in the ground. So how are we've been doing it? What do you do different than other things that are out there? Besides the fact that it's open source which I'll ask you about as a follow up. What makes your research special? >> So why is our report different from any other reports out there? I think there's a couple things. The piece that makes me the proudest is that, the state of DevOps report is so different because it's academically rigorous. It's a true research report and I love that the team has been so loving and so patient with me. Because when I started working with the rest of the group four years ago, I stepped in and I said. This is what I want to do. These are my ideas. I was still a professor at the time, so as you mentioned, I was industry and then academia and I'm now in industry again. But I stepped in and I said, I think there's this really, really fantastic opportunity to take a look of what's going on but we have to measure this in really rigorous ways. And by doing that, it allows us to look at predictive relationships, which is interesting because it let's us say. If we focus on core capabilities, they will predict organization's ability to develop and deliver quality software with speed and stability. Which will in turn drive improvements in organizational performance. Profitability, productivity, market share. Effectiveness, efficiency delivering mission and organizational goals. Notice I'm saying predict and drive. I'm not saying correlate, which is really interesting. And so in these years of research, we've been able to identify core capabilities that drive improvement. So it allows organizations to understand what's important to invest in. It's not just this worked for my team. This worked for that team. Hey, I think this is what I'm going to try because as someone fond of joking. Anecdote is nice but the plural of anecdote isn't anecdata. (laughing) Right, and that was my frustration when I was in tech and before and when I was in consulting. If you want to try a thing and you want to apply it but it's really hard if I only have one or two or three or five maybe even 10 stories. We need so much data to really understand what will likely work for teams and for industries as a whole. And like I said, God bless the team, because I came in and I was really rigorous and I would say that doesn't work, we can't measure that. That doesn't work here and sometimes I'd come back and I'd say that doesn't hold. The stats don't hold and they say, "But it has to." "I know it worked here and I know it worked here." And I'm like, but it's not, we have no evidence to support that. The stats don't hold. This doesn't work. We can't say that and we're like hey, we'll have to try it again next year. Not try it again next year but we have to find a different way to measure it. We have to have a different hypothesis to test. But then we also find really amazing things like I said a couple times, it predicts a team's ability to develop and deliver code with speed and stability. Speed and stability. We found four years ago speed and stability go together. For years, we didn't know that was the case or we thought that in order to get stability, you had to slow down. It doesn't show up anywhere in the data. No where, high performers get both. >> So do the executives, do they realize the leader that having better internal thought for development has an impact on their business relative to saving a few bucks on parts or spending a few more bucks on marketing? As a real driver of value as oppose to it's just always internal apps that we have to build for whatever reason. >> They're starting to get there. And so what we're starting to do is we're really focusing heavily on delivering code with speed and stability. And then, we're saying okay, imagine if you could deliver with speed and stability here. What could you do with delivering features? How does that help you get to market faster? How does that help you beat your competitors? How does it allow you to respond to complaints and regulatory changes? And so that's really what helps us drive and then another way that we are a little different from other reports that are out there. Other industry reports are also very helpful but they are very different. So I don't say things like 27% of the industry is using configuration management. Other report say that and that is interesting. I don't report on percentage of the industry that's doing something. >> Right, right. >> But those other reports can not say what is predictive of improvement. So we are the prediction. Occasionally, I'll report correlations if I don't have the statistics to go as strong as-- >> And what moves it from correlation to prediction is the strength of the algorithms? >> No, it's the strength of the research design. >> The strength of the research design upfront? >> Yep, up front. >> Before you feed it in. >> Upfront and-- >> 'Cause really, you're knocking them at research. >> Yes. >> Rigor. >> Yep. >> That's the underpinning of the whole thing. >> And much more data has been published in academic periodicals, so we are still actively doing research. >> And I would imagine that the annual report is really an ongoing, longitudinal study across a whole lot of the same companies over and over and over, year in, year out. So you get them-- >> So it's open every year. >> As well. >> Yep. >> Awesome, alright Nicole. Well that is fascinating and everyone should go to DORA and get the free research. And then if they want to bring you guys in, and you offer custom services to help the particular company execute and do better. >> Yes, absolutely. So you can go to DevOps-research.com to find all of our research and anything else you want to find out about engaging with us or anything like that. >> Nicole Forsgren. She's DORA the explorer. She'll help you out with your DevOps. I'm Jeff Frick, you're watching theCUBE from PagerDuty Summit. Thanks for watching. (uptempo techno music)