(bright music) >> Okay, welcome back everyone. It's theCUBE's coverage here in San Francisco, California. We're live on the show floor of AWS Summit 2022. I'm John Furrier, your host of theCUBE. And remember AWS Summit in New York city coming up this summer. We'll be there as well. And of course, re:Invent at the end of the year for all theCUBE coverage on cloud computing and AWS. The two great guests here from the APN, Global APN, Serge Shevchenko and Jeff Grimes Partner Leader. Jeff and Serge is doing partnerships. Global APN >> AWS Global Startup Program. >> Okay, say that again. >> AWS Global Startup Program. >> That's the official name. >> I love it >> Too long for me. Thanks for coming on. >> Yeah, of course. >> Yeah, appreciate it. >> Tell us about what's going on with you guys? How was you guys organized? You guys, we're obviously we're in San Francisco bay area, Silicon Valley, zillions of startups here. New York has got another one we're going to be at. Tons of startups, a lot of them getting funded, big growth in cloud, big growth in data, security, hot in all sectors. >> Jeff: Absolutely. >> So maybe we could just start with the global startup program. It's essentially a white glove service that we provide to startups that are built on AWS. And the intention there is to help identify use cases that are being built on top of AWS. And for these startups, we want to provide white glove support in co building products together, co-marketing and co-selling. Essentially, the use cases that our customers need solved that either they don't want to build themselves or perhaps more innovative. So the AWS Global Startup Program provides white glove support, dedicated headcount for each one of those pillars. And within our program, we've also provided incentives, programs, go to market activities like the AWS Startup Showcase that we've built for these startups. >> Yeah, by the way, awsstartups.com is the URL. Check it out. Okay, so your partnerships are key. Jeff, what's your role? >> So I'm responsible for leading the overall effort for the AWS Global Startup Program. So I've got a team of partner managers that are located throughout the US managing a few hundred startup ISVs right now. >> Yeah, you got a lot. >> We've got a lot. There's a lot. >> I got to ask a tough question. I'm a startup founder. I got a team. I just got my series A. We're grown and I'm trying to hire people. I'm super busy. What's in it for me? What do you guys bring to the table? I love the white glove service but translate that. What's in it for me? What do I get out of it? What's the story? >> That's a good question. Focus, I think. >> Jeff: Yeah. >> Because we get to see a lot of partners building their businesses on AWS. So from our perspective, helping these partners focus on what do we truly need to build by working backwards from customer feedback. How do we effectively go to market? Because we've seen startups do various things through trial and error and also just messaging. Because oftentimes partners or rather startups try to boil the ocean with many different use cases. So we really help them sort of laser focus on what are you really good at and how can we bring that to the customer as quickly as possible? >> Yeah, it's truly about helping that founder accelerate the growth of their company. And there's a lot that you can do with AWS but focus is truly the key word there because they're going to be able to find their little piece of real estate and absolutely deliver incredible outcomes for our customers and then they can start their growth curve there. >> What are some of the coolest things you've seen with the APN that you can share publicly? I know you got a lot going on there, a lot of confidentiality, but we're here, a lot of great partners on the floor here. I'm glad we're back at events, a lot of stuff going on, digitally with virtual stuff and hybrid. What are some of the cool things you guys have seen in the APN that you can point to? >> Yeah, absolutely. I can point to few, you can take them. So I think what's been fun over the years for me personally, I came from a startup, ran sales at an early stage startup and I went through the whole thing. So I have a deep appreciation for what these guys are going through. And what's been interesting to see for me is taking some of these early stage guys, watching them progress, go public, get acquired, and see that big day. And being able to point to very specific items that we help them to get to that point. And it's just a really fun journey to watch. >> Yeah, and part of the reason why I really love working at the AWS Global Startup Program is working with passionate founders. I just met with a founder today, he's going to build a very big business one day and watching them grow through these stages and supporting that growth, I like to think of our program as a catalyst for enterprise sort of scale. And through that we provide visibility, credibility and growth opportunities. >> A lot of partners too, what I found, talking to staff founders is when they have that milestone, they worked so hard for it. whether it's a B round, C round, or public or get bought. Then they take a deep breath and they look back at, wow what a journey it's been. So it's kind of emotional for sure, but still it's a grind. When you get funding, it's still day one. You don't stop. It's no celebrate, you got a big round or valuation. You still got to execute. >> And look it's hypercompetitive and it's brutally difficult. And our job is to try to make that a little less difficult and navigate those waters where everyone's going after similar things. >> Yeah, and I think as a group element too. I observe that startups that I meet through the APN has been interesting because they feel part of AWS. >> Serge: Totally. >> As a group of community, as a vibe there. I know they're hustling. They're trying to make things happen. But at the same time, Amazon throws a huge halo effect. That's a huge factor. You guys are the number one cloud in the business. The growth and every sector is booming. And if you're a startup, you don't have that luxury yet. And look at companies like Snowflake that built on top of AWS. People are winning by building on AWS. >> Our program really validates their technology first. So we have what's called a foundation's technical review that we put all of our startups through before we go to market. So that when enterprise customers are looking at startup technology, they know that it's already been vetted. And to take that a step further and help these partners differentiate, we use programs like the competency programs, the DevOps competency, the security competency which continues to help provide a platform for these startups, help them differentiate, and also there's go to market benefits that are associated with that. >> So let me ask the question that's probably on everyone's mind who's watching us. Actually, I asked this a lot. There's a lot of companies startups out there. Who makes the cut? Is there a criteria? God, that's not like it's sports team or anything. >> Sure. >> There's activate program, which is like there's hundreds of thousands of startups out there. Not everyone is at the APN. >> Serge: Correct. >> So ISVs, again, that's a whole nother. That's a more mature partner that might have huge market cap or growth. How do you guys focus? How do you guys focus? >> Serge: Good question. >> A thousand flowers blooming all the time. Is there a new way you guys are looking at it? I know there's been some talk about restructure or new focus. What's the focus? >> It's definitely not an easy task by any means but I recently took over this role and we're really trying to establish focus areas. So obviously a lot of the ISVs that we look after are infrastructure ISVs. That's what we do and so we have very specific pods that look after different type of partners. So we've got a security pod, we've got a DevOps pod, we've got core infrastructure, et cetera. And really we're trying to find these ISVs that can solve really interesting AWS customer challenges. >> Do you guys have a deliberate focus on these pillars? So one, infrastructure. >> Security, DevOps and data and analytics and then line of business. >> Line of business, like web marketing solutions. (group chattering) >> Yeah, exactly. >> So solutions there. >> Yeah. More solutions and the other ones are like hardcore. So infrastructure as well like storage, backup, ransomware, kind of stuff. >> Storage, networking. >> Okay, yeah, the classic. >> Database, et cetera. >> And so there's teams on each pillar. >> Yep. So I think what's fascinating for the startups that we cover is that they truly have support from a build market sell perspective. So you've got someone who's technical to really help them get the technology figured out, someone to help them get the marketing message dialed and spread, and then someone to actually do the co-sell day to day activities to help them get in front of customers. >> Probably the number one request that we always ask for Amazon is can we wish that SOC report, oh download it on the console, which we use all the time. >> Exactly. >> But security's a big deal. SREs are evolve in that role of DevOps is taking on DevSecOps. I could see a lot of customers having that need for a relationship to move things faster. Do you guys provide like escalation or is that a part of a service or not part of? >> So the partner development manager can be an escalation point, absolutely. Think of them as an extension of your business inside of AWS. >> Great and you guys, how is that partner managers measured? >> On those three pillars. >> Got it, okay. >> Are we building valuable use cases? So product development. Go to market, so go to market activities. Think blog posts, webinars, case studies, so on and so forth. And then co-sell. Not only are we helping these partners win their current opportunities that they are sourcing, but can we also help them source net new deals? >> Jeff: Yeah. >> That's very important. Top ask from the partners is get me in front of customers. Not an easy task, but that's a huge goal of ours to help them grow their top line. >> In fact, we have some interviews here on theCUBE earlier talking about that dynamic of how enterprise customers are buying. And it's interesting, a lot more POCs. I have one partner here that you guys work with on observability. They got a huge POC with Capital One and the enterprises are engaging the startups and bringing them in. So the combination of open source software, enterprises are leaning into that hard and bringing young growing startups in. So I could see that as a huge service that you guys can bring people in. >> Right and they're bringing massively differentiated technology to the table. The challenge is they just might not have the brand recognition that the big guys have. And so that's our job is how do you get that great tech in front of the right situations. >> So my next question is about the show here and then we'll talk globally. So here in San Francisco, Silicon Valley, bay area, San Francisco bay area, a lot of startups, a lot of VCs, a lot of action. So probably a big marker for you guys. So what's exciting here in SF? And then outside of SF, you guys have a global program. You see any trends that are geography-based or is it areas more mature? There's certain regions that are better. And I just interviewed a company here that's doing AWS Edge really well. It's interesting that the partners are filling a lot of holes and gaps in the opportunities with AWS. So what's exciting here, and then what's the global perspective? >> Yeah, totally. So obviously, a ton of partners from the bay area that we support, but we're seeing a lot of really interesting technology coming out of EMEA specifically. And making a lot of noise here in the United States, which is great. And so we definitely have that global presence and starting to see super differentiated technology come out of those regions. >> Yeah, especially Tel Aviv. >> Yeah. >> EMEA real quick before you get into surge. It's interesting. The VC market in Europe is hot. They've got a lot of unicorns coming in. We've seen a lot of companies coming in. They're kind of rattling their own cage right now. Hey, look at us. Let's see if they crash, but we don't see that happening. I mean, people have been predicting a crash now in the startup ecosystem for at least a year. It's not crashing. In fact, funding's up. >> The pandemic was hard on a lot of startups for sure. >> Jeff: Yeah. >> But what we've seen is many of these startups, as quickly as they can grow, they can also pivot as well. And so I've actually seen many of our startups grow through the pandemic because their use cases are helping customers either save money, become more operationally efficient, and provide value to leadership teams that need more visibility into their infrastructure during a pandemic. >> It's an interesting point. I talked to Andy Jassy and Adam Selipsky both say the same thing during the pandemic. Necessity is the mother of all invention. And startups can move fast. So with that, you guys are there to assist. If I'm a startup and I got to pivot, 'cause remember iterate and pivot, iterate and pivot so you get your economics. That's the playbook of the ventures and the models. >> Yeah, exactly. >> How do you guys help me do that? Give an example, walk me through. Pretend me I'm a startup. Hey, I am on the cloud. Oh my God, pandemic. They need video conferencing. Hey CUBE, what do I need? Serge, what do I do? >> That's a good question. First thing is just listen. I think what we have to do is a really good job of listening to the partner. What are their needs? What is their problem statement and where do they want to go at the end of the day? And oftentimes because we've worked with so many successful startups that have come out of our program, we of either through intuition or a playbook determine what is going to be the best path forward and how do we get these partners to stop focusing on things that will eventually just be a waste of time and or not provide or bring any fruit to the table, which essentially revenue. >> Well, we love startups here in theCUBE because one, they have good stories, they're on cutting edge, always pushing the envelope, and they're kind of disrupting someone else. And so they usually have an opinion they don't mind sharing on camera. So love talking to startups. We love working with you guys on our Startup Showcase, awsstartups.com. Check out awsstartups.com and check out the showcases. Final word, I'll give you guys the last word. What's the bottom line, bumper sticker for the global APN program? Summarize the opportunity for startups, what you guys bring to the table and we'll close it out. Jeff, we'll start with you. >> Totally, yeah. I think the AWS Global Startup Program's here to help companies truly accelerate their business, full stop. And that's what we're here for. >> I love it. It's a good way to put it. >> Ditto? >> Yeah. >> All right. Serge, Jeff, thanks for coming on. >> Thanks John. >> Great to see you. Love working with you guys. Hey, startups need help and the growing and huge market opportunities, the shift cloud scale, data engineering, security, infrastructure, all the markets are exploding in growth because of the digital transformation of realities here, open source and cloud. All making it happen here in theCUBE in San Francisco, California. I'm John Furrier your host. Thanks for watching. >> Let's go, John. (soft music)

(upbeat music) >> Welcome to the BizOps Manifesto Power panel talking about, "Embracing Agility Across the Business." I'm Lisa Martin, there are three guests here with me today, to break down this topic. Pierre Viljoen, CTO at Global Head of Enterprise Technology and Governance at HTL Enterprise Studio. Hey Pierre, welcome. >> Thank you >> Lisa: Dave West is also here, the CEO of Scrum.org. Hey Dave, good to have you with us. >> Hi Lisa, hi everybody. >> Lisa: And Serge Lucio is here as well, the general manager of Broadcom's Enterprise Software Division. Hey Serge, good to have you on the program. >> Thank you, good to be here. >> So we're going to be talking about the people and the process and technology requirements that businesses need to adopt to be able to embrace agility across the business. We're going to also be talking a lot about this inaugural BizOps industry research survey, on the state of digital business. A lot of very interesting findings that we're going to go through in the next 20 minutes or so. So the first question guys is, the BizOps survey found that over 519 individuals over five countries business and technology executives. This survey found, most organizations still expect this year to be as challenging as last year. I want you to kind of walk us through why that is, and how is that going to impact digital transformation initiatives? Pierre, we'll start with you, then Dave, then Serge. >> Sure, thank you Lisa. So, I think these days, disruption is no longer an exception. It's kind of become the norm or the rule, in terms of how we operate. And as executives in companies have learned over the last year, with everything that's happened is that, you can only modernize to a point, and then you need to do a little bit more. And what really is needed is for us to understand, going forward, how we're actually going to remodel our business by harnessing the resources that we have in a much more agile way, in a more fluent way, from an organizational perspective. And I think our current midterm goal, probably, is that we're capable of remodeling how we can remove roadblocks. These kinds of roadblocks in the future, and get us in a better position where we are. I don't expect things to change dramatically over the next year. More in line with us making sure that we're more future proof in the way in which we're working. >> We still have remote workers, global uncertainty, the vaccine. Dave, what are your thoughts on the impact of this year on digital transformation initiatives? >> Yeah, it's funny. When I think of, sort of uncertainty and chaos, I think that COVID really started it rolling down a hill, but unfortunately it's literally like rolling down a hill, these chaos and complexity. It's getting faster and faster and harder and harder. We're talking about the new norm, right? What is the new normal? We just don't know. And I think the reality is that most organizations were surprised by the impact of COVID-19 and because of that, they responded very quickly. Many of them, people were working at home, they're looking at their supply chain, looking at localization, all sorts of really important things happen, but very quickly, not very strategically. I think the next few years we're going to see, hopefully, some of that realizing into strategy and actually starting to fundamentally change how the business is looking at the world. We've sort of entered the digital age, Lisa, this next age of innovation, we have moved out of mass production and the age of oil, into something very, very different. And I think those organizations, every organization out there is going to have to get a handle on that and COVID was the wake up, right? And I think the next five years are going to be very interesting. >> I agree with you that that accelerant was, I didn't think of it before as a big ball rolling downhill. And now I don't think I'm going to be able to get that out of my head. But Serge, talk to us about your thoughts, the impacts to digital transformation initiatives. >> Yeah, I think back to what Dave was describing. The big challenge is the uncertainty. Many organizations are faced with, currently, a lot of unknowns about, if and when things will go back to, quote unquote, some kind of normality. And with that kind of uncertainty, there's a lot of challenges to the planning from an investment point of view. So, Dave was talking about a short-term versus long-term Like, a lot of these organizations are basically focused on just getting by over the next 12 months and trying to figure out what needs to happen over the next 12 months. At the same time, there's a lot of challenges with respect to readiness and uncertainty. And so, in that context, you got kind of this tension between, "How much do I invest short term "on basically tactical initiatives, "How do I care about teams? "How do I enable these teams "to deliver in weeks as opposed to months? "And then at the same time, "how do I continue to invest "to fundamentally change my operating model?" And that tension is very real. Within many of the organizations we serve. >> One of the things that the survey found was that most of the respondents were very willing to embrace being more agile in order to be able to better respond to rapidly changing market conditions. But I want to get your opinion on what that actually really means, that willingness to embrace being agile. What does it really mean? And what do organize organizations have to do differently? Pierre, I'll start with you. >> Sure, I think we had a discussion a while back and Dave actually got into something interesting where he said, without quoting a famous sneaker brand, just go out and do it. I think that's probably the most important part of this. Most organizations are struggling to figure out, "How should we embrace Agile? "Should we jump in at full scale now? "Should we be looking Scrum? "Should we be doing Scrum-Falls? "Should we be falling over our own feet?" Nobody knows exactly, what might be the right sector? I think the most important part is to pick up a pair of solid principles that you're going to embrace, start executing on them, start learning as you go, and basically improve as you move forward. Over the last year, we've embraced digital product management quite a lot on our side. And it's had tremendous benefits without us, per se, aiming for those benefits at the end of the day. And these are things that you learn as you go. And if you're going to wait around, analysis paralysis is going to be the killer of Agile at the end of day >> "Just do it," I like that. Good advice. Dave, what are your thoughts? >> Yeah, so, I think that what's really interesting is, Agile has been around for 20 years. The manifesto was signed 20 years ago. Scrum came into the world 25 years ago. All of these sort of Agile approaches, but they were predominantly focused on technology. And I think that one thing that I've noticed, over and over again, is that the realization by C-level executives, level sevens, or whatever they're called, they've realized that it's not about technology. (chuckles) It's great, the technologies. I guess the technology's always worked in this complex world because customers never knew what they wanted. We didn't know how are we going to do it. We'd never worked together before. We didn't know how much it was going to cost. So, because of that (chuckles) we had to work in an agile way in technology. But ultimately, I think, one of the big differences going forward, is going to be that, there I say that intersection of business and technology, that BizOps kind of model that we talked about in the manifesto, and what the survey was really trying to tease out. I think that's really, really going to be interesting. And I don't know what that actually means, in terms of the execution. I hope it means that we're going to see teams better aligned to business outcomes. I hope it means that we're going to actually allow those teams that are actually know what they're approaching to make decisions. I hope it means that planning is going to be more directional rather than task level. I hope it means that we're going to start measuring the success in terms of business outcomes, not in terms of the work that we do. I hope it means all of these things. But we will wait and see, because experience would indicate that after a big disaster, lots of people tend to go back to exactly how they worked before, with that sort of emus kind of mentality or ostrich or whatever things sticks its head in the ground. I don't know. >> Sometimes we just want to go back to when things were safe and normal. But in terms of kind of following on, Dave, what you said, 94%, in this survey, 94% of respondents said we should adopt BizOps to increase competitiveness. So, that willingness is there in a vast majority of the respondents. So, I'd like to get your thoughts on what that willingness actually means and what they need to do differently. >> Yes. The problem is that, I think everybody understand that you have to be agile, right? You need to be able to respond quickly to your customer needs. You need to put the customer at the center of everything you do, right? So, conceptually, everybody understands that. The problem is really the operating model that many of these large organizations are dealing with to this day, right? So, you have these sort of Berkeley, kind of organized organization, with functional roles, specialized roles. And when you think about kind of generally, well, one of the big challenges is that you need to start to think horizontally, right? You need to start to start to think about what kind of value streams and what part of the cross functional teams that need to be organized and integrated to deliver on specific business outcomes. You need to start shifting from the traditional contract-based model that(indistinct) to a model which is much more based on trust, right? And we need to move away from vanity measurements and KPIs that many of the organizations typically lead by, to really focus on one thing and one thing only, which is that business value has been delivered. So, fundamentally, I think it requires a bit of a redesign of the operating model in these organizations. And one where, especially when you have a risk adversed kind of organizations, you need to start to be more accepting of risk, fundamentally. >> More accepting of risk. You brought something up there starts that I want to tackle in the next question with respect to culture. But one of the things that the survey uncovered was an interesting kind of seeming contradiction. The majority of respondents said, "We agreed, digital transformation "is about business outcomes "more than it is about technology." But 62% said, "We're still adopting technology for technology's sake." What does that actually mean? And what's the kind of cultural impact there for organizations to really get that more aligned on the digital transformation and the technology and the business outcomes? Pierre, we'll start with you. >> Sure. So, I think there were a number of reports this year talking about what's happened, what's not happened, and the majority of them focused on the fact that, as tech leaders, for years we've been praying to the gods to get budget approved to do all kinds of modernization activities to our infrastructure, our IT, tools, et cetera. And, lo and behold, the ball comes rolling down the hill, smashes a few things and we basically get some blank checks. So, we run around and we buy a whole bunch of stuff to modernize and to embrace this ability to do things differently. And in that whole process, what we basically did was buy more tools and buy more technology. And in that whole process, we didn't really embrace what it is that we're trying to achieve. So, basically aligning the technology to the actual business requirements getting closer to the customer, being able to understand where our market's moving, how we're capable of reducing the journey, if I can put it that way, and make sure that we're more aligned to where we need to be. So, although a lot of CIOs and CTOs got away with doing a lot of great stuff over the last year and users like me are like, "Ooh! I don't have to worry "about stupid VPNs and things anymore." That all went away. But in the same instance, I didn't really get anything that changed the organizational dynamic, which is a challenge. We still have the fundamental problems we have because the business leaders are not yet embracing the deep monitor of the processes that are supported by the technology. And then driving that in such a way that we can gain more business value which is important. To Serge's previous point, we're doing all these great things but we're not focusing on the incremental value that we're supposed to be getting. >> Dave, did it surprise you that there was this seemingly contradictory response? Yes, it's more about business outcomes and technology, but we're still adopting technology for technology's sake. What are your thoughts on that? And how can organizations actually start to move the needle on that? >> E-comm by cultural change, right? But you do know that your board and your leadership want you to do something, and the easiest thing you can do is buy something. I'm a sort of now an American, so, that's kind of my mantra in life, right? "When in doubt go shopping." Which is fantastic, just for the record. (Lisa laughs) But so, you've got to be seen to be doing something, whether it's replacing a VPN, which is always a fun thing to do, or whether it's getting on Slack. Everyone's going to be on Slack. that's going to help. But actually the core is that, exactly what Serge and Pierre have been saying all along, it's that, "Okay. So what is our business all about? "What are our customers? "what did they actually need? "What do our employees need? "How do we build a better value stream "from customer to the organization? "How do we align our teams to that? "How do we incentivize correctly "both the employees that are working "and our partners that are providing things "in this supply chain. How do we do all of those things?" Ultimately though, that means that we have to take a step back which is a very frustrating thing at the moment. And actually look at what is our business all about? What is the mission of it? Who are the customers? Take a moment to find what those are. And then, soon as we have that, and we don't have to do it. As Pierre said, we don't have to do it completely. We can do it incrementally. Organizations are very inward looking. That is the industrial mindset. That is that paradigm. It's looking, as Serge talked about, silos, "optimizing my department," "optimizing my budget, optimizing my kingdom." And what we're talking about is something that cross cuts all of that. So, the decision making is going to change around where the investments go and that's going to be really, really challenging. So, I'm not surprised, I'm not at all surprised that everybody says we should be doing this. And it's like classic. Everybody says we need to be fitter, but we're still all not fit. (Serge laughs) It's sort of, that's just the reality of the world that we live in, right? But we have to start making a stand. And the place we begin is customers. That's the place. And as soon as we start doing that, then everything else just becomes quite easy, actually. >> I like that. Focus on customers and it becomes easy. Serve, I'm kidding. What are your thoughts on this? >> Yeah, I think Dave summarized it well. It's very easy to just buy a tool or buy something, right? Fundamentally changing kind of an operating model is very difficult, but you need to fundamentally rethink for instance, all the responding initiatives. So, something as mundane as, You know, as a leader in my organization I have a budget, right? What's my incentive of collaborating with my peers in terms of delivering credible analysis form. And so, that to me kind of a fundamental shift that we need to operate, and that's probably one of the reasons why many of our largest organizations that we're serving are starting to introduce some new roles like a Chief Digital Officer, as kind of a way to kind of bring kind of a slightly different organization design. The challenge, though, is that, well, all of these teams are still kind of integrated with this fabric of these large systems which exist. So when we look at these value streams, in fact they're not independent from one another. You have a bunch of interdependencies. You are looking at kind of networks of these value streams. But the fundamental shift that we need to see is what we want these organizations to think about, ultimately with the part of the products or services that need to be focused on, all of these become kind of the primary things that we measure from point of view, and how do we align teams and projects and funding along these kinds of outcomes? >> So being customer focused, also being more broadly focused you mentioned the Chief Digital Officer role, which has an interesting role. It's supposed to look more, holistically, internally and externally. And we know that these organizations know we need to be better at this. like Dave's joke about we know we need to be more fit. But what's it going to take to actually create that collaboration, so that IT and business leaders are really working in lockstep and doing so in a timely fashion, so, that they're able to stay competitive. I do want to know from each of you, are you seeing examples of this already in progress? Pierre, let's start with you. >> I can only give you another example and say, one of the interesting things that we did was we try to embrace the delivery of services at HR in kind of a different frame this year, and kind of productize the services that we deliver. Now, if you're most people, you're trying to think about, "How do I set up things like communities "of practice and collaboration between people so "that they can work together on developing new services "new features, new products, et cetera." And we set out with creating this agile way of working. What we didn't anticipate, which was a very nice side effect, is that, because of COVID, because of the catalyst that it provided us, the remote working, people sense of ownership is inherently there. Meaning that self-organization of teams started happening. Nobody needed to crack a whip to get a bunch of guys to talk together with one another to figure out how to get stuff done. It's not like you could walk over to the water cooler and have a chat to Bob. Bob is a thousand miles away, or Bobby's sitting in another State. So, all of a sudden, all dynamic changed. And I have to say, people are a lot more resilient than what they're being given credit for. And if, as organizations, we embrace the culture in such a way and harness it in a positive way, we can actually get this movement to happen. And we actually can make the sum of the parts to be more than the whole. And this year we've seen that happen. And by no means, are we done 'cause we still have a lot of work to do, like Serge said, we have budgets, and budgets give you finite amount of movement left or right. Then you have to do what's best and possible within the frame that you're given. But I think embracing the cultural change and helping people to really excel at that and empowering them makes a huge difference in the way that you can get stuff done. >> Absolutely. Dave, what are your thoughts on this? >> I'm going to say something a little bit controversial, I think. I'm not a big fan of Chief Digital Officers. It just seems like we've got a problem. And some would argue that, "Well, if you've got a problem with somebody "you should get a coach" and all this stuff "and you get it sorted." And that's probably a good thing. But most digital officers, they're going to build a long-term career and create yet another stove pipe and that stove pipe's responsible for bringing all the other stove pipes together. It sounds a bit odd. If a digital officer is really there as a short term enabler, 'cause you asked IT and business leaders, trying to get them to work together better. The best business leaders (bell dings) know about IT, right? The best business leaders are IT sanctuary. Elon Musk or Jeff Bezos are great business leaders, but they know about technology, right? That's what brings them together. Technology is an asset and they may not be the most biggest expert in it, but they care deeply about learning about that stuff. So, I think the next few years we're going to see a lot of C-level and leaders in organizations become a lot more tech savvy, and maybe hire coaches to help them navigate. And the Chief Digital Officer will become more of a coach rather than a person that rolls out Slack or something, you know? (Pierre laughs) So, I think that is going to be the next big jump, really, when we realize that you don't get an additional thing. It's just part of what you do. >> Serge, agree, disagree? >> I agree. The reality is that it is happening, right? Don't get me wrong. We see that every day that so many States are highly integrated, organizations and teams are measuring business value, business outcomes. The problem is that it's oftentimes a very small subset of what these organizations are doing. And so, it's almost like the CEO is coming as kind of these new kind of, as Dave described. And it's got this new style organization which is really there to kind of scale what has been working with these organizations, but we're kind of creating this kind of almost shadow organization, as opposed to fundamentally rethinking and redesigning the organization and redesigning kind of the operating model. And so, we're kind of layering new stuff as opposed to fundamentally transforming. So, as long as it is just kind of just a step towards kind of a true transformation, I think that's fine. The challenge is to, again, create kind of a new set of silos, which are now called value streams, as opposed to young functional silos that we have today. >> So a lot of opportunities identified in this survey but there are still a lot of challenges there. So, I'd love to get you guys and our final question here in this panel to help us understand, from the BizOps coalition's perspective, how are you helping organizations to navigate these challenges, so, they can become successful, transform and actually become agile to respond to rapidly changing market conditions? Pierre, kick us off. >> Sure, from a coalition perspective, we're just trying to make sure that there's a set of sensible principles. That people can look at, can adopt that I think Dave mentioned it in another discussion, that give you that clarity of thought and mind in terms of what should you be thinking? How should you be thinking about it? What are the various aspects you need to consider? And then from that perspective, how do you implement these things in a sensible way for your organization? By no means is this this like, "Here are the 10 steps, you do them, and you're done." You'll be rich beyond your wildest dreams. It's not how it work. You're still going to have to work at it. You're still going to have to figure some stuff out. You're going to have to deep in yourself in your organizational policies, procedures, understand how the organization is actually working. You can't strap a V8 to Mini Cooper and expect to break the land speed record, without the wheels falling off, or something going wrong. So, you really need to harness that in a more sensible manner to move forward. And I think the coalition is on the right path to help organizations realize, "what is the sensible way to go?" "What are principles we can adopt "that we can abide by that will help us drive business "in a different way and close this chasm of disparity "between business and IT?" >> And Dave, your perspective on the BizOps coalition, helping organizations to sort through these challenges. >> Yeah. I'm going to share a little bit of a personal story. So, I must admit that I wasn't keen on the whole idea, and Serge sent me some stuff and he's like, "Could you just provide some feedback." And I did, and then there was a press release with my name on it. I saw, I was like, "Oh my God! "I better get involved because I don't want to "have my name associated "with something that doesn't make sense" But I've actually been surprisingly, I've actually found it a lot more positive than I thought because of exactly what Pierre's saying. So, basically, the coalition is a group of vendors, a bucolic of consultants, some pseudo thought leaders that think they are very thoughtful and maybe they're not, people like me. (Pierre laughs) And what we're doing though, is actually trying to get some clarity of terminology, get some clarity of, what are the principles? What are those key principles? How do they relate to each other? Get some, some synergy to allow, 'cause there's so much noise out there. And hopefully, this is going to say, "Okay, this is what BizOps is. "This is why it's important. "These are some simple things." And then hopefully, because of the breadth that Serge and others have managed to get in terms of membership, we're going to get all of those organizations to be consistently talking about these things, which will then create pressure on the market to actually start adopting these things in the way that we're proposing, or challenge those ideas and then make them better. So, I'm kind of excited about it, surprisingly, 'cause the last thing we need is yet another manifesto and group of people that spend their whole time talking about things and never getting anything done. But actually I think there might be some valuable stuff that comes out here and we're going to inspect and adapt to make sure it is valuable. And if it isn't, we will stop. (chuckles) (Lisa laughs) >> And Serge, strap us up with your thoughts and extending that value. >> Look, we started the BizOps manifesto really with kind of a very simple observation. Everybody's talking about the same stuff, right? But you have a value stream management church, the digital product management's church, the DevOps church, with Scrum church, the safe church. Right? But we're all saying the same thing. But we create so much confusion with our large enterprise customers, but it's just not a grain on a set of principles. And just saying like, look, fundamentally, we're all talking about the same thing. And there are process aspects, there are cultural aspects. There is what you measure. But fundamentally we agree on the same core set of principles. And so for me, the BizOps manifesto, first and foremost is to get the stakeholders from these different communities together, and recognize that, at the end of the day, we share the same values and create some clarity to the market as to how these pieces fit to one another. The second aspect, which is more from our point of view, as one of the vendors of tools, right? There's tons of tools out there. We talk a lot about kind of measuring business outcomes as a primary way to actually align to everybody in our organization. Well, today if you look at any of these organizations, on average, they use about 40 different tools on one of these value streams. None of that stuff integrates with one another. It's extremely difficult for an organization to be able to trace from an investment, all the way to stuff that delivers value and production to a customer. And so, one of my hopes for the coalition is that we start to actually provide some platform, data models, ontologies, to start to integrate those different tools to facilitate that kind of integration. So, those are kind of the two things which I think we can really help kind of develop and and improve on. >> Well, we know that there's a tremendous amount of folks out there that are wanting to embrace agility across the business, identifying areas where they need to do work. So, great advice from the three of you. Thank you so much for joining me on this power panel today and sharing what organizations can do to really embrace that agility across the organization. >> Thank you. >> Thank you. >> Thank you. >> Pierre Viljoen, Dave West and Serge Lucio. I'm Lisa Martin. Thanks for watching. (upbeat music)

>> From around the globe it's the CUBE with digital coverage of Devops Virtual Forum, brought to you by Broadcom. >> Continuing our conversations here at Broadcom's DevOps Virtual Forum. Lisa Martin here, please do welcome back to the program. Serge Lucio, the general manager of the Enterprise Software Division at Broadcom. Hey Serge welcome. >> Thank you. Good to be here. >> So I know you were just participating with the BizOps manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept but I wanted to get your thoughts on, spiritual co-location as really a necessity for BizOps to succeed in this unusual time in which we're living. What are your thoughts on spiritual co-location in terms of cultural change versus adoption of technologies? >> Yeah, it's quite interesting, right. When we think about the major impediments for DevOps implementation, that means all about culture, right? And swore over the last 20 years we've been talking about silos. We'd be talking about the paradox for these teams too when it goes to align. And in many ways it's not so much about these teams aligning but about being in the same car, in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me this is really about kind of changing this culture where people start to look at kind of OKRs instead of the key objective that drives the entire team. Now, what it means in practice is really that's we need to change a lot of behaviors, right? It's not about the ER key, it's not about roles. It's about, you know, who can do what and when, and, you know, driving a bias towards action. It's also means that we need, I mean, especially in this COVID times it becomes very difficult, right? To drive kind of a kind of collaboration and affinity between these teams. And so I think there's a significant role that especially tools can play in terms of providing this conference feedback from teams to be in that preface spiritual qualification. >> Well, and it talked about culture being it's something that, you know, we're so used to talking about DevOps with respect to velocity, all about speed here. But of course this time everything changed so quickly but going from the physical spaces to everybody being remote really does take. It's very different than you can't replicate it digitally but there are collaboration tools that can kind of really be essential to help that cultural shift, right? >> Yeah, so to me we tend to talk about collaboration in a very mundane way, right? Of course we can use zoom. We can all get into the same room. But the point when I think when Jeff says spiritual co-location, it's really about, we all share the same objective. Do we have a means for instance, our pipeline, right? When you talk about DevOps probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team. But just thinking about a pipeline, right? At the end of the day, it's all about what is the meantime to feed back to these teams. If I'm a developer and I commit code, how long does it take for, you know, that code to be processed through pipeline or quick and I get feedback? If I am a finance person, who's funding a product or a project, what is my meantime to beat back? And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of that door effect metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration it's really a balance. How do you provide that feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when checking code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allocation, how can the system help me reduce my change builder rate? Because it's able to predict that some issue was introduced in the application or the product. So I think there's a great role of technology and AI candidates to actually provide kind of that new level of collaboration. >> So we'll get to AI in a second but I'm curious, what are some of the metrics you think that really matter right now is organizations are still in some form probably of transformation to this new almost 100% remote workforce. >> So I'll just say first I'm not a big fan of metrics. And the reason being that, you know, you can look at a change failure rate, right, or a leak time or cycle time. And those are interesting metrics, right? The trend on metric is absolutely critical. But what's more important these I'll do get to the root cause. What is taught to you lead to that metric to degrade or improve over time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll use a very mundane example. You know, we know that cycle time is heavily influenced by organizational boundaries. So, you know, we talk a lot about silos, but we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are pretty, you know, valid set metrics but what's way more important is to understand, what are the anti parents? What are the things that we can detect through the data that actually are affecting those metrics? And I mean, over the last 10, 20 years, we've learned a lot about kind of what are the anti parents within our large enterprise customers? And there are plenty of them. >> What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >> So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle where DevOps is more and more embracing value stream management, value stream mapping. And I think what's interesting is that in many ways, the product is becoming the new silo. The notion of a product is very difficult by itself to actually define. People are starting to recognize that a value stream is not its own little kind of island. That in reality, when I did find a product this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So on that and there is clearly kind of a new sets if you will of anti-patterns where, you know, products are being defined as a set of OTRs. They have interdependencies and you have to have a new set of silos. On the other hand the other kind of key movement to ease around the SRE space, where I think there is a cultural clash. While the DevOps side is very much embracing this notion of OTRs and value stream mapping and value management. On the other end, you have IT operations teams. We still think business services, right? For them they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so there is this boundary where I think, well, SRE has been put in place, and there's lots of thinking about what kind of metrics can be defined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for, you know, true operations teams. >> And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You now talked about AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate and maybe see eye to eye with the SREs? >> Yeah, so there're kind of too, so even for myself, right? As a leader of , you know, 1500 people organization, there's a number of things I don't see, right, on a daily basis. And I think the technologies that we have at our disposal today from the AI are able to mine a lot of data and expose a lot of issues that as leaders we may not be aware of. And some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you start to understand, for instance, what is the is a work in progress, right, during the sprint? When you start to analyze the data you can detect for instance, that maybe the teams are over committed, that there is too much work in profits. You can start to identify kind of interprocess either from a technology or from a people point of view, which were hidden. You can start to understand that maybe the change failure rate is dragging. So I believe that there is a fundamental role to be played by the tools to expose again these anti parents. To make these things visible to the teams to be able to even compare teams, right? One of the things that's amazing is now we have access to tons of data not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate and what's working, what's not working. >> Thoughts on AI and automation as a facilitator of spiritual co-location? >> Yeah, absolutely. It's, you know, there's a the problem we all face is the unknown, right? The velocity, the volume, variety of the data, every day we don't really necessarily completely appreciate what is the impact of our actions, right? And so AI can really act as a safety net that enables us to understand what is the impact of our actions. And so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data and collaborate on the data in the actual matter, I think is a very powerful enabler on, in that respect. I mean, I've seen countless of times that for instance at the SRE boundary to basically show that we'll turn the quality attributes of an incoming release, right? And exposing that to an operations person, an SRE person and enabling that collaboration dialogue through there is a very, very powerful tool. >> Do you have any recommendations for how teams can use, you know, the SRE folks, the DevOps says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be non-productive. >> Yeah, so to me there's a part that the question really is when we talk about data. There are different ways you can use data, right? So you can do a lot of analytics, predictive analytics. So I think there is a tendency to look at, let's say a specific KPI, like an availability KPI or change failure rate. And to basically do a regression analysis and projecting all these things is going to happen in the future. To me that's a bad approach. The reason why I fundamentally think it's a better approach is because we, our systems the way we develop software is a non-leader kind of system, right? Software development is not linear in nature. And so I think there's a, this is probably the worst approach is to actually focus on metrics. On the other hand if you start to actually understand at a more granular level, what are the things which are contributing to this, right? So if you start to understand, for instance that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we have, I've actually a customer who identified that over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed this resulted in this implant outages. So if you start to be able to basically establish causality, right? Cause an effect between kind of data across the last cycle. I think this is the right way to use AI. And so pharma to be, I think it's way more about kind of a classification problem. What are the causes of problems that do exist and affect things as opposed to an hourly predictive which I don't think is as powerful? >> So I mentioned in the beginning of our conversation that just came off the BizOps manifesto. You're one of the authors of that. I want to get your thoughts on DevOps and BizOps overlapping, complimenting each other. What, from the BizOps perspective, what does it mean to the future of DevOps? >> Yeah, so it's interesting, right? If you think about DevOps, there's no founding document, right? We can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways there is no clear definition of what DevOps is. If you go to the DevOps Institute today you'll see that, you know, they are specific trainings for instance on value management on SRE. And so in many ways, the problem we have as an industry is that there are set practices between agile, DevOps, SRE, value stream management, Ital, right? And we all basically talk about the same things, right? We all talk about essentially accelerating in the meantime to feedback, but yet we don't have a common framework to talk about that. The other key thing is that we add to wait for genius, Jean Kim's last book to really start to get into the business aspect, right? And for value mapping to start to emerge for us to start as an industry, right? IT to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about kind of driving these business outcomes. And so to me, BizOps is really about kind of putting a lens on kind of this critical element that it's not business and IT that we in fact need to fuse business and IT. That I need needs to transform itself to recognize that it's this value generator, right? It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this over all kind of framework, if you will. That set the context for what is the reason for IT to exist. What are the core values and principles that IT needs to embrace to, again, change from cost center to value center. And then we need to start to use this as a way to start to unify some of, again, the core practices, whether it's agile, DevOps, value stream mapping, SRE. So, I think over time, my hope is that we start to organize a lot of our practices, language and cultural elements. >> Last question Serge in the last few seconds we have here, talking about this, the relation between BizOps and DevOps. What do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >> So to me the key challenge for the industry is really around. So we were seeing a very rapid shift towards kind of project to product, right? Which we don't want to do is to recreate kind of these new silos, these hard silos. So that's one of the big changes that I think we need to be really careful about. Because it is ultimately, it is about culture. It's not about kind of how we segment the work, right? And any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of kind of the spiritual co-location, I think it's really about that too. It's really about kind of focusing on the business outcomes on kind of aligning, on driving engagement across the teams, but not for create kind of a new set of silos which instead of being vertical are going to be these horizontal products. >> Great advice Serge that looking at culture as kind of a way of really addressing and helping to reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps Virtual Forum. >> Thank you. Thanks for your time. Serge Lucio, Lisa Martin, we'll be right back. (upbeat music)

>> Announcer: From around the globe, it's theCUBE with digital coverage of DevOps virtual forum. Brought to you by Broadcom. >> Hi guys, welcome back. So we have discussed the current state and the near future state of DevOps and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where DevOps needs to go. In order to be successful next year. So our guests today are you've seen them all before. Jeffrey Hammond is here the VP and Principal Analyst serving CIO at Forrester. We've also got Serge Lucio, the GM of Broadcom Enterprise Software Division. And Glyn Martin, the head of QA Transformation at BT. Guys welcome back. Great to have you all three together. >> Hi Lisa. (Serge speaks faintly) >> Good to be here. >> All right. So we're all very socially distanced as we talked about before. Great to have this conversation. So let's start with one of the topics that we kicked off the forum with. Jeff, we're going to start with you spiritual colocation. That's a really interesting topic that we've uncovered. But how much of the challenge is truly cultural? And what can we solve through technology? Jeff, we'll start with you, then Serge, then Glyn, Jeff take it away. >> Yeah I think fundamentally, you can have all the technology in the world. And if you don't make the right investments in the cultural practices in your development organization. You still won't be effective. Almost 10 years ago, I wrote a piece. Where I did a bunch of research around what made high performance teams software delivery teams high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the Agile Manifesto. Let's take that today. Where developers are on their own in their own offices. if you've got teams where the team itself had a high level of autonomy. And they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. And so what we have seen is that organizations that embraced autonomy, and got their teams in the right place. And their teams had the information that they needed to make the right decisions. Have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like well, how do we actually push the software that we've created into production that have become the challenge is not. Are we writing the right software? And that's why I think the term spiritual colocation is so important. Because even though we may be physically distant, we're on the same plane, we're connected from a shared purpose. There's a Surgeon I worked together a long, long time ago, so just it's been what, almost 15-16 years, since we worked at the same place. And yet I would say there's probably still a certain level of spiritual colocation, between us. because of this shared purposes that we've had in the past and what we've seen in the industry, and that's a really powerful tool to build on. So what do tools play as part of that, to the extent that tools make information available to build shared purpose on. To the extent that they enable communication so that we can build that spiritual colocation. To the extent that they reinforce the culture that we want to put in place. They can be incredibly valuable, especially when we don't have the luxury of physical colocation. Hope that makes sense.(chuckles) >> It does. I should have introduced this last segment as we're all spiritually colocated. All right. So Serge, clearly you're still spiritually colocated with Jeff. Talk to me about what your thoughts are about spiritual of colocation. The cultural impact and how technology can move it forward? >> Yes, so I think, while I'm going to sound very similar to Jeff in that respect. I think it starts with kind of shared purpose, and understanding how individuals teams contribute to kind of a business outcome. What is our shared goals our shared vision with what is it we're trying to achieve collectively. And keeping kind of the line to that. And so it really starts with it Now, the big challenge always is over the last 20 years, especially in large organization has been specialization of roles and functions. And so we all have started to basically measure which we do on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome. Or is it on purpose. We kind of revert that to Okay, what is my database uptime? What is my cycle time? Right. And I think which we can do or where we really should be focused as an industry is to start to basically provide a lens for these different stakeholders to look at what they're doing. In the context of benefiting this business outcomes. So, probably one of my theories experience was to actually witness at one of our large financial institution. Two stakeholders across development and operations staring at the same data. Like which was related to economy changes, test execution results, coverage, official liabilities, and all the overran direction of incidents. And when you start to put these things in context, and represent that in a way that these different stakeholders can look at from their different lens. And they can start to basically communicate, and understand how they jointly or complement to do that kind of common vision or objective. >> And Glyn, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part of technology impact? >> Yeah, I mean I agree with Jeffrey that, you know, the people and culture are the most important thing. Actually, that's why it's really important when you're transforming to have partners who have the same vision as you. Who you can work with have the same end goal in mind. And we would constantly found that with our continuing relationship with Broadcom. What it also does, are those tools can accelerate what you're doing and can drive consistency. You know, we've seen within simplify, which is BT's Flagship Transformation Program, where we're trying to as it says, simplify the number of system stacks that we have. The number of products that we have, actually at the moment we've got different value streams within that program. Who have got organizational silos who are trying to rewrite the wheel. Who are still doing things manually. So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT. Which is such a complex and very different environment, depending on what area BT you're in. Whether it's consumer, whether it's a mobile area, whether it's large global or government organizations. We found that we need tools that can drive that consistency. But also flex to Greenfield Brownfield kind of technologies as well. So it's really important that as it's a from a number of different aspects. That you have the right partner, and to drive the right culture here, and the same vision, but also who have the tool sets to help you accelerate, They can't do that on their own. But they can help accelerate what it is you're trying to do. And a really good example of that is we're trying to shift left, which is probably a quite a bit of a buzz phrase. And they're kind of testing well at the moment. But I could talk about things like Continuous Delivery Director to Broadcom tools. And it has many different features to it, but very simply on its own. It allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams around could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of Continuous Delivery Director. But it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >> Thanks, Glyn So we kind of sum that up autonomy, collaboration tools that facilitate that. So let's talk now about metrics. From your perspective, what are the metrics that matter Jeff? >> Well, I'm going to go right back to what Glyn said about data that provides visibility that enables us to to make decisions with shared purpose. And so business value has to be one of the first things that we looking at. How do we assess whether we have built something that is valuable? That could be sales revenue, it could be Net Promoter Score, if you're not selling what you've built, it could even be what the level of reuse is within your organization. Or other teams picking up the services that you've created. One of the things that I've begun to see organizations do is to align value streams with customer journeys. And then to align teams with those value streams. So that's one of the ways that you get to a shared purpose. 'Cause we're all trying to deliver around that customer journey. The value associated with it. And we're all measured on that. There are flow metrics, which are really important. How long does it take us to get a new feature out. From the time that we conceive it to the time that we can run our first experiments with it. There are quality metrics, some of the classics or maybe things like defect density or meantime to response. One of my favorites came from a company called Ultimate Software. Where they looked at the ratio of defects found in Production defects found in pre production. And their developers were in fact measured on that ratio and told them that guess what quality is your job too. Not just the test departments group. The fourth level that I think is really important in the current situation that we're in, is the level of engagement in your development organization. We used to joke that we measured this with the parking lot metric. How how full was the parking lot at 9, and how full was it at 5 o'clock. I can't do that anymore, since we're not physically colocated. But what you can do is you can look at how folks are delivering. You can look at your metrics in your SCCM environment, you can look at the relative rates of churn, you can look at things like well are our developers delivering during longer periods. Earlier in the morning, later in the evening? Are they delivering on the weekends as well. Are those signs that we might be heading toward burnout, because folks are still running at sprint levels instead of marathon levels. So all of those in combination, business value, flow, engagement and quality. I think form the backbone of any sort of metrics program. The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Unfortunately I see organizations where they weaponize the data. And that's completely the wrong way to look at it. What you need to do is you need to say. "How is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing? And then what do we do to remove those blockers to make sure that we're giving these autonomous teams, the context that they need to do their job in a way that creates the most value for the customers?" >> Great advice, Jeff. Glyn over to you metrics that matter to you that really make a big impact. And also how do you measure quality kind of following on to the advice that Jeff provided? >> I mean, Jeff provided some great advice. Actually, he talks about value, he talks about flow, both of those things are very much on my mind at the moment. But there was a time, listen to a speaker called Mia Kirsten, a couple of months ago, he talked very much around how important flow management is. And remove and using that to remove waste, to understand in terms of, making software changes. What is it that's causing us to do it longer than we need to? So where are those areas where it takes too long. So I think that's a very important thing. For us, it's even more basic than that at the moment. We're on a journey from moving from waterfall to agile. And the problem with moving from waterfall to agile is, with waterfall, the the business had a kind of comfort that everything was tested together, and therefore it's safer. And with agile, there's that kind of how do we make sure that you know, if we're doing things quick, and we're getting stuff out the door that we give that confidence, that that's ready to go? Or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence. And some of the metrics around that and how healthy those releases are and actually saying we spend a lot of money, in an investment setting up agile teams training agile teams. Are we actually seeing them deliver more quickly? And are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment. But I think it's also about, generally bringing it all together DevOps. We've got the kind of value ops, AI Ops. How do we actually bring that together to so we can make quick decisions, and making sure that we are delivering the biggest bang for our partners. >> Absolutely biggest bang for the partners. Serge your thoughts. >> Yes I think we all agree, right? It starts with business metrics, flow metrics. These are one of the most important metrics and ultimately, I mean, one of the things that's very common across I highly functional teams is engagements, right? When you see a team that's highly functional, and that's agile, that practices DevOps everyday. They are highly engaged. That definitely true. Now back to you, I think, Jeff's points on weaponization of metrics. One of the key challenges we see is that organizations traditionally have been kind of, setting up benchmarks. Right. So what is a good cycle time? What is a good mean time? What is a good mean time to repair? The problem is that this is very contextual, right? It's going to vary quite a bit, depending on the nature of application and system. And so one of the things that we really need to evolve as an industry. Is to understand that it's not so much about those flow metrics is about are these flow metrics ultimately contribute to the business metric. To the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood, is that when you have a bad cycle time or what you perceive as being a bad cycle time or bad quality. The problem is oftentimes like, how do you go and explore why, right? What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics. And not on the I type patterns, which are pretty common across the industry. If you look at for instance things like, lead time for instance. It's very common that organizational boundaries are going to be a key contributor to bad lead time. And so I think that there is reviewing the metrics, there is I think a lot of work that we need to do in terms of classifying this untied PaaS. Back to you, Jeff, I think you're one of the cool offers of Water-Scrum Fall as a key pattern in the industry or anti-patterns. >> Yeah >> But Water Scrum Fall, right. Is the key one right? And you will detect that through kind of a defect rival rates. That's right, that looks like an S curve. And so I think it's the output of the metrics is what do you do with those metrics. >> Right. I'll tell you Serge, one of the things that is really interesting to me in that space is. I think those of us had been in industry for a long time, we know the anti patterns, 'cause we've seen them in our career,(laughs) maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti patterns based on the telemetry that comes in. I think it would be a really interesting place to apply machine learning and reinforcement learning techniques. So hopefully something that we'd see in the future with DevOps tools. 'Cause as a manager that maybe only a 10 year veteran or a 15 year veteran. You may be seeing these anti patterns for the first time, and it would sure be nice to know what to do when they start to pop up.(chuckles) >> That would right? Insight, always helpful. All right guys, I would like to get your final thoughts on the fit one thing that you believe our audience really needs to be on the lookout for. and to put on our agendas. For the next 12 months. Jeff will be back to you. >> I would say, look for the opportunities that this disruption presents. And there are a couple that I see. First of all, as we shift to remote central working, we're unlocking new pools of talent. Where it's possible to implement more geographic diversity. So look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low code tools. To very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers. That agile and DevOps are the peanut butter and chocolate to support creative workers with algorithmic capabilities. >> Peanut butter and chocolate. Glyn where do we go from there? What's the one silver bullet that you think that needs to be on the look out for? >> (indistinct) out I certainly agree that low code is next year, we'll see much more low code. We've already started going moving towards more of a SaaS based world but low code also. I think as well for me, we've still got one foot in the kind of cloud camp. We'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last thing for me is, how do you really instill quality throughout the kind of the life cycle When I heard the word scrum for it kind of made me shut it. 'Cause I know that's a problem. That's where we're at with some of our things at the moment. So we need to get beyond that we need to be releasing changes more frequently into production. And actually being a bit more brave and having the confidence to actually do more testing in production and going straight to production itself. So expect to see much more of that next year. Yeah, thank you. I haven't got any food analogies unfortunately. (laughs) >> We all need some peanut butter and chocolate. All right Serge, Just take us on that sir. What's that nugget you think everyone needs to have on their agendas? >> That's interesting, right? So a couple of days ago, we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about velocity, right? It's all about we still are perceiving DevOps as being all about speed. And so to me the key advice is, in order to create kind of this spiritual colocation in order to foster engagement. We have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams. To understand how they're delivering value into allowing everything they do from a metrics to delivery to flow to those metrics. And only with data, I think, are we going to be able to actually start to to restart to align kind of all these roles across the organizations and drive not just speed, but business outcomes. >> All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me. Today and our guests, I think this was an incredibly valuable, fruitful conversation. And we appreciate all of you taking the time to spiritually colocate with us today. Guys, thank you. >> Thank you Lisa. >> Thank you. >> Thank you. >> For Jeff Hammond, Serge Lucio and Glyn Martin. I'm Lisa Martin. Thank you for watching the Broadcom DevOps virtual forum. (upbeat music)

>>from around the globe. It's the Cube with digital coverage of biz ops Manifesto unveiled Brought to you by Biz Ops Coalition >>Hey, welcome back, everybody. Jeffrey here with the Q. Come to you from our Palo Alto studios today for a big big reveal. We're excited to be here. It's the biz. Opps manifesto, unveiling things been in the works for a while and we're excited. Have our next guest one of the really the powers behind this whole effort. And he's joining us from Boston. It's surge Lucio, the vice president and general manager Enterprise software division that Broadcom Serge, Great to see you. >>Good to see. Oh, absolutely. So you've been >>in this business for a very long time? You've seen a lot of changes in technology. What is the biz Ops manifesto? What is this coalition all about? Why do we need this today in in 2020? >>Yeah, so? So I've been in this business for close to 25 years, writes about 25 years ago, the agile manifesto was created, and the goal of the actual manifesto was was really to address the uncertainty around software development and the inability to predict the effort to build software. And if you if you roll that kind of 20 years later and if you look at the current state of the industry, the Product Project Management Institute estimates that we're wasting about a million dollars every 20 seconds in digital transformation initiatives that do not deliver on business results. In fact, we we recently served, uh, the number of executives in partnership with Harvard Business Review and 77% off. Those executives think that one of the key challenges that they have is really at the collaboration between business and I t. And that that's been kind of a case for almost 20 years now. Eso the key challenge we're faced with is really that we need a new approach. And many of the players in the industry, including ourselves, have been using different terms. Right? Some are. We are talking about value stream management. Some are talking about software delivery management. If you look at the site reliability engineering movement, in many ways it embodies a lot of this kind of concepts and principles. So we believe that it became really imperative for us to crystallize around kind of one concept and so In many ways, the Bezos concept and the bazaars manifesto are out, bringing together a number of ideas which have been emerging in the last five years or so and defining the key values and principles to finally helped these organizations truly transform and become digital businesses. And so the hope is that by joining our forces and defining kind of key principles and values, we can help kind of the industry not just by, you know, providing them with support, but also tools and consulting that is required for them to truly achieve that kind of transformation, that everybody's >>right, right? So co vid Now we're six months into it, approximately seven months into it. Um, a lot of pain, a lot of bad stuff still happening. We've got a ways to go. But one of the things that on the positive side, right and you've seen all the memes and social media is a driver of digital transformation and a driver of change. Because we have this light switch moment in the middle of March and there was no more planning, there was no more conversation. You suddenly got remote. Workforce is everybody's working from home, and you gotta go, right, So the reliance on these tools increases dramatically. But I'm curious, you know, kind of short of the beginnings of this effort and short of kind of covert, which, you know, came along unexpectedly. I mean, what were those inhibitors? Because we've been making software for a very long time. Write the software development community has has adopted kind of rapid change and and iterative delivery and and sprints what was holding back the connection with the business side to make sure that those investments were properly aligned with outcomes. >>Well, so So that you have to understand that I ts is kind of its own silos. And traditionally it has been treated as a cost center within large organizations and not as a value center. And so as a result, kind of the traditional dynamic between the I t. And the business is basically one of a kind of supplier to to kind of a business on. Do you know if you could go back? Thio? I think Elon Musk a few years ago, basically, at these concepts, off the machines to build the machines and you went as far as saying that the machines or the production line is actually the product, so meaning that the core of the innovation is really about building kind of the engine to deliver on the value. And so, in many ways, way have missed on this shift from, um, kind of I t becoming this kind of value center within the enterprises and any told about culture now culture is is the sum total of behaviors and the realities that if you look at the i t, especially in the last decade with the agile with develops with hybrid infrastructures, it's it's way more volatile today than it was 10 years ago. And so the when you start to look at the velocity of the data, the volume of data, the variety of data to analyze kind of the system, um, it's very challenging for I t. To actually even understand and optimize its own processes, let alone to actually include business as kind of an integral part of kind of a delivery chain. And so it's both kind of a combination off culture which is required a za well as tools, right to be able to start to bring together all these data together and then given the volume variety velocity of the data. We have to apply some core technologies which have only really, truly emerging last 5 to 10 years around machine learning and knowledge. And so it's really kind of a combination of those freaks which are coming together today. Truly, organizations get to the next level, >>right? Right. So let's talk about the manifesto. Let's talk about the coalition, the Biz Ops Coalition. I just like that you put down these really simple you know, kind of straightforward core values. You guys have four core values that you're highlighting, you know, business outcomes over individual projects and outputs, trust and collaboration over side load teams and organizations, data driven decisions. What you just talked about, you know, over opinions and judgment on learned, responded Pivot. I mean, surgery sounds like pretty basic stuff, right? I mean, aren't isn't everyone working to these values already? And I think you touched on it on culture, right? Trust and collaboration, data driven decisions. I mean, these air fundamental ways that people must run their business today or the person that's across the street that's doing it is gonna knock him right off the block. >>Yeah, so that's very true. But so I'll mention the novel survey. We need, uh, think about six months ago and twist in partnership with an industry analyst, and we serve it again. The number of 80 executives to understand how many were tracking business outcomes somebody you have, the software executives I T executives were tracking business outcomes, and the there were. Less than 15% of these executives were actually tracking the outcomes of the software delivery. And you see that every day, right? So in my own teams, for instance, we've bean adopting a lot of these core principles in the last year or so, and we've uncovered that 16% of our resource is we're basically aligned around initiatives which were not strategic for us. I take, you know, another example. For instance, one of our customers in the airline industry uncovered, for instance, that a number of that they had software issues that led to people searching for flights and not returning any kind of availability. And yet, you know, the I T teams whether its operations software involvement were completely oblivious to that because they were completely blindsided to it. And so the connectivity between the in words metrics that Turkey is using, whether it's database I, time cycle, time or whatever metric we use in I t are typically completely divorced from the business metrics. And so at its core, it's really about starting to align the business metrics with with the the software delivered change. Right, this, uh, this system, which is really a core differentiator for these organizations. It's about connecting those two things and and starting Thio infuse some of the actual culture and principles. Um, that's emerged from the software side into the business side. Of course, the lien movement and over movements have started to change some of these dynamics on the business side. And and so I think this thesis is the moment where we were starting to see kind of the imperative to transform. Now Cuvee the obviously has been a key driver for that. The the technology is right to start to be able to leave data together and really kind of also the cultural shifts through agile fruit develops through the SRE movement, fueling business transformation. All of these things are coming together and that are really creating kind of conditions. For the Bezos Manifesto to exist. >>So, uh, Clayton Christensen, great hard professor innovator's dilemma might still my all time favorite business books, you know, talks about how difficult it is for in comments to react to to disruptive change, right, because they're always working on incremental change because that's what their customers are asking for. And there's a good our ally when you talk about, you know, companies not measuring the right thing. I mean, clearly, I t has some portion of their budget that has to go to keeping the lights on, right, that that's always the case. But hopefully that's a an ever decreasing percentage of their total activity. So, you know what should people be measuring? I mean, what are kind of the new metrics? Um, in biz ops that drive people to be looking at the right things, measuring the right things and subsequently making the right decisions investment decisions on whether they should do, you know, move Project a along or Project B. >>So there are really two things, right? So So I think what you are talking about this portfolio management, investment management, right and which, which is a key challenge, right in my own experience, right driving strategy or large scale kind of software organization for years. It's very difficult to even get kind of a base data as to who is doing what. Uh, I mean, some of our largest customers were engaged with right now are simply trying to get a very simple answer, which is how many people do I have, and that specific initiative at any point in time and just tracking that information is extremely difficult. So and and again, back to Product Project Management Institute, they have estimated that on average, I two organizations have anywhere between 10 to 20% of their resource is focused on initiatives which are not strategically aligned. So so that's one dimensional portfolio management. I think the key aspect, though that we are we're really keen on is really around kind of the alignment of the business metrics to the ICTY metrics eso I'll use kind of two simple examples, right and my background is around quality and I have always believed that fitness for purpose is really kind of a key, um, a philosophy, if you will. And so if you start to think about quality is fitness for purpose, you start to look at it from a customer point of view, right? And fitness for purpose for core banking application or mobile application are different, right? So the definition of a business value that you're trying to achieve is different on DSO the And yet if you look at our I t operations are operating there are using kind of the same type of kind of inward metrics like a database off time or a cycle time or what is my point? Velocity, right? And s o the challenge really is this inward facing metrics that the I t. Is using which are divorced from ultimately the outcome. And so, you know, if I'm if I'm trying to build a poor banking application, my core metric is likely going to be up time, right? If I'm if I'm trying to build a mobile application or maybe a social mobile app, it's probably going to be engagement. And so what you want is for everybody across I t to look at these metric and what part of the metrics withing the software delivery chain which ultimately contribute to that business metric in some cases, cycle time, maybe completely relevant. Right again. My core banking up. Maybe I don't care about cycle time. And so it's really about aligning those metrics and be able to start to differentiate. Um, the key challenge you mentioned around the around the disruption that we see is or the investors is. Dilemma now is really around the fact that many idea organizations are essentially applying the same approaches for innovation right for basically scrap work, Then they would apply to kind of over more traditional projects. And so, you know, there's been a lot of talk about to speed I t. And yes, it exists. But in reality are are really organizations truly differentiating out of the operate their their projects and products based on the outcomes that they're trying to achieve? And and this is really where bizarre is trying to affect. >>I love that. You know, again, it doesn't seem like brain surgery, but focus on the outcomes right and and it's horses for courses. As you said this project, you know what you're measuring and how you define success isn't necessarily the same as it is on this other project. So let's talk about some of the principles we talked about the values, but you know I think it's interesting that that that the bishops coalition, you know, just basically took the time to write these things down, and they don't seem all that super insightful. But I guess you just got to get him down and have them on paper and have it in front of your face. But I want to talk about, you know, one of the key ones which you just talked about, which is changing requirements right and working in a dynamic situation, which is really what's driven. You know this, the software to change and software development because, you know, if you're in a game app and your competitors comes out with a new blue sword, you've got to come out with a new blue swords. So whether you have that on your compound wall, we're not. So it's It's really this embracing of the speed of change and and and making that you know the rule, not the exception. I think that's a phenomenon. And the other one you talked about his data right and that today's organizations generate more data than humans can process. So informed decisions must be generated by machine learning and ai and you know and the big data thing with a dupe you know, started years ago. But we are seeing more and more that people are finally figuring it out that it's not just big data on It's not even generic machine learning artificial intelligence. But it's applying those particular data sets and that particular types of algorithms to a specific problem to your point, to try to actually reach an objective. Whether that's, you know, increasing the your average ticket or, you know, increasing your check out rate with with with shopping carts that don't get left behind and these types of things. So it's a really different way to think about the world in the good old days, probably when you got started, when we had big Giant you know, M R D s and P R. D s and sat down and coded for two years and and came out with a product release and hopefully not too many patches subsequently to that. Yeah, >>it's interesting right again, back to one of these service that we did with about 600 the ICTY executives and we we purposely designed those questions to be pretty open. Andi and one of them was really wrong requirements, and it was really around. Kind of. What is the best approach? What is your preferred approach towards requirements? And if I remember correctly, Over 80% of the ICTY executives said that the best approach their preferred approach is for requires to be completely defined before self for the bombing starts, let me pause there. We're 20 years after the agile manifesto, right, and for 80% of these idea executives to basically claimed that the best approach is for requires to be fully baked before solved before software development starts basically shows that we still have a very major issue again. Our apotheosis in working with many organizations is that the key challenges really the boundary between business and I t. Which is still very much contract based. If you look at the business side, they basically are expecting for I t deliver on time on budget, Right? But what is the incentive for I t to actually deliver on the business outcomes, right? How often is I t measured on the business outcomes and not on S L. A or on a budget secretary, and so that that's really the fundamental shift that we need to. We really need to drive up to send industry andi way. Talk about kind of this dis imperative for organizations to operate. That's one. And back to the, you know, various doors still, Um, no. The key difference between these large organization is really kind of a. If you look at the amount of capital investment that they can put into pretty much anything, why are they losing compared Thio? You know, startups. What? Why is it that more than 40% off personal loans today are issued not by your traditional brick and mortar banks, but by start ups? Well, the reason, Yes, it's the traditional culture of doing incremental changes, not disrupting ourselves, which Christenson covered at length. But it's also the inability to really fundamentally change kind of dynamic between business I t and partner, right, to to deliver on a specific business. All >>right, I love that. That's a great That's a great summary and in fact, getting ready for this interview. I saw you mentioning another thing where you know the problem with the agile development is that you're actually now getting mawr silos because you have all these autonomous people working you know, kind of independently. So it's even harder challenge for for the business leaders toe, as you said to know what's actually going on. But But, sir, I want to close um, and talk about the coalition eso clearly These are all great concepts, these air concepts. You want to apply to your business every day. Why the coalition? Why, you know, take these concepts out to a broader audience, including your competition and the broader industry to say, Hey, we as a group need to put a stamp of approval on these concepts. These values these principles. It's >>so first, I think we we want everybody to realize that we are all talking about the same things, the same concepts e think we were all from our own different vantage point, realizing that things have to change and again back to you know, whether it's value stream management or site reliability, engineering or biz Opps we're all kind of using slightly different languages on DSO. I think one of the important aspects of these apps is for us, all of us, whether we're talking about consulting actual transformation experts, whether we're talking about vendors right to provide sort of tools and technologies or these larger enterprises to transform for all of us to basically have kind of a reference that lets us speak around kind of in a much more consistent way. The second aspect is for to me is for these concepts to start to be embraced not just by us or trying or vendors, um, system integrators, consulting firms, educators, spot leaders but also for some of our own customers to start to become evangelists of their own in the industry. So we are. Our objective with the coalition is to be pretty, pretty broad, Um, and our hope is by by starting to basically educate our joint customers or our partners that we can start to really foster disbelievers and and start to really change some of dynamics. So we're very pleased that if you look at what some of the companies which have joined the the manifesto eso, we have vendors such as stashed up or advance or pager duty, for instance, or even planned you one of my direct competitors but also fought leaders like Tom Davenport or or Cap Gemini or smaller firms like Business Agility Institute or Agility Elf on DSO our goal really is to start to bring together. For three years, people have bean LP. Large organizations do digital transformation. Vendors were providing the technologies that many of these organizations used to deliver all these digital transformation and for all of us to start to provide the kind of education, support and tools that the industry need. >>That's great search. And, you know, congratulations to you and the team. I know this has been going on for a while putting all this together, getting people to sign onto the manifesto of putting the coalition together and finally today getting to unveil it to the world in a little bit more of a public opportunity. So again, you know, really good values, really simple principles, something that that shouldn't have to be written down. But it's nice because it is. And now you can print it out and stick it on your wall. So thank you for for sharing the story. And again, congrats to you on the team. >>Thank you. Thank you. Appreciate it. >>My pleasure. Alright, He surge If you wanna learn more about the bizarre manifesto goto biz Opps manifesto dot or greed it and you can sign it and you can stay here from or coverage on. The Cube of the bizarre manifesto unveiled. Thanks for watching. See you next time.

>> Announcer: From around the globe, it's theCUBE with digital coverage of BizOps Manifesto Unveiled, brought to you by BizOps Coalition. >> Hey, welcome back, everybody. Jeff Frick here with theCUBE for our ongoing coverage of the big unveil. It's the BizOps Manifesto Unveil and we're going to start that again. >> From the top. >> Three. >> Crew Member: Yeah, from the top. Little bleep bleep bleep, there we go. >> Manifesto. >> Crew Member: Second time's the charm, coming to you in five, four, three, two. >> Hey, welcome back, everybody. Jeff Frick here with theCUBE coming to you from our Palo Alto studios today for a big, big reveal. We're excited to be here. It's the BizOps Manifesto Unveiling. Things have been in the works for a while and we're excited to have our next guest, one of the really the powers behind this whole effort and he's joining us from Boston. It's Serge Lucio, the Vice President and General Manager, Enterprise Software Division at Broadcom. Serge, great to see you. >> Good to see you, Jeff, Glad to be here. >> Absolutely. So, you've been in this business for a very long time, you've seen a lot of changes in technology. What is the BizOps Manifesto? What is this coalition all about? Why do we need this today in 2020? >> Yeah, so I've been in this business for close to 25 years, right? So, about 20 years ago, the Agile Manifesto was created. And the goal of the Agile Manifesto was really to address the uncertainty around software development and the inability to predict the effort to build software. And if you roll back kind of 20 years later and if you look at the current state of the industry, the Project Management Institute estimates that we're wasting about a million dollars every 20 seconds in digital transformation initiatives that do not deliver on business results. In fact, we recently surveyed a number of executives in partnership with Harvard Business Review and 77% of those executives think that one of the key challenges that they have is really at the collaboration between business and IT. And that's been kind of the case for almost 20 years now. So, the key challenge we're faced with is really that we need a new approach. And many of the players in the industry, including ourselves, have been using different terms, right? Some are talking about value stream management, some are talking about software delivery management. If you look at the Site Reliability Engineering movement, in many ways, it embodies a lot of these kind of concepts and principles. So, we believe that it became really imperative for us to crystallize around that one concept. And so, in many ways, the BizOps concept and the BizOps Manifesto are around bringing together a number of ideas which have been emerging in the last five years or so and defining the key values and principles to finally help these organizations truly transform and become digital businesses. And so, the hope is that by joining our forces and defining the key principles and values, we can help the industry, not just by providing them with support, but also the tools and consulting that is required for them to truly achieve the kind of transformation that everybody is seeking. >> Right, right. So, COVID, now, we're six months into it approximately, seven months into it, a lot of pain, a lot of bad stuff still happening, we've got two ways to go. But one of the things that on the positive side, right, and you seen all the memes in social media is a driver of digital transformation and a driver of change 'cause we had this light switch moment in the middle of March and there was no more planning, there was no more conversation, you suddenly got remote workforces, everybody's working from home and you got to go, right? So, the reliance on these tools increases dramatically. But I'm curious kind of short of the beginnings of this effort and short of kind of COVID which came along unexpectedly, I mean, what were those inhibitors 'cause we've been making software for a very long time, right? The software development community has adopted kind of rapid change and iterative delivery and sprints, what was holding back the connection with the business side to make sure that those investments were properly aligned with outcomes? >> Well, you have to understand that IT is kind of its own silos and traditionally, IT has been treated as a cost center within large organizations and not as a value center. And so as a result, kind of the traditional dynamic between IT and the business is basically one of kind of supplier up to kind of a business. And if you go back to I think Elon Musk a few years ago basically had these concepts of the machines to build the machines and he went as far as saying that the machines or the production line is actually the product. So, meaning that the core of the innovation is really about building kind of the engine to deliver on the value. And so, in many ways, we have missed on this shift from kind of IT becoming this kind of value center within the enterprises. And it's all about culture. Now, culture is the sum total of behaviors and the reality is that if you look at IT, especially in the last decade, with Agile, with DevOps, with hybrid infrastructures, it's way more volatile today than it was 10 years ago. And so, when you start to look at the velocity of the data, the volume of data, the variety of data to analyze the system, it's very challenging for IT to actually even understand and optimize its own processes, let alone to actually include business as kind of an integral part of a delivery chain. And so, it's both kind of a combination of culture, which is required, as well as tools, right? To be able to start to bring together all these data together. And then, given the volume, variety, velocity of the data, we have to apply some core technologies, which have only really truly emerged in the last five to 10 years around machine learning and analytics. And so, it's really kind of a combination of those things, which are coming together today to really help organizations kind of get to the next level. >> Right, right. So, let's talk about the manifesto. Let's talk about the coalition, the BizOps Coalition. I just like that you put down these really simple kind of straightforward core values. You guys have four core values that you're highlighting, business outcomes over individual projects and outputs, trust and collaboration over siloed teams and organizations, data driven decisions, what you just talked about, over opinions and judgment and learn to respond and pivot. I mean, Serge, these sounds like pretty basic stuff, right? I mean, isn't everyone working to these values already? And I think you touched on it, on culture, right? Trust and collaboration, data driven decisions. I mean, these are fundamental ways that people must run their business today or the person that's across the street that's doing it is going to knock them right off their block. >> Yeah, so that's very true. So, I'll mention another survey we did I think about six months ago. It was in partnership with an industry analyst. And we surveyed, again, a number of IT executives to understand how many were tracking business outcomes, how many of these software executives, IT executives were tracking business outcomes. And there were less than 15% of these executives who were actually tracking the outcomes of the software delivery. And you see that every day, right? So, in my own teams, for instance, we've been adopting a lot of these core principles in the last year or so. And we've uncovered that 16% of our resources were basically aligned around initiatives which were not strategic for us. I take another example. For instance, one of our customers in the airline industry uncovered, for instance, that a number of... That they had software issues that led to people searching for flights and not returning any kind of availability. And yet, the IT teams, whether it's operations or software development, were completely oblivious to that because they were completely blindsided to it. And so, the connectivity between the inwards metrics that IT is using, whether it's database uptime, cycle time or whatever metric we use in IT, are typically completely divorced from the business metrics. And so, at its core, it's really about starting to align the business metrics with the software delivery chain, right? This system which is really a core differentiator for these organizations. It's about connecting those two things and starting to infuse some of the Agile culture and principles that emerge from the software side into the business side. Of course, the Lean movement and other movements have started to change some of these dynamic on the business side. And so, I think this is the moment where we are starting to see kind of the imperative to transform now, COVID obviously has been a key driver for that. The technology is right to start to be able to weave data together and really kind of also the cultural shifts through Agile, through DevOps, through the SRE movement, through Lean business transformation. All these things are coming together and are really creating kind of conditions for the BizOps Manifesto to exist. So, Clayton Christensen, great Harvard Professor, "Innovator's Dilemma", still my all-time favorite business book, talks about how difficult it is for incumbents to react to disruptive change, right? Because they're always working on incremental change 'cause that's what their customers are asking for and there's a good ROI.' When you talk about companies not measuring the right thing, I mean, clearly, IT has some portion of their budget that has to go to keeping the lights on, right? That's always the case, but hopefully, that's an ever decreasing percentage of their total activity. So, what should people be measuring? I mean, what are kind of the new metrics in BizOps that drive people to be looking at the right things, measuring the right things and subsequently making the right decisions, investment decisions, on whether they should move project A along or project B? >> So, there are really two things, right? So, I think what you were talking about is portfolio management, investment management, right? And which is a key challenge, right? In my own experience, right? Driving strategy or a large scale kind of software organization for years, it's very difficult to even get kind of a base data as to who's doing what. I mean, some of our largest customers we're engaged with right now are simply trying to get a very simple answer, which is, how many people do I have in that specific initiative at any point in time and just tracking down information is extremely difficult. And again, back to the Project Management Institute, they have estimated that on average, IT organizations have anywhere between 10 to 20% of their resources focused on initiatives which are not strategically aligned. So, that's one dimension on portfolio management. I think the key aspect though, that's we're really keen on is really around kind of the alignment of a business metrics to the IT metrics. So, I'll use kind of two simple examples, right? And my background is around quality and I've always believed that fitness for purpose is really kind of a key philosophy, if you will. And so, if you start to think about quality as fitness for purpose, you start to look at it from a customer point of view, right? And fitness for purpose for a core banking application or mobile application are different, right? So, the definition of a business value that you're trying to achieve is different. And yet, if you look at our IT operations are operating, they were using kind of a same type of inward metrics, like a database uptime or a cycle time or what is my point velocity, right? And so, the challenge really is this inward facing metrics that the IT is using which are divorced from ultimately the outcome. And so, if I'm trying to build a core banking application, my core metric is likely going to be uptime, right? If I'm trying to build a mobile application or maybe a social mobile app, it's probably going to be engagement. And so, what you want is for everybody across IT to look at these metric and what are the metrics within the software delivery chain which ultimately contribute to that business metric? In some cases, cycle time may be completely irrelevant, right? Again, my core banking app, maybe I don't care about cycle time. And so, it's really about aligning those metrics and be able to start to differentiate. The key challenge you mentioned around the disruption that we see is or the investor's dilemma is really around the fact that many IT organizations are essentially applying the same approaches for innovation, right? For basically scrap work than they would apply to kind of other more traditional projects. And so, there's been a lot of talk about two-speed IT. And yes, it exists, but in reality, are really organizations truly differentiating how they operate their projects and products based on the outcomes that they're trying to achieve? And this is really where BizOps is trying to affect. >> I love that. Again, it doesn't seem like brain surgery, but focus on the outcomes, right? And it's horses for courses, as you said. This project, what you're measuring and how you define success isn't necessarily the same as on this other project. So, let's talk about some of the principles. We talked about the values, but I think it's interesting that the BizOps coalition just basically took the time to write these things down and they don't seem all that super insightful, but I guess you just got to get them down and have them on paper and have them in front of your face. But I want to talk about one of the key ones, which you just talked about, which is changing requirements, right? And working in a dynamic situation, which is really what's driven the software to change in software development because if you're in a game app and your competitor comes out with a new blue sword, you got to come out with a new blue sword. So, whether you had that on your Kanban wall or not. So, it's really this embracing of the speed of change and making that the rule, not the exception. I think that's a phenomenal one. And the other one you talked about is data, right? And that today's organizations generate more data than humans can process. So, informed decisions must be generated by machine learning and AI. And the big data thing with Hadoop started years ago, but we are seeing more and more that people are finally figuring it out, that it's not just big data and it's not even generic machine learning or artificial intelligence, but it's applying those particular data sets and that particular types of algorithms to a specific problem to your point, to try to actually reach an objective, whether that's increasing your average ticket or increasing your checkout rate with shopping carts that don't get left behind and these types of things. So, it's a really different way to think about the world in the good old days, probably when you guys started when we had big giant MRDs and PRDS and sat down and coded for two years and came out with a product release and hopefully, not too many patches subsequently to that. >> It's interesting, right? Again, back to one of these surveys that we did with about 600 IT executives. And we purposely designed those questions to be pretty open. And one of them was really around requirements. And it was really around kind of what is the best approach? What is your preferred approach towards requirements? And if I remember correctly, over 80% of the IT executives said that the best approach, their preferred approach, is for requirements to be completely defined before software development starts. So, let me pause there. We're 20 years after the Agile Manifesto, right? And for 80% of these IT executives to basically claim that the best approach is for requirements to be fully baked before software development starts, basically shows that we still have a very major issue. And again, our hypothesis in working with many organizations is that the key challenge is really the boundary between business and IT, which is still very much contract-based. If you look at the business side, they basically are expecting for IT to deliver on time on budget, right? But what is the incentive for IT to actually deliver on the business outcomes, right? How often is IT measured on the business outcomes and not on an SLA or on a budget type criteria. And so, that's really the fundamental shift that we really need to drive out as an industry. And, we talk about kind of this imperative for organizations to operate as one. And back to the the "Innovator's Dilemma", the key difference between these larger organization is really kind of a... If you look at the amount of capital investment that they can put into pretty much anything, why are they losing compared to startups? Why is it that more than 40% of personal loans today are issued, not by your traditional brick and mortar banks, but by startups? Well, the reason, yes, it's the traditional culture of doing incremental changes and not disrupting ourselves, which Christensen covered at length, but it's also the inability to really fundamentally change kind of the dynamic between business and IT and partner, right? To deliver on a specific business outcome. >> Right, I love that. That's a great summary and in fact, getting ready for this interview, I saw you mentioning another thing where the problem with the Agile development is that you're actually now getting more silos 'cause you have all these autonomous people working kind of independently. So, it's even a harder challenge for the business leaders, as you said, to know what's actually going on. But Serge, I want to close and talk about the coalition. So clearly, these are all great concepts. These are concepts you want to apply to your business every day. Why the coalition? Why take these concepts out to a broader audience, including your competition and the broader industry to say, "Hey, we as a group need to put a stamp of approval on these concepts, these values, these principles?" >> So first, I think we want everybody to realize that we are all talking about the same things, the same concepts. I think we're all from our own different vantage point realizing that things have to change. And again, back to whether it's value stream management or Site Reliability Engineering or BizOps, we're all kind of using slightly different languages. And so, I think one of the important aspects of BizOps is for us, all of us, whether we're talking about consulting, Agile transformation experts, whether we're talking about vendors, right? To provides kind of tools and technologies or these large enterprises to transform for all of us to basically have kind of a reference that lets us speak around kind of in a much more consistent way. The second aspect, to me, is for these concepts to start to be embraced, not just by us or vendors, system integrators, consulting firms, educators, thought leaders, but also for some of our own customers to start to become evangelists of their own in the industry. So, our objective with the coalition is to be pretty, pretty broad. And our hope is by starting to basically educate our joint customers or partners, that we can start to really foster these behaviors and start to really change some of dynamics. So, we're very pleased that if you look at some of the companies which have joined the manifesto, so we have vendors, such as Tasktop, or Appvance or PagerDuty, for instance, or even Planview, one of my direct competitors, but also thought leaders like Tom Davenport or Capgemini or smaller firms like Business Agility Institute or AgilityHealth. And so, our goal really is to start to bring together thought leaders, people who've been helping large organizations do digital transformation, vendors who are providing the technologies that many of these organizations use to deliver on this digital transformation and for all of us to start to provide the kind of education, support and tools that the industry needs. >> Yeah, that's great, Serge, and congratulations to you and the team. I know this has been going on for a while, putting all this together, getting people to sign on to the manifesto, putting the coalition together and finally today, getting to unveil it to the world in a little bit more of a public opportunity. So again, really good values, really simple principles, something that shouldn't have to be written down, but it's nice 'cause it is and now you can print it out and stick it on your wall. So, thank you for sharing the story and again, congrats to you and the team. >> Thank you, thanks, Jeff, appreciate it. >> My pleasure, all righty, Serge. If you want to learn more about the BizOps Manifesto, go to bizopsmanifesto.org, read it and you can sign it and you can stay here for more coverage on theCUBE of the BizOps Manifesto Unveiled. Thanks for watching, see you next time. (upbeat music)

(gentle upbeat music) >> Hello, and welcome to theCUBE presentation of the AWS Startup Showcase Open Cloud Innovations. This is season two episode one of an ongoing series covering setting status from the AWS ecosystem. Talking about innovation, here it's open source for this theme. We do this every episode, we pick a theme and have a lot of fun talking to the leaders in the industry and the hottest startups. I'm your host John Furrier here with Lisa Martin in our Palo Alto studios. Lisa great series, great to see you again. >> Good to see you too. Great series, always such spirited conversations with very empowered and enlightened individuals. >> I love the episodic nature of these events, we get more stories out there than ever before. They're the hottest startups in the AWS ecosystem, which is dominating the cloud sector. And there's a lot of them really changing the game on cloud native and the enablement, the stories that are coming out here are pretty compelling, not just from startups they're actually penetrating the enterprise and the buyers are changing their architectures, and it's just really fun to catch the wave here. >> They are, and one of the things too about the open source community is these companies embracing that and how that's opening up their entry to your point into the enterprise. I was talking with several customers, companies who were talking about the 70% of their pipeline comes from the open source community. That's using the premium version of the technology. So, it's really been a very smart, strategic way into the enterprise. >> Yeah, and I love the format too. We get the keynote we're doing now, opening keynote, some great guests. We have Sir John on from AWS started program, he is the global startups lead. We got Swami coming on and then closing keynote with Deepak Singh. Who's really grown in the Amazon organization from containers now, compute services, which now span how modern applications are being built. And I think the big trend that we're seeing that these startups are riding on that big wave is cloud natives driving the modern architecture for software development, not just startups, but existing, large ISV and software companies are rearchitecting and the customers who buy their products and services in the cloud are rearchitecting too. So, it's a whole new growth wave coming in, the modern era of cloud some say, and it's exciting a small startup could be the next big name tomorrow. >> One of the things that kind of was a theme throughout the conversations that I had with these different guests was from a modern application security perspective is, security is key, but it's not just about shifting lab. It's about doing so empowering the developers. They don't have to be security experts. They need to have a developer brain and a security heart, and how those two organizations within companies can work better together, more collaboratively, but ultimately empowering those developers, which goes a long way. >> Well, for the folks who are watching this, the format is very simple. We have a keynote, editorial keynote speakers come in, and then we're going to have a bunch of companies who are going to present their story and their showcase. We've interviewed them, myself, you Dave Vallante and Dave Nicholson from theCUBE team. They're going to tell their stories and between the companies and the AWS heroes, 14 companies are represented and some of them new business models and Deepak Singh who leads the AWS team, he's going to have the closing keynote. He talks about the new changing business model in open source, not just the tech, which has a lot of tech, but how companies are being started around the new business models around open source. It's really, really amazing. >> I bet, and does he see any specific verticals that are taking off? >> Well, he's seeing the contribution from big companies like AWS and the Facebook's of the world and large companies, Netflix, Intuit, all contributing content to the open source and then startups forming around them. So Netflix does some great work. They donated to open source and next thing you know a small group of people get together entrepreneurs, they form a company and they create a platform around it with unification and scale. So, the cloud is enabling this new super application environment, superclouds as we call them, that's emerging and this new supercloud and super applications are scaling data-driven machine learning and AI that's the new formula for success. >> The new formula for success also has to have that velocity that developers expect, but also that the consumerization of tech has kind of driven all of us to expect things very quickly. >> Well, we're going to bring in Serge Shevchenko, AWS Global Startup program into the program. Serge is our partner. He is the leader at AWS who has been working on this program Serge, great to see you. Thanks for coming on. >> Yeah, likewise, John, thank you for having me very excited to be here. >> We've been working together on collaborating on this for over a year. Again, season two of this new innovative program, which is a combination of CUBE Media partnership, and AWS getting the stories out. And this has been a real success because there's a real hunger to discover content. And then in the marketplace, as these new solutions coming from startups are the next big thing coming. So, you're starting to see this going on. So I have to ask you, first and foremost, what's the AWS startup showcase about. Can you explain in your terms, your team's vision behind it, and why those startup focus? >> Yeah, absolutely. You know John, we curated the AWS Startup Showcase really to bring meaningful and oftentimes educational content to our customers and partners highlighting innovative solutions within these themes and ultimately to help customers find the best solutions for their use cases, which is a combination of AWS and our partners. And really from pre-seed to IPO, John, the world's most innovative startups build on AWS. From leadership downward, very intentional about cultivating vigorous AWS community and since 2019 at re:Invent at the launch of the AWS Global Startup program, we've helped hundreds of startups accelerate their growth through product development support, go to market and co-sell programs. >> So Serge question for you on the theme of today, John mentioned our showcases having themes. Today's theme is going to cover open source software. Talk to us about how Amazon thinks about opensource. >> Sure, absolutely. And I'll just touch on it briefly, but I'm very excited for the keynote at the end of today, that will be delivered by Deepak the VP of compute services at AWS. We here at Amazon believe in open source. In fact, Amazon contributes to open source in multiple ways, whether that's through directly contributing to third-party project, repos or significant code contributions to Kubernetes, Rust and other projects. And all the way down to leadership participation in organizations such as the CNCF. And supporting of dozens of ISV myself over the years, I've seen explosive growth when it comes to open source adoption. I mean, look at projects like Checkov, within 12 months of launching their open source project, they had about a million users. And another great example is Falco within, under a decade actually they've had about 37 million downloads and that's about 300% increase since it's become an incubating project in the CNCF. So, very exciting things that we're seeing here at AWS. >> So explosive growth, lot of content. What do you hope that our viewers and our guests are going to be able to get out of today? >> Yeah, great question, Lisa. I really hope that today's event will help customers understand why AWS is the best place for them to run open source, commercial and which partner solutions will help them along their journey. I think that today the lineup through the partner solutions and Deepak at the end with the ending keynote is going to present a very valuable narrative for customers and startups in selecting where and which projects to run on AWS. >> That's great stuff Serge would love to have you on and again, I want to just say really congratulate your team and we enjoy working with them. We think this showcase does a great service for the community. It's kind of open source in its own way if I can co contributing working on out there, but you're really getting the voices out at scale. We've got companies like Armory, Kubecost, Sysdig, Tidelift, Codefresh. I mean, these are some of the companies that are changing the game. We even had Patreon a customer and one of the partners sneak with security, all the big names in the startup scene. Plus AWS Deepak saying Swami is going to be on the AWS Heroes. I mean really at scale and this is really a great. So, thank you so much for participating and enabling all of this. >> No, thank you to theCUBE. You've been a great partner in this whole process, very excited for today. >> Thanks Serge really appreciate it. Lisa, what a great segment that was kicking off the event. We've got a great lineup coming up. We've got the keynote, final keynote fireside chat with Deepak Singh a big name at AWS, but Serge in the startup showcase really innovative. >> Very innovative and in a short time period, he talked about the launch of this at re:Invent 2019. They've helped hundreds of startups. We've had over 50 I think on the showcase in the last year or so John. So we really gotten to cover a lot of great customers, a lot of great stories, a lot of great content coming out of theCUBE. >> I love the openness of it. I love the scale, the storytelling. I love the collaboration, a great model, Lisa, great to work with you. We also Dave Vallante and Dave Nicholson interview. They're not here, but let's kick off the show. Let's get started with our next guest Swami. The leader at AWS Swami just got promoted to VP of the database, but also he ran machine learning and AI at AWS. He is a leader. He's the author of the original DynamoDB paper, which is celebrating its 10th year anniversary really impacted distributed computing and open source. Swami's introduced many opensource aspects of products within AWS and has been a leader in the engineering side for many, many years at AWS, from an intern to now an executive. Swami, great to see you. Thanks for coming on our AWS startup showcase. Thanks for spending the time with us. >> My pleasure, thanks again, John. Thanks for having me. >> I wanted to just, if you don't mind asking about the database market over the past 10 to 20 years cloud and application development as you see, has changed a lot. You've been involved in so many product launches over the years. Cloud and machine learning are the biggest waves happening to your point to what you're doing now. Software is under the covers it's powering it all infrastructure is code. Open source has been a big part of it and it continues to grow and change. Deepak Singh from AWS talks about the business model transformation of how like Netflix donates to the open source. Then a company starts around it and creates more growth. Machine learnings and all the open source conversations around automation as developers and builders, like software as cloud and machine learning become the key pistons in the engine. This is a big wave, what's your view on this? How how has cloud scale and data impacting the software market? >> I mean, that's a broad question. So I'm going to break it down to kind of give some of the back data. So now how we are thinking about it first, I'd say when it comes to the open source, I'll start off by saying first the longevity and by ability of open sources are very important to our customers and that is why we have been a significant contributor and supporter of these communities. I mean, there are several efforts in open source, even internally by actually open sourcing some of our key Amazon technologies like Firecracker or BottleRocket or our CDK to help advance the industry. For example, CDK itself provides some really powerful way to build and configure cloud services as well. And we also contribute to a lot of different open source projects that are existing ones, open telemetries and Linux, Java, Redis and Kubernetes, Grafana and Kafka and Robotics Operating System and Hadoop, Leucine and so forth. So, I think, I can go on and on, but even now I'd say the database and observability space say machine learning we have always started with embracing open source in a big material way. If you see, even in deep learning framework, we championed MX Linux and some of the core components and we open sourced our auto ML technology auto Glue on, and also be open sourced and collaborated with partners like Facebook Meta on Fighter showing some major components and there, and then we are open search Edge Compiler. So, I would say the number one thing is, I mean, we are actually are very, very excited to partner with broader community on problems that really mattered to the customers and actually ensure that they are able to get amazing benefit of this. >> And I see machine learning is a huge thing. If you look at how cloud group and when you had DynamoDB paper, when you wrote it, that that was the beginning of, I call the cloud surge. It was the beginning of not just being a resource versus building a data center, certainly a great alternative. Every startup did it. That's history phase one inning and a half, first half inning. Then it became a large scale. Machine learning feels like the same way now. You feel like you're seeing a lot of people using it. A lot of people are playing around with it. It's evolving. It's been around as a science, but combined with cloud scale, this is a big thing. What should people who are in the enterprise think about how should they think about machine learning? How has some of your top customers thought about machine learning as they refactor their applications? What are some of the things that you can share from your experience and journey here? >> I mean, one of the key things I'd say just to set some context on scale and numbers. More than one and a half million customers use our database analytics or ML services end-to-end. Part of which machine learning services and capabilities are easily used by more than a hundred thousand customers at a really good scale. However, I still think in Amazon, we tend to use the phrase, "It's day one in the age of internet," even though it's an, or the phrase, "Now, but it's a golden one," but I would say in the world of machine learning, yes it's day one but I also think we just woke up and we haven't even had a cup of coffee yet. That's really that early, so. And, but when you it's interesting, you've compared it to where cloud was like 10, 12 years ago. That's early days when I used to talk to engineering leaders who are running their own data center and then we talked about cloud and various disruptive technologies. I still used to get a sense about like why cloud and basic and whatnot at that time, Whereas now with machine learning though almost every CIO, CEO, all of them never asked me why machine learning. Instead, the number one question, I get is, how do I get started with it? What are the best use cases? which is great, and this is where I always tell them one of the learnings that we actually learned in Amazon. So again, a few years ago, probably seven or eight years ago, and Amazon itself realized as a company, the impact of what machine learning could do in terms of changing how we actually run our business and what it means to provide better customer experience optimize our supply chain and so far we realized that the we need to help our builders learn machine learning and the help even our business leaders understand the power of machine learning. So we did two things. One, we actually, from a bottom-up level, we built what I call as machine learning university, which is run in my team. It's literally stocked with professors and teachers who offer curriculum to builders so that they get educated on machine learning. And now from a top-down level we also, in our yearly planning process, we call it the operational planning process where we write Amazon style narratives six pages and then answer FAQ's. We asked everyone to answer one question around, like how do you plan to leverage machine learning in your business? And typically when someone says, I really don't play into our, it does not apply. It's usually it doesn't go well. So we kind of politely encourage them to do better and come back with a better answer. This kind of dynamic on top-down and bottom-up, changed the conversation and we started seeing more and more measurable growth. And these are some of the things you're starting to see more and more among our customers too. They see the business benefit, but this is where to address the talent gap. We also made machine learning university curriculum actually now open source and freely available. And we launched SageMaker Studio Lab, which is a no cost, no set up SageMaker notebook service for educating learner profiles and all the students as well. And we are excited to also announce AIMLE scholarship for underrepresented students as well. So, so much more we can do well. >> Well, congratulations on the DynamoDB paper. That's the 10 year anniversary, which is a revolutionary product, changed the game that did change the world and that a huge impact. And now as machine learning goes to the next level, the next intern out there is at school with machine learning. They're going to be writing that next paper, your advice to them real quick. >> My biggest advice is, always, I encourage all the builders to always dream big, and don't be hesitant to speak your mind as long as you have the right conviction saying you're addressing a real customer problem. So when you feel like you have an amazing solution to address a customer problem, take the time to articulate your thoughts better, and then feel free to speak up and communicate to the folks you're working with. And I'm sure any company that nurtures good talent and knows how to hire and develop the best they will be willing to listen and then you will be able to have an amazing impact in the industry. >> Swami, great to know you're CUBE alumni love our conversations from intern on the paper of DynamoDB to the technical leader at AWS and database analyst machine learning, congratulations on all your success and continue innovating on behalf of the customers and the industry. Thanks for spending the time here on theCUBE and our program, appreciate it. >> Thanks again, John. Really appreciate it. >> Okay, now let's kick off our program. That ends the keynote track here on the AWS startup showcase. Season two, episode one, enjoy the program and don't miss the closing keynote with Deepak Singh. He goes into great detail on the changing business models, all the exciting open source innovation. (gentle bright music)

>>Well, hello everybody. I'm John Walls here with the cube, and we're very happy to continue our coverage here of a splunk.com 21. And today we're going to talk about cyber security. Uh, obviously everybody is well aware of a number of, uh, breaches that have happened around the globe, but you might say there's been a surge in trying to prevent those from happening down the road. And I'm going to let our guests explain that Ryan Covar, who is the security strategist at Splunk. Ryan. Good to see you with, uh, with us here on the cube. Glad you could join us today. >>Thank you very much. I've wished we could have been doing this in person, but such as the time of life we live. >>Yeah. We have learned to live on zoom that's for sure. And, uh, it's the next best thing to being there. So, uh, again, thanks for that. Um, well, let's talk about surge, if you will. Um, uh, I know obviously Splunk and data security go hand in hand that is a high priority with the, with the company, but now you have a new initiative that you're just now rolling out to take that to an even higher level. Tell us about that. >>Yeah, something I'm extremely excited to announce. Uh, it's the first time we're really talking about it is that.com 21, which is wonderful. And it's kind of the culmination of my seven years here at Splunk. Uh, before I came to Splunk, I did about 20 years of cyber security research and defense and nation state hunting and threat intelligence and policy and compliance, and just about everything, uh, public sector in the U S and the UK private sector, a couple of different places. So I've kind of been around the block. And one of the things I've found that I'm really passionate about is just being a network defender or a blue teamer. And a lot of my time here at Splunk has been around that. It's been speaking at conferences, doing research, um, coming up with ways to basically defend organizations, but the tools they have at hand and something that we say Alon is, uh, we, we work on the problems of today and tomorrow, not the distant future, right? >>The really practical things. And we had an, you know, there was a little bit of a thing called solar winds. You might've heard of it. Um, that happened earlier in December and we were able to stand up kind of on an ad hoc ragtag group of Splunkers around the world, uh, in a matter of hours. And we worked about 24 hours for panning over to Australia, into a Mia, and then back over to America and able to publish really helpful work to, for our customers to detect or defend or mitigate against what we knew at the time around solar winds, the attack. And then as time went on, we were continuing to write and create material, but we didn't have a group that was focused on it. We were all kind of chipping in after hours or, you know, deep deprecating, other bits of work. >>And I said, you know, we really need to focus on this. This is a big deal. And how can we actually surge up to meet these needs if you will, uh, the play on the punter. So we created an idea of a small team, a dedicated to current events and also doing security research around the problems that are facing around the world insecurity who use Splunk and maybe even those who don't. And that's where the idea of this team was formed. And we've been working all summer. We're releasing our first research project, excuse me, uh, at.com, which is around supply chain, compromise using jaw three Zeke and Splunk, uh, author by myself and primarily Marcus law era. And we have other research projects coming out every quarter, along with doing this work around, just helping people with any sort of immediate cybersecurity threat that we're able to assist with. >>So what are you hoping that security teams can get out of this work? Obviously you're investing a lot of resources and doing the research, I assume, diversifying, you know, the areas and to which you're, um, exploring, um, ultimately what would be the takeaway if I was on the other end, if I was on the client and what would you hope that I would be, uh, extracting from this work? >>Sure. We want to get you promoted. I mean, that's kind of the, the joke of it, but we, we talk a lot. I want to make everyone in the world who use a Splunk or cybersecurity, looked into their bosses and defend their company as fast and quickly as possible. So one of the big, mandates for my team is creating consumable, actionable work and research. So we, you know, we joke a lot that, you know, I have a pretty thick beard here. One might even call it a neck beard and a lot of people in our community, we create things for what I would call wizards, cybersecurity wizards, and we go to conferences and we talk from wizard to wizard, and we kind of sit on our ivory tower on stage and kind of proclaim out how to do things. And I've sat on the other side and sometimes those sound great, but they're not actually helping people with their job today. And so the takeaway for me, what I hope people are able to take away is we're here for you. We're here for the little guys, the network defenders, we're creating things that we're hoping you can immediately take home and implement and do and make better detections and really find the things that are immediate threats to your network and not necessarily having to, you know, create a whole new environment or apply magic. So >>Is there a difference then in terms of say enterprise threats, as opposed to, if I'm a small business or of a medium sized business, maybe I have four or 500 employees as opposed to four or 5,000 or 40,000. Um, what about, you know, finding that ground where you can address both of those levels of, of business and of concern, >>You know, 20 years ago or 10 years ago? I would've answered that question very differently and I fully acknowledge I have a bias in nation state threats. That's what I'm primarily trained in, however, in the last five years, uh, thanks or not. Thanks to ransomware. What we're seeing is the same threats that are affecting and impacting fortune 100 fortune 10 companies. The entire federal government of the United States are the exact same threats that are actually impacting and causing havoc on smaller organizations and businesses. So the reality is in today's threat landscape. I do believe actually the threat is the same to each, but it is not the same level of capabilities for a 100% or 500 person company to a company, the size of Splunk or a fortune 100 company. Um, and that's something that we are actually focusing on is how do we create things to help every size of that business, >>Giving me the tools, right, exactly. >>Which is giving you the power to fight that battle yourself as much as possible, because you may never be able to have the head count of a fortune 100 company, but thanks to the power of software and tools and things like the cloud, you might have some force multipliers that we're hoping to create for you in a much more package consumable method. >>Yeah. Let's go back to the research that you mentioned. Um, how did you pick the first topic? I mean, because this is your, your splash and, and I'm sure there was a lot of thought put into where do we want to dive in >>First? You know, I'd love to say there was a lot of thought put into it because it would make me sound smarter, but it was something we all just immediately knew was a gap. Um, you know, solar winds, which was a supply chain, compromise attack really revealed to many of us something that, um, you know, reporters had been talking about for years, but we never really saw come to fruition was a real actionable threat. And when we started looking at our library of offerings and what we could actually help customers with, I talked over 175 federal and private sector companies around the world in a month and a half after solar winds. And a lot of times the answer was, yeah, we can't really help you with this specific part of the problem. We can help you around all sorts of other places, but like, gosh, how do you actually detect this? >>And there's not a great answer. And that really bothered me. And to be perfectly honest, that was part of the reason that we founded the team. So it was a very obvious next step was, well, this is why we're creating the team. Then our first product should probably be around this problem. And then you say, okay, supply chain, that's really big. That's a huge chunk of work. So the first question is like, well, what can we actually affect change on without talking about things like quantum computing, right? Which are all things that are, you know, blockchain, quantum computing, these are all solutions that are actually possible to solve or mitigate supply chain compromise, but it's not happening today. And it sure as heck isn't even happening tomorrow. So how do we create something that's digestible today? And so what Marcus did, and one of his true skillsets is really refining the problem down, down, down, down. >>And where can we get to the point of, Hey, this is data that we think most organizations have a chance of collecting. These are methodologies that we think people can do and how can they actually implement them with success in their network. And then we test that and then we kind of keep doing a huge fan of the concept of OODA loop, orient, orient, observe, decide, and act. And we do that through our hypothesizing. We kind of keep looking at that and iterating over and over and over again, until we're able to come up with a solution that seems to be applicable for the personas that we're trying to help. And that's where we got out with this research of, Hey, collect network data, use a tool like Splunk and some of our built-in statistical analysis functions and come out the other side. And I'll be honest, we're not solving the problem. >>We're helping you with the problem. And I think that's a key differentiator of what we're saying is there is no silver bullet and frankly, anyone that tells you they can solve supply chain, uh, let me know, cause I want to join that hot new startup. Um, the reality is we can help you go from a field of haystacks to a single haystack and inside that single haystack, there's a needle, right? And there's actually a lot of value in that because before the PR problem was unapproachable, and now we've gotten it down to saying like, Hey, use your traditional tools, use your traditional analytic craft on a much smaller set of data where we've pretty much verified that there's something here, but look right here. And that's where we kind of focused. >>You talked about, you know, and we all know about the importance and really the emphasis that's put on data protection, right? Um, at the same time, can you use data to help you protect? I mean, is there information or insight that could be gleaned from, from data that whether it's behavior or whatever the case might be, that, that not only, uh, is something that you can operationalize and it's a good thing for your business, but you could also put it into practice in terms of your security practices to >>A hundred percent. The, the undervalued aspect of cybersecurity in my opinion, is elbow grease. Um, you can buy a lot of tools, uh, but the reality is to get value immediately. Usually the easiest place to start is just doing the hard detail oriented work. And so when you ask, is there data that can help you immediately data analytics? Actually, I go to, um, knowing what you have in your network, knowing what you have, that you're actually trying to protect asset and inventory, CMDB, things like this, which is not attractive. It's not something people want to talk about, but it's actually the basis of all good security. How do you possibly defend something if you don't know what you're defending and where it is. And something that we found in our research was in order to detect and find anomalous behavior of systems communicating outbound, um, it's too much. >>So what you have to do is limit the scope down to those critical assets that you're most concerned about and a perfect example of critical asset. And there's no, no shame or victim blaming here, put on solar winds. Uh, it's just that, that is an example of an appliance server that has massive impact on the organization as we saw in 2020. And how can you actually find that if you don't know where it is? So really that first step is taking the data that you already have and saying, let's find all the systems that we're trying to protect. And what's often known as a crown jewels approach, and then applying these advanced analytics on top of those crown jewel approaches to limit the data scope and really get it to just what you're trying to protect. And once you're positive that you have that fairly well defended, then you go out to the next tier and the next tier in next year. And that's a great approach, take things you're already doing today and applying them and getting better results tomorrow. >>No, before I let you go, um, I I'd like to just have you put a, uh, a bow on surge, if you will, on that package, why is this a big deal to you? It's been a long time in the making. I know you're very happy about the rollout of this week. Um, you know, what's the impact you want to have? Why is it important? >>We did a lot of literature review. I have a very analytical background. My time working at DARPA taught me a lot about doing research and development and on laying out the value of failure, um, and how much sometimes even failing as long as you talk about it and talk about your approach and methodology and share that is important. And the other part of this is I see a lot of work done by many other wonderful organizations, uh, but they're really solving for a problem further down the road or they're creating solutions that not everyone can implement. And so what I think is so important and what's different about our team is we're not only thinking differently, we're hiring differently. You know, we have people who have a threat intelligence background from the white house. We have another researcher who did 10 years at DARPA insecurity, research and development. >>Uh, we've recently hired a, a former journalist who she's made a career pivot into cybersecurity, and she's helping us really review the data and what people are facing and come up with a real connection to make sure we are tackling the right problems. And so to me, what I'm most excited about is we're not only trying to solve different problems. And I think what most of the world is looking at for cybersecurity research, we've staffed it to be different, think different and come up with things that are probably a little less, um, normal than everyone's seen before. And I'm excited about that. >>Well, and, and rightly so, uh, Ryan, thanks for the time, a pleasure to have you here on the cube and, uh, the information again, the initiative is Serge, check it out, uh, spunk very much active in the cyber security protection business. And so we have certainly appreciate that effort. Thank you, Ryan. >>Well, thank you very much, John. You bet Ryan, >>Covar joining us here on our cube coverage. We continue our coverage of.com for 21.

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of DevOps and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also got Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago, surgery it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen, uh, in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate, physical colocation. Hope. That makes sense. >>It does. I should have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with junk. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other, I, Oh, individuals teams, uh, contributed to kind of a business outcome. What is our shared goal or shared vision? What's what is it we're trying to achieve collectively and, uh, keeping it aligned to that. Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organizations, there's the specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of revert back to, okay, what is my database all the time? What is my cycle time like? >>And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens for these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, theories of experience was to actually weakness at one of a large financial institution, um, you know, to stakeholders and quote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, testing, execution results, you know, covert coverage, um, official liabilities and all the all ran. It could have a direction leveling. So that's when you start to put these things in context and represent that in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and understand of they jointly are competing to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your on spiritual co-location and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it says, simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos who were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very different environments. But in all areas, BT you're in whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in there kind of testing world at the moment. >>But, you know, I could talk about things like continuous delivery director, one of Broadcom's tools, and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum that up. Autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter, Jeff? >>Well, I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value associated with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test? Uh, department's a group. The fourth level that I think is really important, uh, in, in the current, uh, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric. How full was the parking lot at nine? And how full was it at five o'clock? I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customer. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big, and also, >>How do you measure quality kind of following onto the advice that Jeff provided? I mean, Jeff provided some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker called me Kirsten a couple of months ago. It talked very much about how important flight management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes too long? So I think that's a very important thing for us. It's, um, even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up, Pat, our teams training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, GemCis point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about, are these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing, the second aspect, I think that's oftentimes misunderstood. >>Yeah. >>So that cycle time, or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, you look at, for instance, things like, you know, lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. And so I think that there is, you know, the metrics there is, I think a lot of, uh, work that we need to do in terms of classifying this antibiograms, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as a, as a key patterning industry or anti-fat. Um, but what our scrum fall right, is a key one, right. And you will detect that through defect, arrival rates. That's where that looks like an escort. And so I think it's beyond kind of the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on the fit. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff, we'll go back to you. >>I would say, look for the opportunities that this disruption presents. And there are a couple that I see, first of all, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks should be on the lookout for? >>I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but Loco also, um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and, and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the life cycle, um, where, when I heard the word scrum for it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment. So we need to get beyond that. We need to be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production in going straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately >>We all need some peanut butter and chocolate. All right. It starts to take us on that's what's that nugget you think everyone needs to have on their agendas. >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's, it's all about the lost city, right? It's all about, we still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of that spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value and to align everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi, Lisa Martin here covering the Broadcom dev ops virtual forum. I'm very pleased to be joined today by a cube alumni, Jeffrey Hammond, the vice president and principal analyst serving CIO is at Forester. Jeffrey. Nice to talk with you today. >>Good morning. It's good to be here. Yeah. >>So a virtual forum, great opportunity to engage with our audiences so much has changed in the last it's an understatement, right? Or it's an overstated thing, but it's an obvious, so much has changed when we think of dev ops. One of the things that we think of is speed, you know, enabling organizations to be able to better serve customers or adapt to changing markets like we're in now, speaking of the need to adapt, talk to us about what you're seeing with respect to dev ops and agile in the age of COVID, what are things looking like? >>Yeah, I think that, um, for most organizations, we're in a, uh, a period of adjustment, uh, when we initially started, it was essentially a sprint, you know, you run as hard as you can for as fast as you can for as long as you can and you just kind of power through it. And, and that's actually what, um, the folks that get hub saw in may when they ran an analysis of how developers, uh, commit times and a level of work that they were committing and how they were working, uh, in the first couple of months of COVID was, was progressing. They found that developers, at least in the Pacific time zone were actually increasing their work volume, maybe because they didn't have two hour commutes or maybe because they were stuck away in their homes, but for whatever reason, they were doing more work. >>And it's almost like, you know, if you've ever run a marathon the first mile or two in the marathon, you feel great and you just want to run and you want to power through it and you want to go hard. And if you do that by the time you get to mile 18 or 19, you're going to be gassed. It's sucking for wind. Uh, and, and that's, I think where we're starting to hit. So as we start to, um, gear our development chops out for the reality that most of us won't be returning into an office until 2021 at the earliest and many organizations will, will be fundamentally changing, uh, their remote workforce, uh, policies. We have to make sure that the agile processes that we use and the dev ops processes and tools that we use to support these teams are essentially aligned to help developers run that marathon instead of just kind of power through. >>So, um, let me give you a couple of specifics for many organizations, they have been in an environment where they will, um, tolerate Rover remote work and what I would call remote work around the edges like developers can be remote, but product managers and, um, you know, essentially scrum masters and all the administrators that are running the, uh, uh, the SCM repositories and, and the dev ops pipelines are all in the office. And it's essentially centralized work. That's not, we are anymore. We're moving from remote workers at the edge to remote workers at the center of what we do. And so one of the implications of that is that, um, we have to think about all the activities that you need to do from a dev ops perspective or from an agile perspective, they have to be remote people. One of the things I found with some of the organizations I talked to early on was there were things that administrators had to do that required them to go into the office to reboot the SCM server as an example, or to make sure that the final approvals for production, uh, were made. >>And so the code could be moved into the production environment. And so it actually was a little bit difficult because they had to get specific approval from the HR organizations to actually be allowed to go into the office in some States. And so one of the, the results of that is that while we've traditionally said, you know, tools are important, but they're not as important as culture as structure as organization as process. I think we have to rethink that a little bit because to the extent that tools enable us to be more digitally organized and to hiring, you know, achieve higher levels of digitization in our processes and be able to support the idea of remote workers in the center. They're now on an equal footing with so many of the other levers, uh, that, that, um, uh, that organizations have at their disposal. Um, I'll give you another example for years. >>We've said that the key to success with agile at the team level is cross-functional co located teams that are working together physically co located. It's the easiest way to show agile success. We can't do that anymore. We can't be physically located at least for the foreseeable future. So, you know, how do you take the low hanging fruits of an agile transformation and apply it in, in, in, in the time of COVID? Well, I think what you have to do is that you have to look at what physical co-location has enabled in the past and understand that it's not so much the fact that we're together looking at each other across the table. It's the fact that we're able to get into a shared mindspace, uh, from, um, uh, from a measurement perspective, we can have shared purpose. We can engage in high bandwidth communications. It's the spiritual aspect of that physical co-location that is actually important. So one of the biggest things that organizations need to start to ask themselves is how do we achieve spiritual colocation with our agile teams? Because we don't have the, the ease of physical co-location available to us anymore? >>Well, the spiritual co-location is such an interesting kind of provocative phrase there, but something that probably was a challenge here, we are seven, eight months in for many organizations, as you say, going from, you know, physical workspaces, co-location being able to collaborate face to face to a, a light switch flip overnight. And this undefined period of time where all we were living with with was uncertainty, how does spiritual, what do you, when you talk about spiritual co-location in terms of collaboration and processes and technology help us unpack that, and how are you seeing organizations adopted? >>Yeah, it's, it's, um, it's a great question. And, and I think it goes to the very root of how organizations are trying to transform themselves to be more agile and to embrace dev ops. Um, if you go all the way back to the, to the original, uh, agile manifesto, you know, there were four principles that were espoused individuals and interactions over processes and tools. That's still important. Individuals and interactions are at the core of software development, processes and tools that support those individual and interact. Uh, those individuals in those interactions are more important than ever working software over comprehensive documentation. Working software is still more important, but when you are trying to onboard employees and they can't come into the office and they can't do the two day training session and kind of understand how things work and they can't just holler over the cube, uh, to ask a question, you may need to invest a little bit more in documentation to help that onboarding process be successful in a remote context, uh, customer collaboration over contract negotiation. >>Absolutely still important, but employee collaboration is equally as important if you want to be spiritually, spiritually co-located. And if you want to have a shared purpose and then, um, responding to change over following a plan. I think one of the things that's happened in a lot of organizations is we have focused so much of our dev ops effort around velocity getting faster. We need to run as fast as we can like that sprinter. Okay. You know, trying to just power through it as quickly as possible. But as we shift to, to the, to the marathon way of thinking, um, velocity is still important, but agility becomes even more important. So when you have to create an application in three weeks to do track and trace for your employees, agility is more important. Um, and then just flat out velocity. Um, and so changing some of the ways that we think about dev ops practices, um, is, is important to make sure that that agility is there for one thing, you have to defer decisions as far down the chain to the team level as possible. >>So those teams have to be empowered to make decisions because you can't have a program level meeting of six or seven teams and one large hall and say, here's the lay of the land. Here's what we're going to do here are our processes. And here are our guardrails. Those teams have to make decisions much more quickly that developers are actually developing code in smaller chunks of flow. They have to be able to take two hours here or 50 minutes there and do something useful. And so the tools that support us have to become tolerant of the reality of, of, of, of how we're working. So if they work in a way that it allows the team together to take as much autonomy as they can handle, um, to, uh, allow them to communicate in a way that, that, that delivers shared purpose and allows them to adapt and master new technologies, then they're in the zone in their spiritual, they'll get spiritually connected. I hope that makes sense. >>It does. I think we all could use some of that, but, you know, you talked about in the beginning and I've, I've talked to numerous companies during the pandemic on the cube about the productivity, or rather the number of hours of work has gone way up for many roles, you know, and, and, and times that they normally late at night on the weekends. So, but it's a cultural, it's a mind shift to your point about dev ops focused on velocity, sprints, sprints, sprints, and now we have to, so that cultural shift is not an easy one for developers. And even at this folks to flip so quickly, what have you seen in terms of the velocity at which businesses are able to get more of that balance between the velocity, the sprint and the agility? >>I think, I think at the core, this really comes down to management sensitivity. Um, when everybody was in the office, you could kind of see the mental health of development teams by, by watching how they work. You know, you call it management by walking around, right. We can't do that. Managers have to, um, to, to be more aware of what their teams are doing, because they're not going to see that, that developer doing a check-in at 9:00 PM on a Friday, uh, because that's what they had to do, uh, to meet the objectives. And, um, and, and they're going to have to, to, um, to find new ways to measure engagement and also potential burnout. Um, friend of mine once had, uh, had a great metric that he called the parking lot metric. It was helpful as the parking lot at nine. And how full was it at five? >>And that gives you an indication of how engaged your developers are. Um, what's the digital equivalent equivalent to the parking lot metric in the time of COVID it's commit stats, it's commit rates. It's, um, you know, the, uh, the turn rate, uh, that we have in our code. So we have this information, we may not be collecting it, but then the next question becomes, how do we use that information? Do we use that information to say, well, this team isn't delivering as at the same level of productivity as another team, do we weaponize that data or do we use that data to identify impedances in the process? Um, why isn't a team working effectively? Is it because they have higher levels of family obligations and they've got kids that, that are at home? Um, is it because they're working with, um, you know, hardware technology, and guess what, they, it's not easy to get the hardware technology into their home office because it's in the lab at the, uh, at the corporate office, uh, or they're trying to communicate, uh, you know, halfway around the world. >>And, uh, they're communicating with a, with an office lab that is also shut down and, and, and the bandwidth just doesn't enable the, the level of high bandwidth communications. So from a dev ops perspective, managers have to get much more sensitive to the, the exhaust that the dev ops tools are throwing off, but also how they're going to use that in a constructive way to, to prevent burnout. And then they also need to, if they're not already managing or monitoring or measuring the level of developer engagement, they have, they really need to start whether that's surveys around developer satisfaction, um, whether it's, you know, more regular social events, uh, where developers can kind of just get together and drink a beer and talk about what's going on in the project, uh, and monitoring who checks in and who doesn't, uh, they have to, to, um, work harder, I think, than they ever have before. >>Well, and you mentioned burnout, and that's something that I think we've all faced in this time at varying levels and it changes. And it's a real, there's a tension in the air, regardless of where you are. There's a challenge, as you mentioned, people having, you know, coworker, their kids as coworkers and fighting for bandwidth, because everyone is forced in this situation. I'd love to get your perspective on some businesses that are, that have done this well, this adaptation, what can you share in terms of some real-world examples that might inspire the audience? >>Yeah. Uh, I'll start with, uh, stack overflow. Uh, they recently published a piece in the journal of the ACM around some of the things that they had discovered. Um, you know, first of all, just a cultural philosophy. If one person is remote, everybody is remote. And you just think that way from an executive level, um, social spaces. One of the things that they talk about doing is leaving a video conference room open at a team level all day long, and the team members, you know, we'll go on mute, you know, so that they don't have to, that they don't necessarily have to be there with somebody else listening to them. But if they have a question, they can just pop off mute really quickly and ask the question. And if anybody else knows the answer, it's kind of like being in that virtual pod. Uh, if you, uh, if you will, um, even here at Forrester, one of the things that we've done is we've invested in social ceremonies. >>We've actually moved our to our team meetings on, on my analyst team from, from once every two weeks to weekly. And we have built more time in for social Ajay socialization, just so we can see, uh, how, how, how we're doing. Um, I think Microsoft has really made some good, uh, information available in how they've managed things like the onboarding process. I think I'm Amanda silver over there mentioned that a couple of weeks ago when, uh, uh, a presentation they did that, uh, uh, Microsoft onboarded over 150,000 people since the start of COVID, if you don't have good remote onboarding processes, that's going to be a disaster. Now they're not all developers, but if you think about it, um, everything from how you do the interviewing process, uh, to how you get people, their badges, to how they get their equipment. Um, security is a, is another issue that they called out typically, uh, it security, um, the security of, of developers machines ends at, at, at the corporate desktop. >>But, you know, since we're increasingly using our own machines, our own hardware, um, security organizations kind of have to extend their security policies to cover, uh, employee devices, and that's caused them to scramble a little bit. Uh, so, so the examples are out there. It's not a lot of, like, we have to do everything completely differently, but it's a lot of subtle changes that, that have to be made. Um, I'll give you another example. Um, one of the things that, that we are seeing is that, um, more and more organizations to deal with the challenges around agility, with respect to delivering software, embracing low-code tools. In fact, uh, we see about 50% of firms are using low-code tools right now. We predict it's going to be 75% by the end of next year. So figuring out how your dev ops processes support an organization that might be using Mendix or OutSystems, or, you know, the power platform building the front end of an application, like a track and trace application really, really quickly, but then hooking it up to your backend infrastructure. Does that happen completely outside the dev ops investments that you're making and the agile processes that you're making, or do you adapt your organization? Um, our hybrid teams now teams that not just have professional developers, but also have business users that are doing some development with a low-code tool. Those are the kinds of things that we have to be, um, willing to, um, to entertain in order to shift the focus a little bit more toward the agility side, I think >>Lot of obstacles, but also a lot of opportunities for businesses to really learn, pay attention here, pivot and grow, and hopefully some good opportunities for the developers and the business folks to just get better at what they're doing and learning to embrace spiritual co-location Jeffrey, thank you so much for joining us on the program today. Very insightful conversation. >>My pleasure. It's it's, it's an important thing. Just remember if you're going to run that marathon, break it into 26, 10 minute runs, take a walk break in between each and you'll find that you'll get there. >>Digestible components, wise advice. Jeffery Hammond. Thank you so much for joining for Jeffrey I'm Lisa Martin, you're watching Broadcom's dev ops virtual forum >>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom, >>Continuing our conversations here at Broadcom's dev ops virtual forum. Lisa Martin here, please. To welcome back to the program, Serge Lucio, the general manager of the enterprise software division at Broadcom. Hey, Serge. Welcome. Thank you. Good to be here. So I know you were just, uh, participating with the biz ops manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept, but I wanted to get your thoughts on spiritual co-location as really a necessity for biz ops to succeed in this unusual time in which we're living. What are your thoughts on spiritual colocation in terms of cultural change versus adoption of technologies? >>Yeah, it's a, it's, it's quite interesting, right? When we, when we think about the major impediments for, uh, for dev ops implementation, it's all about culture, right? And swore over the last 20 years, we've been talking about silos. We'd be talking about the paradox for these teams to when it went to align in many ways, it's not so much about these teams aligning, but about being in the same car in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me, the, this, this is really about kind of changing this culture where people start to look at a kind of OKR is instead of the key objective, um, that, that drives the entire team. Now, what it means in practice is really that's, uh, we need to change a lot of behaviors, right? It's not about the Yarki, it's not about roles. It's about, you know, who can do what and when, and, uh, you know, driving a bias towards action. It also means that we need, I mean, especially in this school times, it becomes very difficult, right? To drive kind of a kind of collaboration between these teams. And so I think there there's a significant role that especially tools can play in terms of providing this complex feedback from teams to, uh, to be in that preface spiritual qualification. >>Well, and it talked about culture being, it's something that, you know, we're so used to talking about dev ops with respect to velocity, all about speed here. But of course this time everything changed so quickly, but going from the physical spaces to everybody being remote really does take it. It's very different than you can't replicate it digitally, but there are collaboration tools that can kind of really be essential to help that cultural shift. Right? >>Yeah. So 2020, we, we touch to talk about collaboration in a very mundane way. Like, of course we can use zoom. We can all get into, into the same room. But the point when I think when Jeff says spiritual, co-location, it's really about, we all share the same objective. Do we, do we have a niece who, for instance, our pipeline, right? When you talk about dev ops, probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team, but just thinking about a pipeline, right, at the end of the day, it's all about what is the meantime to beat back to these teams. If I'm a developer and a commit code, I don't, does it take where, you know, that code to be processed through pipeline pushy? Can I get feedback if I am a finance person who is funding a product or a project, what is my meantime to beat back? >>And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of the Dora metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration, it's really a balance. How do you provide the feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this, this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when I checked in code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allegation, how can the system help me reduce my change pillar rates? Because it's, it's able to predict that some issue was introduced in the outpatient or work product. Um, so I think there's, there's a great role of technology and AI candidate Lynch to, to actually provide that new level of collaboration. >>So we'll get to AI in a second, but I'm curious, what are some of the, of the metrics you think that really matter right now is organizations are still in some form of transformation to this new almost 100% remote workforce. >>So I'll just say first, I'm not a big fan of metrics. Um, and the reason being that, you know, you can look at a change killer rate, right, or a lead time or cycle time. And those are, those are interesting metrics, right? The trend on metric is absolutely critical, but what's more important is you get to the root cause what is taught to you lean to that metric to degrade or improve or time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll give you a very mundane example. You know, we know that cycle time is heavily influenced by, um, organizational boundaries. So, you know, we talk a lot about silos, but, uh, we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are a pretty, you know, valid set metrics, but what's way more important is to understand what are the antiperspirants, what are the things that we can detect through the data that actually are affecting those metrics. And, uh, I mean, over the last 10, 20 years, we've learned a lot about kind of what are, what are the antiperspirants within our large enterprise customers. And there are plenty of them. >>What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >>So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle, where DevOps is more and more embracing value stream management value stream mapping. Um, and I think what's interesting is that in many ways the product is becoming the new silo. Uh, the notion of a product is very difficult by itself to actually define people are starting to recognize that a value stream is not its own little kind of Island. That in reality, when I define a product, this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So, so even on that, and there is clearly kind of a new sets, if you will, of anti-patterns where products are being defined as a set of OTRs, they have interdependencies and you have have a new set of silos on the operands, uh, the Abra key movement to Israel and the SRE space where, um, I think there is a cultural clash while the dev ops side is very much embracing this notion of OTRs and value stream mapping and Belgium management. >>On the other end, you have the it operations teams. We still think business services, right? For them, they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so the, the, there is there's this boundary where, um, I think, well, SRE is being put in place. And there's lots of thinking about what kind of metrics can be fined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for true operations team. >>And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You talked about, uh, AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate, maybe see eye to eye with the SRS? >>Yeah. Um, so th you're kind of too. So even for myself, as a leader of a, you know, 1500 people organization, there's a number of things I don't see right. On a daily basis. And, um, I think the, the, the, the technologies that we have at our disposal today from the AI are able to mind a lot of data and expose a lot of, uh, issues that's as leaders we may not be aware of. And some of the, some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you, when you start to understand, for instance, uh, what is the, what is the working progress right to during the sprint? Um, when you start to analyze the data you can detect, for instance, that maybe the teams are over committed, that there is too much work in progress. >>You can start to identify kind of, interdepencies either from a technology, from a people point of view, which were hidden, uh, you can start to understand maybe the change filler rates he's he is dragging. So I believe that there is a, there's a fundamental role to be played by the tools to, to expose again, these anti parents, to, to make these things visible to the teams, to be able to even compare teams. Right. One of the things that's, that's, uh, that's amazing is now we have access to tons of data, not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate, and what's working, what's not working >>Thoughts on AI and automation as, as a facilitator of spiritual co-location. >>Yeah, absolutely. Absolutely. It's um, you know, th there's, uh, the problem we all face is the unknown, right? The, the law city, but volume variety of the data, uh, everyday we don't really necessarily completely appreciate what is the impact of our actions, right? And so, um, AI can really act as a safety net that enables us to, to understand what is the impact of our actions. Um, and so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data, um, and collaborate on the data. And the actual matter, I think is, is a, is a very powerful enabler, uh, on, in that respect. I mean, I, I've seen, um, I've seen countless of times that, uh, for instance, at the SRE boundary, um, to basically show that we'll turn the quality attributes, so an incoming release, right. And exposing that to, uh, an operations person and a sorry person, and enabling that collaboration dialogue through data is a very, very powerful tool. >>Do you have any recommendations for how teams can use, you know, the SRE folks, the dev ops says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be nonproductive. >>Yeah. So to me, the th there, there's a part of the question really is when, when we talk about data, there are there different ways you can use data, right? Um, so you can, you can do a lot of an analytics, predictive analytics. So I think there is a, there's a tendency, uh, to look at, let's say a, um, a specific KPI, like a, an availability KPI, or change filler rate, and to basically do a regression analysis and projecting all these things, going to happen in the future. To me, that that's, that's a, that's a bad approach. The reason why I fundamentally think it's a better approach is because we are systems. The way we develop software is, is a, is a non-leader kind of system, right? Software development is not linear nature. And so I think there's a D this is probably the worst approach is to actually focus on metrics on the other end. >>Um, if you, if you start to actually understand at a more granular level, what har, uh, which are the things which are contributing to this, right? So if you start to understand, for instance, that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we, we have, I've actually, uh, a customer who, uh, identified that, uh, over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed, this resulted in these plant outages. So if you start to be able to basically establish causality, right, cause an effect between kind of data across the last cycle. I think, I think this is the right way to, uh, to, to use AI. And so pharma to be, I think it's way more God could have a classification problem. What are the classes of problems that do exist and affect things as opposed to analytics, predictive, which I don't think is as powerful. >>So I mentioned in the beginning of our conversation, that just came off the biz ops manifesto. You're one of the authors of that. I want to get your thoughts on dev ops and biz ops overlapping, complimenting each other, what, from a, the biz ops perspective, what does it mean to the future of dev ops? >>Yeah, so, so it's interesting, right? If you think about DevOps, um, there's no felony document, right? Can we, we can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways, there's no clear definition of what dev ops is. Uh, if you go to the dev ops Institute today, you'll see that they are specific, um, trainings for instance, on value management on SRE. And so in many ways, the problem we have as an industry is that, um, there are set practices between agile dev ops, SRE Valley should management. I told, right. And we all basically talk about the same things, right. We all talk about essentially, um, accelerating in the meantime fee to feedback, but yet we don't have the common framework to talk about that. The other key thing is that we add to wait, uh, for, uh, for jeans, Jean Kim's Lascaux, um, to, uh, to really start to get into the business aspect, right? >>And for value stream mapping to start to emerge for us to start as an industry, right. It, to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about driving these business outcomes. And so to me, these ops is really about kind of, uh, putting a lens on this critical element that it's not business and it, that we in fact need to fuse business 19 that I need needs to transform itself to recognize that it's, it's this value generator, right. It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this Oliver or kind of framework, if you will. That set the context for what is the reason, uh, for it to exist. What's part of the core values and principles that it needs to embrace to, again, change from a cost center to a value center. And then we need to start to use this as a way to start to unify some of the, again, the core practices, whether it's agile, DevOps value, stream mapping SRE. Um, so, so I think over time, my hope is that we start to optimize a lot of our practices, language, um, and, uh, and cultural elements. >>Last question surgeon, the last few seconds we have here talking about this, the relation between biz ops and dev ops, um, what do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >>So to me, the key, the key, um, challenge for, for the industry is really around. So we were seeing a very rapid shift towards kind of, uh, product to product, right. Which we don't want to do is to recreate kind of these new silos, these hard silos. Um, so that, that's one of the big changes, uh, that I think we need to be, uh, to be really careful about, um, because it is ultimately, it is about culture. It's not about, uh, it's not about, um, kind of how we segment the work, right. And, uh, any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of, um, kind of the spiritual co-location, I think it's, it's really about that too. It's really about kind of, uh, uh, focusing on the business outcomes on kind of aligning on driving engagement across the teams, but, but not for create a, kind of a new set of silos, which instead of being vertical are going to be these horizontal products >>Crazy by surge that looking at culture as kind of a way of really, uh, uh, addressing and helping to, uh, re re reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps virtual forum. >>Thank you. Thanks for your time. >>I'll be right back >>From around the globe it's the cube with digital coverage of devops virtual forum brought to you by Broadcom. >>Welcome to Broadcom's DevOps virtual forum, I'm Lisa Martin, and I'm joined by another Martin, very socially distanced from me all the way coming from Birmingham, England is Glynn Martin, the head of QA transformation at BT. Glynn, it's great to have you on the program. Thank you, Lisa. I'm looking forward to it. As we said before, we went live to Martins for the person one in one segment. So this is going to be an interesting segment guys, what we're going to do is Glynn's going to give us a really kind of deep inside out view of devops from an evolution perspective. So Glynn, let's start. Transformation is at the heart of what you do. It's obviously been a very transformative year. How have the events of this year affected the >> transformation that you are still responsible for driving? Yeah. Thank you, Lisa. I mean, yeah, it has been a difficult year. >>Um, and although working for BT, which is a global telecommunications company, um, I'm relatively resilient, I suppose, as a, an industry, um, through COVID obviously still has been affected and has got its challenges. And if anything, it's actually caused us to accelerate our transformation journey. Um, you know, we had to do some great things during this time around, um, you know, in the UK for our emergency and, um, health workers give them unlimited data and for vulnerable people to support them. And that's spent that we've had to deliver changes quickly. Um, but what we want to be able to do is deliver those kinds of changes quickly, but sustainably for everything that we do, not just because there's an emergency. Um, so we were already on the kind of journey to agile, but ever more important now that we are, we are able to do those, that kind of work, do it more quickly. >>Um, and that it works because the, the implications of it not working is, can be terrible in terms of you know, we've been supporting testing centers,  new hospitals to treat COVID patients. So we need to get it right. And then therefore the coverage of what we do, the quality of what we do and how quickly we do it really has taken on a new scale and what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that, you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously, um, deal with the fact that, you know, COVID 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less costs, but having to deliver more value quicker and  to higher quality. So yeah, certainly the finances is, um, on our minds and that's why we need flexible models, cost models that allow us to kind of do growth, but we get that growth by showing that we're delivering value. Um, especially in these times when there are financial challenges on companies. So one of the things that I want to ask you about, I'm again, looking at DevOps from the inside >>Out and the evolution that you've seen, you talked about the speed of things really accelerating in this last nine months or so. When we think dev ops, we think speed. But one of the things I'd love to get your perspective on is we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that you've seen there as, as needing to get, as you said, get things right, but done so quickly to support essential businesses, essential workers. How have you seen that cultural shift? >>Yeah, I think, you know, before test teams for themselves at this part of the software delivery cycle, um, and actually now really our customers are expecting that quality and to deliver for our customers what they want, quality has to be ingrained throughout the life cycle. Obviously, you know, there's lots of buzzwords like shift left. Um, how do we do shift left testing? Um, but for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle that drive automation, drive improvements. I always say that, you know, you're only as good as your lowest common denominator. And one thing that we were finding on our dev ops journey was that we  would be trying to do certain things quick, we had automated build, automated tests. But if we were taking a weeks to create test scripts, or we were taking weeks to manually craft data, and even then when we had taken so long to do it, that the coverage was quite poor and that led to lots of defects later on in the life cycle, or even in our production environment, we just couldn't afford to do that. >>And actually, focusing on continuous testing over the last nine to 12 months has really given us the ability to deliver quickly across the whole life cycle. And therefore actually go from doing a kind of semi agile kind of thing, where we did the user stories, we did a few of the kind of agile ceremonies, but we weren't really deploying any quicker into production because our stakeholders were scared that we didn't have the same control that we had when we had more waterfall releases. And, you know, when we didn't think of ourselves. So we've done a lot of work on every aspect, um, especially from a testing point of view, every aspect of every activity, rather than just looking at automated tests, you know, whether it is actually creating the test in the first place, whether it's doing security testing earlier in the lot and performance testing in the life cycle, et cetera. So, yeah,  it's been a real key thing that for CT, for us to drive DevOps, >>Talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this. Um, you know, there's a thing that I think people will probably call it a customer experience gap, and it reminds me of a Gilbert cartoon, where we start with the requirements here and you're almost like a Chinese whisper effects and what we deliver is completely different. So we think the testing team or the delivery teams, um, know in our teeth has done a great job. This is what it said in the acceptance criteria, but then our customers are saying, well, actually that's not working this isn't working and there's this kind of gap. Um, we had a great launch this year of agile requirements, it's one of the Broadcom tools. And that was the first time in, ever since I remember actually working within BT, I had customers saying to me, wow, you know, we want more of this. >>We want more projects to have extra requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do we actually, you know, do that and have something that both the business and technical people can understand. And we've actually been working with the business , using agile requirements designer to really look at what the requirements are, tease out requirements we hadn't even thought of and making sure that we've got high levels of test coverage. And what we actually deliver at the end of it, not only have we been able to generate tests more quickly, but we've got much higher test coverage and also can more smartly, using the kind of AI within the tool and then some of the other kinds of pipeline tools, actually deliver to choose the right tasks, and actually doing a risk based testing approach. So that's been a great launch this year, but just the start of many kinds of things that we're doing >>Well, what I hear in that, Glynn is a lot of positives that have come out of a very challenging situation. Talk to me about it. And I liked that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration perspective you're right, we talk about that a lot critical with devops. But those challenges there, you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pivot so fast? >>I mean, you talked about culture. You know, BT is like most companies  So it's very siloed. You know we're still trying to work to become closer as a company. So I think there's a lot of challenges around how would you integrate with other tools? How would you integrate with the various different technologies. And BT, we have 58 different IT stacks. That's not systems, that's stacks, all of those stacks can have hundreds of systems. And we're trying to, we've got a drive at the moment, a simplified program where we're trying to you know, reduce that number to 14 stacks. And even then there'll be complexity behind the scenes that we will be challenged more and more as we go forward. How do we actually highlight that to our users? And as an it organization, how do we make ourselves leaner, so that even when we've still got some of that legacy, and we'll never fully get rid of it and that's the kind of trade off that we have to make, how do we actually deal with that and hide that from our users and drive those programs, so we can, as I say, accelerate change,  reduce that kind of waste and that kind of legacy costs out of our business. You know, the other thing as well, I'm sure telecoms is probably no different to insurance or finance. When you take the number of products that we do, and then you combine them, the permutations are tens and hundreds of thousands of products. So we, as a business are trying to simplify, we are trying to do that in an agile way. >>And haven't tried to do agile in the proper way and really actually work at pace, really deliver value. So I think what we're looking more and more at the moment is actually  more value focused. Before we used to deliver changes sometimes into production. Someone had a great idea, or it was a great idea nine months ago or 12 months ago, but actually then we ended up deploying it and then we'd look at the users, the usage of that product or that application or whatever it is, and it's not being used for six months. So we haven't got, you know, the cost of the last 12 months. We certainly haven't gotten room for that kind of waste and, you know, for not really understanding the value of changes that we are doing. So I think that's the most important thing of the moment, it's really taking that waste out. You know, there's lots of focus on things like flow management, what bits of our process are actually taking too long. And we've started on that journey, but we've got a hell of a long way to go. But that involves looking at every aspect of the software delivery cycle. >> Going from, what 58 IT stacks down to 14 or whatever it's going to be, simplifying sounds magical to everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we were started on a continuous testing journey, and I think that's just the start. I mean as I say, looking at every aspect of, you know, from a QA point of view is every aspect of what we do. And it's also looking at, you know, we've started to branch into more like AI, uh, AI ops and, you know, really the full life cycle. Um, and you know, that's just a stepping stone to, you know, I think autonomics is the way forward, right. You know, all of this kind of stuff that happens, um, you know, monitoring, uh, you know, watching the systems what's happening in production, how do we feed that back? How'd you get to a point where actually we think about change and then suddenly it's in production safely, or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey, but if we want to, you know, in a world where the pace is in ever-increasing and the demands for the team, and, you know, with the pressures on, at the moment where we're being asked to do things, uh, you know, more efficiently and as lean as possible, we need to be thinking about every part of the process and how we put the kind of stepping stones in place to lead us to a more automated kind of, um, you know, um, the future. >>Do you feel that that planned outcomes are starting to align with what's delivered, given this massive shift that you're experiencing? >>I think it's starting to, and I think, you know, as I say, as we look at more of a value based approach, um, and, um, you know, as I say, print, this was a kind of flow management. I think that that will become ever, uh, ever more important. So, um, I think it starting to people certainly realize that, you know, teams need to work together, you know, the kind of the cousin between business and it, especially as we go to more kind of SAS based solutions, low code solutions, you know, there's not such a gap anymore, actually, some of our business partners that expense to be much more tech savvy. Um, so I think, you know, this is what we have to kind of appreciate what is its role, how do we give the capabilities, um, become more of a centers of excellence rather than actually doing mounds amounts of work. And for me, and from a testing point of view, you know, mounds and mounds of testing, actually, how do we automate that? How do we actually generate that instead of, um, create it? I think that's the kind of challenge going forward. >>What are some, as we look forward, what are some of the things that you would like to see implemented or deployed in the next, say six to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think, um, you know, certainly for, for where we are as a company from a QA perspective, we are, um, you let's start in bits that we do well, you know, we've started creating, um, continuous delivery and DevOps pipelines. Um, there's still manual aspects of that. So, you know, certainly for me, I I've challenged my team with saying how do we do an automated journey? So if I put a requirement in JIRA or rally or wherever it is and why then click a button and, you know, with either zero touch for one such, then put that into production and have confidence that, that has been done safely and that it works and what happens if it doesn't work. So, you know, that's, that's the next, um, the next few months, that's what our concentration, um, is, is about. But it's also about decision-making, you know, how do you actually understand those value judgments? >>And I think there's lots of the things dev ops, AI ops, kind of that always ask aspects of business operations. I think it's about having the information in one place to make those kinds of decisions. How does it all try and tie it together? As I say, even still with kind of dev ops, we've still got elements within my company where we've got lots of different organizations doing some, doing similar kinds of things, but they're all kind of working in silos. So I think having AI ops as it comes more and more to the fore as we go to cloud, and that's what we need to, you know, we're still very early on in our cloud journey, you know, so we need to make sure the technologies work with cloud as well as you can have, um, legacy systems, but it's about bringing that all together and having a full, visible pipeline, um, that everybody can see and make decisions. >>You said the word confidence, which jumped out at me right away, because absolutely you've got to have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question then for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops, to be able to gain the confidence that they're making the right decisions for their business? >>I think the, the, the, the, the approach that we've taken actually is not started with technology. Um, we've actually taken a human centered design, uh, as a core principle of what we do, um, within the it part of BT. So by using human centered design, that means we talk to our customers, we understand their pain points, we map out their current processes. Um, and then when we mapped out what this process does, it also understand their aspirations as well, you know? Um, and where do they want to be in six months? You know, do they want it to be, um, more agile and, you know, or do they want to, you know, is, is this a part of their business that they want to do one better? We actually then looked at why that's not running well, and then see what, what solutions are out there. >>We've been lucky that, you know, with our partnership, with Broadcom within the payer line, lots of the tools and the PLA have directly answered some of the business's problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which in, in, uh, you know, in some companies include not as they do there is that kind of, you know, almost by understanding their, their pain points and then starting, this is how we can solve your problem. Um, is we've, we've tended to be much more successful than trying to impose something and say, well, here's the technology that they don't quite understand. It doesn't really understand how it kind of resonates with their problems. So I think that's the heart of it. It's really about, you know, getting, looking at the data, looking at the processes, looking at where the kind of waste is. >>And then actually then looking at the right solutions. Then, as I say, continuous testing is massive for us. We've also got a good relationship with Apple towards looking at visual AI. And actually there's a common theme through that. And I mean, AI is becoming more and more prevalent. And I know, you know, sometimes what is AI and people have kind of this semantics of, is it true AI or not, but it's certainly, you know, AI machine learning is becoming more and more prevalent in the way that we work. And it's allowing us to be much more effective, be quicker in what we do and be more accurate. And, you know, whether it's finding defects running the right tests or, um, you know, being able to anticipate problems before they're happening in a production environment. >>Well, thank you so much for giving us this sort of insight outlook at dev ops sharing the successes that you're having, taking those challenges, converting them to opportunities and forgiving folks who might be in your shoes, or maybe slightly behind advice enter. They appreciate it. We appreciate your time. >>Well, it's been an absolute pleasure, really. Thank you for inviting me. I have a extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glenn Martin. I'm Lisa Martin. You're watching the cube >>Driving revenue today means getting better, more valuable software features into the hands of your customers. If you don't do it quickly, your competitors as well, but going faster without quality creates risks that can damage your brand destroy customer loyalty and cost millions to fix dev ops from Broadcom is a complete solution for balancing speed and risk, allowing you to accelerate the flow of value while minimizing the risk and severity of critical issues with Broadcom quality becomes integrated across the entire DevOps pipeline from planning to production, actionable insights, including our unique readiness score, provide a three 60 degree view of software quality giving you visibility into potential issues before they become disasters. Dev ops leaders can manage these risks with tools like Canary deployments tested on a small subset of users, or immediately roll back to limit the impact of defects for subsequent cycles. Dev ops from Broadcom makes innovation improvement easier with integrated planning and continuous testing tools that accelerate the flow of value product requirements are used to automatically generate tests to ensure complete quality coverage and tests are easily updated. >>As requirements change developers can perform unit testing without ever leaving their preferred environment, improving efficiency and productivity for the ultimate in shift left testing the platform also integrates virtual services and test data on demand. Eliminating two common roadblocks to fast and complete continuous testing. When software is ready for the CIC CD pipeline, only DevOps from Broadcom uses AI to prioritize the most critical and relevant tests dramatically improving feedback speed with no decrease in quality. This release is ready to go wherever you are in your DevOps journey. Broadcom helps maximize innovation velocity while managing risk. So you can deploy ideas into production faster and release with more confidence from around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of dev ops and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high-performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago. So it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate physical co-location. Okay. That makes sense. >>It does. I shouldn't have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with jump. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other understanding, Oh, individuals teams, uh, contributed to kind of a business outcome, what is our shared goal or shared vision? What's what is it we're trying to achieve collectively and keeping it kind of aligned to that? Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organization, there's been specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of reverted back to, okay, what is my database all the time? What is my cycle time? >>Right. And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens or these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, favorites experience was to actually weakness at one of a large financial institution. Um, you know, Tuesday Golder's unquote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, test execution results, you know, Coverity coverage, um, official liabilities and all the all ran. It could have a direction level links. And that's when you start to put these things in context and represent that to you in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and, and understand have they joined our company to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And w I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it can, it says simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos. We were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very dev, uh, different environments, depending on what area of BT you're in, whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can, um, drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. >>And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in their kind of testing world at the moment. But, you know, I could talk about things like continuous delivery direct to when a ball comes tools and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum it up, autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter? Jeff, >>I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we look at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test, uh, departments, a group, the fourth level that I think is really important, uh, in, in the current, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric helpful was the parking lot at nine. And how full was it at five o'clock. I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customers. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big impact. And, and, and also how do you measure quality kind of following onto the advice that Jeff provided? >>That's some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker, uh, called me Kirsten a couple of months ago. It taught very much around how important flow management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes long? So I think that's a very important thing for us. It's even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, you know, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how, how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up our teams, training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are, um, delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, Jeff's point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. >>And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about our, these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood is that, you know, when you have a bad cycle time or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, if you look at, for instance, things like lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. >>And so I think that there is, you know, the only the metrics there is, I think a lot of work that we need to do in terms of classifying, descend type patterns, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as, as a key pattern, the industry or anti-spatter. Um, but waterscrumfall right is a key one, right? And you will detect that through kind of a defect arrival rates. That's where that looks like an S-curve. And so I think it's beyond kind of the, the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on this. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff will go back to you. Okay. >>I would say look for the opportunities that this disruption presents. And there are a couple that I see, first of all, uh, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks to be on the lookout for now? I, I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but low code also. Um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the, the life cycle, um, where, when I heard the word scrum fall, it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment we need to get beyond that. We need >>To be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production and go straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately we all need some peanut butter and chocolate. All right. It starts to take us home. That's what's that nugget you think everyone needs to have on their agendas? >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about the lost city, but it's all about sweet. We still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of a spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value into aligning everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>> Announcer: From around the globe. It's theCUBE. With digital coverage of BizOps Manifesto Unveiled. Brought to you by BizOps Coalition. >> Hey, welcome back everybody ,Jeff Frick here with theCUBE. Welcome back to our ongoing coverage of the BizOps Manifesto Unveiled. Something has been in the works for a little while. Today's the formal unveiling and we're excited to have three of the core founding members of the manifesto, authors of the manifesto, if you will. And joining us again, we've had them all on individually, now we're going to have a great power panel. First up, we're going to have Mik Kersten returning. He's the founder and CEO of Tasktop. Mik, good to see you again. Where are you dialing in from? >> Great to see you again, Jeff. I'm dialing from Vancouver, Canada. >> Vancouver, Canada. One of my favorite cities in the whole wide world. Also we've got Tom Davenport, coming in from across the country. He's a distinguished professor and author from Babson College. Tom, great to see you. And I think you said you're at a fun exotic place on the East Coast. >> From Massachusetts, Cape Cod. >> Nice, great to see you again. And also joining Serge Lucio. He is the VP and General Manager Enterprise Software Division at Broadcom. Serge, great to see you again, where are you coming in from? >> From Boston right next to Cape Cod. >> Terrific. So welcome back, everybody again. Congratulations on this day. I know it's been a lot of work to get here for this unveil. But let's just jump into it. BizOps Manifesto, what was the initial reason to do this? And how did you decide to do it in a kind of a coalition, way bringing together a group of people versus just making it an internal company initiative that you know, you can do better stuff within your own company? Serge, why don't we start with you? >> Yeah, so I think we were at a really critical juncture, right. Many large enterprises are basically struggling with their digital transformation. In fact, many recognized that the business (indistinct) collaboration has been one of the major impediments to drive that kind of transformation. And if we look at the industry today, many people are, whether we're talking about vendors or system decorators, consulting firms, are talking about the same kind of concepts, but using very different language. And so we believe that bringing all these different players together as part of the coalition and formalizing, basically the core principles and values in a BizOps Manifesto, we can really start to kind of have a much bigger movement where we can all talk about kind of the same concepts and we can really start to provide, could have a much better support for large organizations to transform. So whether it is technology or services or training, I think that's really the value of bringing all of these players together. >> Great. And Mik to you. Why did you get involved in this effort? >> So I've been close and follow the agile movement since it started two decades ago with that manifesto. And I think we got a lot of improvement at the team level and I think as Serge has noted, we really need to improve at the business level. Every company is trying to become a software innovator, trying to make sure that they can pivot that quickly and then changing market economy and what everyone's dealing with in terms of needing to deliver value to customers sooner. However, agile practices have really focused that these metrics, these measures and understanding processes that help teams be productive. Those things now need to be elevated to the business as a whole. And that just hasn't happened. Organizations are actually failing because they're measuring activities and how they're becoming more agile, how teams are functioning not how much quickly they're delivering value to the customer. So we need to now move past that. And that's exactly what the BizOps Manifesto provides. >> Right, great And Tom to you, you've been covering tech for a very very long time. You've been looking at really hard challenges and a lot of work around analytics and data and data evolution. So there's a definitely a data angle here. I wonder if you could kind of share your perspective of what you got excited to sign onto this manifesto. >> Sure. Well, I have, you know, for the past 15 or 20 years, I've been focusing on Data Analytics and AI, but before that I was a process management guy and a knowledge management guy. And in general, I think, you know we've just kind of optimize that to narrow a level whether you're talking about agile or DevOps or MLops, any of these kind of ops oriented movements. We're making individual project performance and productivity better but we're not changing the business effectively enough. And that's the thing that appealed to me about the BizOps idea that we're finally creating a closer connection between what we do with technology and how it changes the business and provides value to it. >> That's great. Serge back to you, right. I mean, people have been talking about digital transformation for a long time and it's been you know, kind of trucking along and then COVID hit and it was instant light switch. Everyone's working from home, you've got a lot more reliance on your digital tools, digital communication, both within your customer base and your partner base but also then your employees. One of you can share how that really pushed this all along, right. Because now suddenly the acceleration of digital transformation is higher. Even more importantly, you got much more critical decisions to make into what you do next. So kind of your portfolio management of projects has been elevated significantly when maybe revenues are down and you really have to prioritize and get it right. >> Yeah. Maybe I'll just start by quoting Satina Nello, basically recently said that there's been two years of digital transformation just last two months. And in any many ways that's true. But yet when we look at large enterprises, they're still struggling with a kind of a changes in culture. That they really need to drive to be able to disrupt themselves. And not surprisingly you know, when we look at certain parts of the industry you know, we see some things which are very disturbing, right? About 40% of the personal loans today, are being originated by fintechs of a like of Sophie or LendingClub, right? Not to traditional brick and mortar for a bank. And so the, well, there is kind of a much more of an appetite and it's a more of a survival type of driver these days. The reality is that in order for these large enterprises to truly transform and engage on this digital transformation they need to start to really align the business in IT. You know, in many ways and make cover that agile really emerge from the core desire to truly improve software predictability which we've really missed is all that we start to aligning the software predictability to business predictability and to be able to have continual sleep continuous improvement and measurement of business outcomes. So by aligning that of these discuss inward metrics that's, IT is typically being using to business outcomes. We think we can start to really help different stakeholders within the organization to collaborate. So I think there is more than ever. There's an imperative to acts now and resolves I think is kind of the right approach to drive that kind of transformation. >> Great. I want to follow up on the culture comment with you, Tom because you've talked before about kind of process flow and process flow throughout a whore and an organization. And, you know, we talk about people process and tech all the time. And I think the tech is the easy part compared to actually changing the people the way they think. And then the actual processes that they put in place. It's a much more difficult issue than just the tech issue to get this digital transformation in your organization. >> Yeah. You know, I've always found that the soft stuff about, you know, the culture of a behavior the values is the hard stuff to change and more and more we realized that to be successful with any kind of digital transformation you have to change people's behaviors and attitudes. We haven't made as much progress in that area as we might have. I mean, I've done some surveys suggesting that most organizations still don't have data driven cultures. And in many cases there is a lower percentage of companies that say they have that then did a few years ago. So we're kind of moving in the wrong direction, which means I think that we have to start explicitly addressing that cultural, behavioral dimension and not just assuming that it will happen if we build system. You know, if we build it, they won't necessarily come. >> Right. So I want to go to you Nick. 'Cause you know, we're talking about workflows and flow and, and you've written about flow both in terms of, you know, moving things along a process and trying to find bottlenecks, identify bottlenecks which is now even more important again when these decisions are much more critical 'cause you have a lot less wiggle room in tough times, but you also talked about flow from the culture side and the people side. So, I wanted if you can just share your thoughts on, you know, using flow as a way to think about things, to get the answers better. >> Yeah, absolutely. And I'll refer back to what Tom has said. If you're optimized, you need to optimize your system. You need to optimize how you innovate and how you deliver value to the business and the customer. Now, what we've noticed in the data, since that we've learned from customers, value streams, enterprise organizations value streams, is that when it's taking six months at the end to deliver that value with the flow is that slow. You've got a bunch of unhappy developers unhappy customers when you're innovating house. So high performing organizations we can measure their end flow time and dates. All of a sudden that feedback loop the satisfaction your developer's measurably goes up. So not only do you have people context, switching glass you're delivering so much more value to customers at a lower cost because you've optimized for flow rather than optimizing for these other approximate tricks that we use which is how efficient is my agile team. How quickly can we deploy software? Those are important, but they do not provide the value of agility of fast learning of adaptability to the business. And that's exactly what the BizOps Manifesto pushes your organization to do. You need to put in place this new operating model that's based on flow on the delivery of business value and on bringing value to market much more quickly than you were before. >> Right. I love that. And I'm going back to you, Tom, on that to follow up 'cause I think, I don't think people think enough about how they prioritize what they're optimizing for 'cause you know if you're optimizing for A versus B, you know you can have a very different product that you kick out and let you know. My favorite example is with Clayton Christensen and innovator's dilemma talking about the three inch hard drive. If you optimize it for power, you know, is one thing if you optimize it for vibration is another thing and sure enough, you know, they missed it on the poem because it was the game console which drove that whole business. So when you when you're talking to customers and we think we hear it with cloud all the time people optimizing for a cost efficiency instead of thinking about it as an innovation tool. How do you help them kind of rethink and really, you know, force them to look at the prioritization and make sure they're prioritizing on the right thing is make just said what are you optimizing for? >> Oh yeah, you have one of the most important aspects of any decision or attempt to resolve a problem in an organization is the framing process. And you know, it's a difficult aspect to the decision to frame it correctly in the first place. There, it's not a technology issue. In many cases, it's largely a human issue, but if you frame that decision or that problem incorrectly to narrowly say, or you frame it as an either or situation where you could actually have some of both, it's very difficult for the process to work out correctly. So in many cases that I think we need to think more at the beginning about how we bring this issue or this decision in the best way possible before we charge off and build a system to support it. You know, it's worth that extra time to think carefully about how the decision has been structured. >> Right. Serge, I want to go back to you and talk about the human factors, because as we've just discussed, you could put it in great technology, but if the culture doesn't adopt it and people don't feel good about it, you know, it's not going to be successful and that's going to reflect poorly on the technology, even if it had nothing to do with it. And you know, when you look at the core values of the Bezos Manifesto, you know, a big one is trust and collaboration, you know, learn, respond and pivot. One of you can share your thoughts on trying to get that cultural shift so that you can have success with the people or excuse me, with the technology in the process and helping customers, you know, take this more trustworthy and kind of proactive position. >> So I think, at the ground level, it truly starts with the realization that we're all different. We come from different backgrounds. Often times we tend to blame the data. It's not uncommon my experiments that we spend the first you know 30 minutes of any kind of one hour conversation to debate the validity of the data. And so one of the first kind of probably manifestations that we've had or revelations as we start to engage with our customers is like just exposing high-fidelity data sets to different stakeholders from their different lens. We start to enable these different stakeholders to not debate the data. That's really collaborate to find a solution. So in many ways, when we think about kind of the types of changes that we're trying to truly effect around data driven decision making it's all about bringing the data in context, the context that is relevant and understandable for different stakeholders, whether we're talking about an operator or a developer or a business analyst. So that's, the first thing. The second layer I think, is really to provide context to what people are doing in their specific cycle. And so I think one of the best examples I have is if you start to be able to align business KPI whether you are counting you know, sales per hour, or the engagements of your users on your mobile applications, whatever it is. You can start to connect that KPI to business KPI to the KPIs that developers might be looking at, whether it is the number of defects or a velocity or whatever, you know metrics that they are used to actually track. You start to be able to actually contextualize in what we are the effecting, basically a metric that is really relevant in which we see is that this is a much more systematic way to approach the transformation than say, you know, some organizations kind of creating some of these new products or services or initiatives to drive engagements, right? So if you look at zoom for instance, zoom giving away it's service to education, is all about, I mean, there's obviously a marketing aspect in therapists. It's fundamentally about trying to drive also the engagement of their own teams. And because now they're doing something for good and the organizations are trying to do that. But you only can do this kind of things in a limited way. And so you really want to start to rethink how you connect to everybody's kind of a business objective through data and now you start to get people to stare at the same data from their own lens and collaborate on all the data. >> Right, great That's a good. Tom I want to go back to you. You've been studying IT for a long time, writing lots of books and getting into it. Why now, you know, what why now (laughs) are we finally aligning business objectives with IT objectives? You know, why didn't this happen before? And you know, what are the factors that are making now the time for this move with the BizOps? >> Well, much of a past, IT was sort of a back office related activity. And, you know, it was important for producing your pay check and capturing the customer orders but the business wasn't built around it. Now, every organization needs to be a software business data business a digital business, the auntie has been raised considerably. And if you aren't making that connection between your business objectives and the technology that supports it you run a pretty big risk of, you know going out of business or losing out to competitors totally. So, and even if you're you know, an industry that hasn't historically been terribly technology oriented customer expectations flow from, you know, the digital native companies that they work with to basically every industry. So you're compared against the best in the world. So we don't really have the luxury anymore of screwing up our IT projects or building things that don't really work for the business. It's mission critical that we do that well almost every time. >> Right. And I just want to follow up by that, Tom In terms of the, you've talked extensively about kind of these evolutions of data and analytics from artisanal stage to the big data stage, the data economy stage the AI driven stage and what I find diff interesting that all those stages, you always put a start date. You never put an end date. So, you know, is the big data I'm just going to use that generically moment in time, finally here, where we're you know, off mahogany row with the data scientists but actually can start to see the promise of delivering the right insight to the right person at the right time to make that decision. >> Well, I think it is true that in general, these previous stages never seemed to go away. The artisanal stuff is still being done but we would like for less and lesser of it to be artisanal, we can't really afford for everything to be artisanal anymore. It's too labor and time consuming to do things that way. So we shift more and more of it to be done through automation and to be done with a higher level of productivity. And, you know at some point maybe we reached the stage where we don't do anything artisanally anymore. I'm not sure we're there yet but you know, we are making progress. >> Right And Mick, back to you in terms of looking at agile 'cause you're such a student of agile, when you look at the opportunity with BizOps and taking the lessons from agile, you know what's been the inhibitor to stop this in the past. And what are you so excited about? You know, taking this approach will enable. >> Yeah. I think both Serge and Tom hit on this is that in agile what's happened is that we've been you know measuring tiny subsets of the value stream right. We need to elevate the data's there. Developers are working on these tools that delivering features that the foundations for great culture are there. I spent two decades as a developer. And when I was really happy is when I was able to deliver value to customers, the quicker I was able to do that the fewer impediments are in my way the quicker was deployed and running in the cloud the happier I was, and that's exactly what's happening. If we can just get the right data elevated to the business, not just to the agile teams but really these values of ours are to make sure that you've got these data driven decisions with meaningful data that's oriented around delivering value to customers. Not only these legacies that Tom touched on, which has cost center metrics from an IT, for IT being a cost center and something that provided email and then back office systems. So we need to rapidly shift to those new meaningful metrics that are customized business centric and make sure that every developer the organization is focused on those as well as the business itself, that we're measuring value and we're helping that value flow without interruptions. >> I love that Mik 'cause if you don't measure it, you can't improve on it but you got to be measuring the right thing. So gentlemen, thank you again for your time. Congratulations on the unveil of the BizOps Manifesto and bringing together this coalition of industry experts to get behind this. And you know there's probably never been a more important time than now to make sure that your prioritization is in the right spot and you're not wasting resources where you're not going to get the ROI. So congratulations again. And thank you for sharing your thoughts with us here on theCUBE. >> Thank you. >> Thank you from Vancouver. >> Alright, so we had Serge, Tom and Mik. I'm Jeff, you're watching theCUBE. It's a BizOps Manifesto Unveiled. Thanks for watching. We'll see you next time. (soft music)

>>long from Las Vegas. It's the Q covering a ws re invent 2019. Brought to you by Amazon Web service is and in along with its ecosystem partners. >>Hey, welcome back to the Cube. Lisa Martin at AWS Reinvent 19. This is Day three of the Cubes coverage. We have two sets here. Lots of cute content are joined by Justin Warren, the founder and chief analyst at Pivot nine. Justin. How's it going? Great, right? You still have a voice? Three days? >>Just barely. I've been I've been trying to take care of it. >>Impressed. And you probably have talked to at least half of the 65,000 attendees. >>I'm trying to talk to as many as I can. >>Well, we're gonna talk to another guy here. Joining us from data ICU is well, Novak, the solutions architect will be the Cube. >>Thanks for having me. >>You have a good voice too. After a three day is that you >>have been doing the best I can. >>Yeah, he's good. So did ICU. Interesting name. Let's start off by sharing with our audience. Who did a coup is and what you guys do in technology. >>Yes. So the Entomology of date ICU. It's like hi cooze for data. So we say we take your data and, you know, we make poetry out of it. Make your data so beautiful. Wow, Now, But for those who are unaware Day like it was an enterprise data science platform. Eso we provide a collaborative environment for we say coders and clickers kind of business analyst and native data scientists to make use of organizations, data bill reports and Bill productive machine learning base models and deploy them. >>I'm only the guy's been around around for eight years. Eight years. Okay, >>so start up. Still >>mourning the cloud, the opportunity there That data is no longer a liability. It's an asset or should be. >>So we've been server based from the start, which is one of our differentiators. And so by that we see ourselves as a collaborative platform. Users access it through a Web browser, log into a shared space and share code, can share visual recipes, as we call them to prepare data. >>Okay, so what customers using the platform to do with machine learning is pretty hot at the moment. I think it might be nearing the peak of the life cycle pretty hot. Yeah, what a customer is actually actually doing on the platform, >>you know, So we really focus on enabling the enterprise. So, for example, G has been a customer for some time now, and Sergey is a great prototypical example on that. They have many disparate use cases, like simple things like doing customer segmentation for, you know, marketing campaigns but also stuff like Coyote predicted maintenance. So use cases kind of run the gamut, and so did ICU. Based on open source, we're enabling all of G's users to come into a centralized platform, access their data manipulated for whatever purposes. Maybe >>nobody talked about marketing campaigns for a second. I'm wondering. Are, is their integration with serum technologies? Or how would a customer like wanting to understand customer segmentation or had a segment it for marketing campaign? How would they work in conjunction with a serum and data ICU, for example? >>It's a great question. So again, us being a platform way sit on a single server, something like an Amazon ec2 instance, and then we make connections into an organization's data sources. So if using something like Salesforce weaken seamlessly, pull in data from Salesforce Yuka manipulated in date ICU, but the same time. Maybe also have some excel file someone you know me. I can bring that into my data to work environment. And I also have a red shift data table. All those things would come into the same environment. I can visualize. I can analyze, and I can prepare the data. I see. >>So you tell you it's based on open source? I'm a longtime fan of over. It's always been involved in it for longer than I care to remember. Actually, that's an interesting way t base your product on that. So maybe talk us through how you how you came to found the company based on basic an open source. What? What led to that choice? What? What was that decision based on? >>Yeah, for sure. So you talked about how you know the hype cycle? A. I saw how hot is a I and so I think again, our founders astutely recognize that this is a very fast moving place to be. And so I'm kind of betting on one particular technology can be risky. So instead, by being a platform, we say, like sequel has been the data transformation language do jour for many days now. So, of course, that you can easily write Sequel and a lot of our visual data Transformations are based on the sequel language, but also something like Python again. It's like the language de jour for machine law machine learning model building right now, so you can easily code in python. Maintain your python libraries in date, ICU And so by leveraging open source, we figured we're making our clients more future proof as long as they're staying in date ICU. But using data ICU to leverage the best in breed and open source, they'll always be kind of where they want to be in the technological landscape by supposed to locked into some tech that is now out of date. >>What's been the appetite for making data beautiful for a legacy enterprise, like a G E that's been around for a very long time versus a more modern either. Born in the Cloud er's our CEO says, reborn in the cloud. What are some of the differences but also similarities that you see in terms of we have to be able to use emerging tech. Otherwise someone's gonna come in behind us and replace us. >>Yeah, I mean, I think it's complicated in that there's still a lot of value to be had in someone says, like a bar chart you can rely on right, So it's maybe not sexy. But having good reporting and analytics is something that both you know, 200 year old enterprise organizations and data native organizations startups needs. At the same time, building predicted machine learning models and deploying those is rest a p i n points that developers can use in your organization to provide a data driven product for your consumers. Like that's amore advanced use case that everyone kind of wants to be a part of again data. Who's a nice tool, which says Maybe you don't have developers who are very fluent in turning out flashed applications. We could give you a place to build a predictive model and deploy that predictive model, saving you time to write all that code on the back end. >>One of the themes of the show has been transformation, so it sounds like data ICU would be It's something that you can dip your toes in and start to get used to using. Even if you're not particularly familiar with Time machine learning model a model building. >>Yeah, that's exactly right. So a big part of our product and encourage watchers to go try it out themselves and go to our website. Download a free version pretrial, but is enablement. So if you're the most sophisticated applied math PhD there is, like, Who's a great environment for you to Code and Bill predictive models. If you never built the machine learning model before you can use data ICU to run visual machine learning recipes, we call them, and also we give you documentation, which is, Hey, this is a random forest model. What is a random forest model? We'll tell you a little bit about it. And that's another thing that some of these enterprises have really appreciated about date I could. It is helping up skill there user base >>in terms of that transformation theme that Justin just mention which we're hearing a lot about, not visit this show. It's a big thing, but we hear it all the time, right? But in terms of customers transformation, journey, whatever you wanna call it, cloud is gonna be an essential enabler of being able to really love it value from a I. So I'm just wondering from a strategic positioning standpoint. Is did ICU positioned as a facilitator or as fuel for a cloud transformation that on enterprise would undergo >>again? Yes, great point. So for us, I can't take the credit. This credit goes to our founders, but we've thought from the start the clouds and exciting proposition Not everyone is. They're still in 2019. Most people, if not all of them, want to get there. Also, people want too many of our clients want the multi cloud on a day. Like who says, If you want to be on prim, if you want to be in a single cloud subscription. If you want to be multi cloud again as a platform, we're just gonna give you connection to your underlying infrastructure. You could use the infrastructure that you like and just use our front end to help your analyst get value. They can. I >>think I think a lot of vendors across the entire ecosystem around to say the customer choice is really important, and the customers, particularly enterprise customers, want to be able to have lots of different options, and not all of them will be ready to go completely. All in on cloud today. They made it may take them years, possibly decades, to get there. So having that choice is like it's something that it would work with you today and we'll work with you tomorrow, depending on what choices you make. >>It's exactly right. Another thing we've seen a lot of to that day, like who helps with and whether it's like you or other tools. Like, of course, you want best in breed, but you also want particularly for a large enterprise. You don't want people operating kind of in a wild West, particularly in like the ML data science space. So you know we integrate with Jupiter notebooks, but some of our clients come to us initially. Just have I won't say rogues that has a negative connotation. But maybe I will say Road road data Scientists are just tapping into some day the store. They're using Jupiter notebooks to build a predictive model, but then to actually production allies that to get sustainable value out of it like it's to one off and so having a centralized platform like date ICU, where you can say this is where we're going to use our central model depository, that something where businesses like they can sleep easier at night because they know where is my ML development happening? It's happening in one ecosystem. What tools that happening with, well, best in breed of open source. So again, you kind of get best of both worlds like they like you. >>It sounds like it's more about the operations of machine learning. It is really, really important rather than just. It's the pure technology. Yes, that's important as well, and you need to have the data Sinus to build it, but having something that allows you to operationalize it so that you can just bake it into what we do every day as a business. >>Yeah, I think in a conference like this all about tech, it's easy to forget what we firmly believe, which is a I and maybe tech. More broadly, it's still human problems at the core, right? Once you get the tech right, the code runs corrected. The code is written correctly. Therefore, like human interactions, project management model deployment in an organization. These are really hard, human centered problems, but so having tech that enables that human centric collaboration helps with that, we find >>Let's talk about some of the things that we can't ever go to an event and not talk about. Nut is respected data quality, reliability and security. Understood? I could facilitate those three cornerstones. >>Yeah, sure. So, again, viewers, I would encourage you to check out the date. ICU has some nice visual indications of data quality. So an analyst or data scientists and come in very easily understand, you know, is this quality to conform to the standards that my organization has set and what I mean by standards that could be configured. Right? So does this column have the appropriate schema? Does it have the appropriate carnality? These are things that an individual might decide to use on then for security. So Data has its own security mechanisms. However, we also to this point about incorporating best Retek. We'll work with whatever underlying security mechanisms organizations organizations have in place. So, for instance, if you're using a W s, you have, I am rolls to manage your security. Did ICU comport those that apply those to the date ICU environment or using something like on prime miss, uh, duke waken you something like Kerberos has the technology to again manage access to resources. So we're taking the best in breed that this organization already has invested time, energy and resources into and saying We're not trying to compete with them but rather were trying to enable organizations to use these technologies efficiently. >>Yeah, I like that consistency of customer choice. We spoke about that just before. I'm seeing that here with their choices around. Well, if you're on this particular platform will integrate with whatever the tools are there. People underestimate how important that is for enterprises, that it has to be ahead. Virginia's environment, playing well with others is actually quite important. >>Yeah, I don't know that point. Like the combination of heterogeneity but also uniformity. It's a hard balance to strike, and I think it's really important, giving someone a unified environment but still choice. At the same time. A good restaurant or something like you won't be able to pick your dish, but you want to know that the entire quality is high. And so having that consistent ecosystem, I think, really helps >>what are, in your opinion, some of the next industries that you see there really right to start Really leveraging machine learning to transfer You mentioned g e a very old legacy business. If we think of you know what happened with the ride hailing industry uber, for example, or fitness with Saletan or pinchers with visible Serge, what do you think is the next industry? That's like you guys taking advantage of machine learning will completely transform this and our lives. >>I mean, the easy answer that I'll give because it's easy to say it's gonna transform. But hard to operationalize is health care, right? So there is structured data, but the data quality is so desperate and had a row genius s, I think you know, if organizations in a lot of this again it's a human centered problem. If people could decide on data standards and also data privacy is, of course, a huge issue. We talked about data security internally, but also as a customer. What day to do I want you know, this hospital, this health care provider, to have access to that human issues we have to result but conditional on that being resolved that staring out a way to anonymous eyes data and respect data privacy but have consistent data structure. And we could say, Hey, let's really set these a I M L models loose and figure out things like personalized medicine which were starting to get to. But I feel like there's still a lot of room to go. That >>sounds like it's exciting time to be in machine learning. People should definitely check out products such as Dead Rock you and see what happens. >>Last question for you is so much news has come out in the last three days. It's mind boggling sum of the takeaways, that of some of the things that you've heard from Andy Jassy to border This'll Morning. >>Yeah, I think a big thing for me, which was something for me before this week. But it's always nice to hear an Amazon reassures the concept of white box. Aye, aye. We've been talking about that a date ICU for some time, but everyone wants performance A. I R ml solutions, but increasing. There's a really appetite publicly for interpret ability, and so you have to be responsible. You have to have interpret belay I and so it's nice to hear a leader like Amazon echo that day like you. That's something we've been talking about since our start. >>A little bit validating them for data ICU, for sure, for sure. Well, thank you for joining. Just to be on the kid, the suffering. And we appreciate it. Appreciate it. All right. For my co host, Justin Warren, I'm Lisa Martin and your work to the Cube from Vegas. It's AWS reinvent 19.