>>from around the globe. It's >>the Cube with digital coverage of Dev >>Ops Virtual Forum Brought to You by Broadcom. Welcome to Broadcom, Step Ups, Virtual Forum I and Lisa Martin and I'm joined by another Martin very socially. Distance from me all the way. Coming from Birmingham, England, is Glynn Martin, head of Q. A transformation at BT Glenn. It's great to have you on the program. >>Thank you, Lisa. I'm looking forward, Toa. >>As we said before, we went live to Martin's for the price of one in one segment. So this is gonna be an interesting segment, Guesses. What we're gonna do is Glen's gonna give us a really kind of deep inside out view of Dev ops. From an evolution perspective, Soglo's Let's start transformation is at the heart of what you dio. It's obviously been a very transformative year. How have the events of this year affected the transformation that you are so responsible for driving? >>Yeah. Thank you, Leigh. So I mean, yeah, it has been a difficult year Bond, although working for BT, which is ah, global telecommunications company. Relatively resilient, I suppose, as an industry through covert, it obviously still has been affected and has got its challenges on bond. If anything is actually caused us to accelerate of our transformation journey, you know, we had to do some great things during this time around. You know, in the UK for our emergency and health workers give them unlimited data and for vulnerable people to support them and that spent that we've had to deliver changes quickly. Um, but what? We want to be able to do it, deliver those kind of changes quickly, but sustainably for everything that we do, not just because there's an emergency eso we were already on the kind of journey to by John, but ever so ever more important now that we are what we're able to do, those that kind of work, do it more quickly on. But it works because the implications of it not working is could be terrible in terms of, you know, we've been supporting testing centers, new hospitals to treat covert patients, so we need to get it right and therefore the coverage of what we do, the quality of what we do and how quickly we do. It really has taken on a new scowling what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously deal with the fact that you know, Cove in 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less cost, but they're having to deliver more value quicker on bond, you know, to higher quality. So, yeah, certainly the finances is on our minds. And that's why we need flexible models, cost models that allow us to kind of do growth. But we get that growth by showing that we're delivering value, especially in, you know, these times when there are financial challenges on companies. >>So one of the things that I want to ask you about again looking at, develops from the inside out on the evolution that you've seen you talked about the speed of things really accelerating in this last nine months or so. When we think Dev ops, we think speed. But one of the things I love to get your perspective on we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that scene there as as needing to get, as you said, get things right but done so quickly to support essential businesses, essential workers? How have you seen that cultural shift? >>Yeah, I think you know, before, you know, test test team saw themselves of this part of the software delivery cycle. Andi, actually, now, really, our customers were expecting their quality and to deliver for our customers what they want. Quality has to be ingrained throughout the life cycle. Obviously that you know, there's lots of buzzwords like shift left. How do you do? Shift left testing. But for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle. That Dr you know, Dr Automation drive improvements. I always say that you know, you're only as good as your lowest common denominator on one thing that we're finding on our Dev Ops Journey Waas that we were you know, we would be trying thio do certain things quicker and had automated build automated tests. But if we were taking weeks to create test scripts or we were taking weeks to manly craft data, and even then when we had taken so long to do it that the coverage was quite poor and that led to lots of defects later in the lifecycle or even in in our production environment, we just couldn't afford to do that. And actually, you know, focusing on continuous testing over the last 9 to 12 months has really given us the ability Thio delivered quickly across the the whole life cycle and therefore actually go from doing a kind of semi agile kind of thing where we did you use the stories we did a few of the kind of, you know, as our ceremonies. But we weren't really deploying any quicker into production because, you know, our stakeholders were scared that we didn't have the same control that we had when we had more water for releases. And, you know, when way didn't think ourselves. So we've done a lot of work on every aspect, especially from a testing point of view, every aspect of every activity, rather than just looking at automated test, you know, whether it is actually creating the test in the first place, Whether it's doing security testing earlier in the light and performance testing. Learn the life cycle, etcetera. So, yeah, it Z It's been a riel key thing that for for C T for us to drive, develops, >>talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this, you know, there's a thing. I think people were pretty quiet. Customer experience. Gap. It reminds me of a cart, a Gilbert cartoon where, you know, we start with the requirements here on Do you know, we almost like a Chinese whisper effects and what we deliver eyes completely, completely different. So we think the testing team or the the delivery team, you know, you know, you think they've done a great job. This is what it said in the acceptance criteria, but then our customers the same Well, actually, that's not working. This isn't working, you know, on there's this kind of gap Way had a great launched this year of actual Requirement Society, one of the board common tools Onda that for the first time in in since I remember actually working within B. T, I had customers saying to may, Wow, you know, we want more of this. We want more projects, um, to have a actual requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do you actually, you know, do that have something that both the business on technical people can understand? And we've actually been working with the business using at our requirement. Designer Thio, you know, really look about what the requirements are. Tease out requirements to the hadn't even thought off and making sure that we've got high levels of test coverage. And so what we actually deliver at the end of it, not only have you been able Thio generate test more quickly, but we've got much higher test coverage and also can more smartly, you're using the kind of AI within the tour and with some of the other kind of pipeline tools actually deliver to choose the right tests on the bar, still actually doing a risk based testing approach. So that's been a great launched this year, but just the start of many kind of things that we're >>doing. But what I hear in that Glenn is a lot of positives that have come out of a very challenging situation. Uh, talk to me about it and I like that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration, perspective is you're right. We talk about that a lot critical with Dev Ops. But those challenges there you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pit it so fast? >>I mean, you talked about culture. I mean, you know, Bt is like most come countries companies. So, um, is very siloed. You know, we're still trying to work to become closer as a company. So I think there's a lot of challenges around. How do you integrate with other tools? How do you integrate with you know, the various different technologies and bt we have 58 different whitey stacks? That's not systems that stacks all of those stacks of can have, you know, hundreds of systems on we're trying to. We're gonna drive at the moment a simplified program where we're trying Thio, you know, reduce that number 2 14 stacks. And even then they'll be complexity behind the scenes that that we will be challenged. Maurin Mawr As we go forward, how do you actually hired that to our users on as an I T organization? How do we make ourselves Lena so that even when we you know, we've still got some of that legacy and we'll never fully get rid of it on that's the kind of trade off that we have to make. How do we actually deal with that and and hide that for my users a say and and and drive those programs so we can actually accelerate change. So we take, you know, reduce that kind of waste, and that kind of legacy costs out of our business. You know, the other thing is, well, beating. And I'm sure you know telecoms probably no difference to insurance or finance we've got You know, when you take the number of products that we do and then you combine them, the permutations are tens and hundreds of thousands of products. So we as a business to trying to simplify. We are trying Thio do that in a natural way and haven't trying to do agile in the proper way, you know, and really actually work it paste really deliver value. So I think what we're looking Maura, Maura, at the moment is actually, um is more value focus? Before we used to deliver changes, sometimes into production, someone had a great idea or it was a great idea nine months ago or 12 months ago. But actually, then we end up deploying it. And then we look at the the the users, you know, the usage of that product of that application or whatever it is on. It's not being used for six months, so we're getting much we haven't got, you know, because of the last 12 months, we certainly haven't got room for that kind of waste and you know, the for not really understanding the value of changes that we we are doing. So I think that's the most important thing at the moment is really taken that waste out. You know, there's lots of focus on things like flow management. What bits of the our process are actually taking too long, and we've We've started on that journey, but we've got a hell of a long way to go, you know, But that that involves looking every aspect off the kind of software delivery cycle. >>What are some? Because that that going from, what, 58 i t stocks down to 14 or whatever it's going to be go simplifying is sounds magical. Took everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we've started on a continuous testing journey, and I think that's just the start. I mean, that's really, as I say, looking at every aspect off, you know, from a Q, a point of view. It's every aspect of what we dio. But it's also looking at, you know, we're starting to branch into more like a AI ops and, you know, really, the full life cycle on. But, you know, that's just a stepping stone onto, you know, I think oughta Nomics is the way forward, right? You know all of this kind of stuff that happens um, you know, monitoring, you know, monitoring systems, what's happening in production had to be feed that back. How do you get to a point where actually we think about a change on then suddenly it's in production safely. Or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey. But if we want Thio, you know, in a world where the pace is ever increasing the demands of the team and you know, with the pressures on at the moment where with we're being asked to do things, you know more efficiently Ondas leaving as possible. We need to be, you know, thinking about every part of the process. And how do we put the kind of stepping stones in players to lead us to a more automated kind of, you know, their future? >>Do you feel that that plant outcomes are starting to align with what's delivered? Given this massive shift that you're experiencing, >>I think it's starting to, and I think you know, Azzawi. Look at more of a value based approach on. Do you know a Zeiss? A princess was a kind of flight management. I think that's that will become ever evermore important. So I think it's starting to people. Certainly realized that, you know, people teams need to work together. You know, the kind of the cousin between business and ICT, especially as we go Teoh Mawr kind of sad space solutions, low cold solutions. You know there's not such a gap anymore. Actually, some of our business partners expects to be much more tech savvy. Eso I think you know, this is what we have to kind of appreciate. What is I ts role? How do we give the capabilities become more for centers of excellence rather than actually doing Mount amount of work And for May and from a testing point of view, you know, amount, amount of testing, actually, how do we automate that? How do we actually generate that instead of created? I think that's the kind of challenge going forward. >>What are some? As we look forward, what are some of the things that you would like to see implemented or deployed in the next say, 6 to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think you know, certainly for for where we are as a company from a Q A perspective. We are. Yeah, there's certain bits that we do Well, you know, we've started creating continuous delivery. A day evokes pipelines. Um, there's still manual aspects of that. So, you know, certainly for May I I've challenged my team with saying, How do we do an automated journey? So if I, you know, I put a requirement injera or value whoever it is, that's why. Then click a button on bond, you know, with either zero touch of one touch, then put that into production and have confidence that that has been done safely on that it works. And what happens if it doesn't work? So you know, that's that's the next in the next few months, that's what our concentration is about. But it's also about decision making, you know, how do we actually understand those value judgements? And I think there's lots of the things Dev ops, ai ops, kind of always that aspects of business operations. I think it's about having the information in one place to make those kind of decisions. How does it all tied together, as I say, even still with kind of Dev ops, we've still got elements within my company where we've got lots of different organizations doing some doing similar kind of things but the walking of working in silos Still. So I think, having a eye ops Aziz becomes more and more to the fore as we go to the cloud. And that's what we need to. You know, we're still very early on in our cloud journey, you know. So we need to make sure the technologies work with Cloud as well as you kind of legacy systems. But it's about bringing that all together and having a full visible pipeline. Everybody can see and make decisions against >>you said the word confidence, which jumped out at me right away. Because absolutely, you've gotta have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops to be able to gain the confidence that they're making the right decisions for their business? >>Yeah, I mean, I think the the approach that we've taken actually is not started with technology we've actually taken human centered design a za core principle of what we dio within the i t part of BT. So by using humans tend to design. That means we talked to our customers. We understand their pain points, we map out their current processes on. But when we mapped out, those processes also understand their aspirations as well, you know, Where do they want to be in six months? You know, Do they want to be more agile and you know, or do they want Teoh? Is this apart their business that they want thio run better? We have to Then look at why that's not running well and then see what solutions are out there. We've been lucky that, you know, with our partnership with Broadcom within the P l. A. A lot of the tortures and the P l. A have directly answered some of the businesses problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which is you know, in some companies, including as they do there is that kind of, you know, almost by understanding their their pain points and then saying This is how we can solve your problem We've tended to be much more successful than trying Thio impose something and say We're here to technology that they don't quite understand doesn't really understand how it could have resonate with their problems. So I think that's the heart of it is really about, you know, getting looking at the data, looking at the processes, looking at where the kind of waste is on. Then actually then looking at the right solutions. And as I say, continuous testing is a massive for us. We've also got a good relationship with capitals looking at visual ai on. Actually, there's a common theme through that, and I mean, AI is becoming more and more prevalent, and I know yeah, sometimes what is A I and people have kind of the semantics of it. Is it true, ai or not? But yes, certainly, you know, AI and machine learning is becoming more and more prevalent in the way that we work, and it's allowing us to be much more effective, the quicker and what we do on being more accurate. You know, whether it's finding defects, running the right tests or, you know, being able to anticipate problems before they're happening in a production environment. >>Welcome. Thank you so much for giving us this sort of insight. Outlook at Dev Ops, sharing the successes that you're having taking those challenges, converting them toe opportunities and forgiving folks who might be in your shoes or maybe slightly behind advice. I'm sure they appreciate it. We appreciate your time. >>It's been an absolute pleasure, Really. Thank you for inviting me of Extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glynn Martin and Lisa Martin. You're watching the Cube?

>> Announcer: From around the globe, it's theCUBE with digital coverage of DevOps virtual forum. Brought to you by Broadcom. >> Hi guys, welcome back. So we have discussed the current state and the near future state of DevOps and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where DevOps needs to go. In order to be successful next year. So our guests today are you've seen them all before. Jeffrey Hammond is here the VP and Principal Analyst serving CIO at Forrester. We've also got Serge Lucio, the GM of Broadcom Enterprise Software Division. And Glyn Martin, the head of QA Transformation at BT. Guys welcome back. Great to have you all three together. >> Hi Lisa. (Serge speaks faintly) >> Good to be here. >> All right. So we're all very socially distanced as we talked about before. Great to have this conversation. So let's start with one of the topics that we kicked off the forum with. Jeff, we're going to start with you spiritual colocation. That's a really interesting topic that we've uncovered. But how much of the challenge is truly cultural? And what can we solve through technology? Jeff, we'll start with you, then Serge, then Glyn, Jeff take it away. >> Yeah I think fundamentally, you can have all the technology in the world. And if you don't make the right investments in the cultural practices in your development organization. You still won't be effective. Almost 10 years ago, I wrote a piece. Where I did a bunch of research around what made high performance teams software delivery teams high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the Agile Manifesto. Let's take that today. Where developers are on their own in their own offices. if you've got teams where the team itself had a high level of autonomy. And they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. And so what we have seen is that organizations that embraced autonomy, and got their teams in the right place. And their teams had the information that they needed to make the right decisions. Have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like well, how do we actually push the software that we've created into production that have become the challenge is not. Are we writing the right software? And that's why I think the term spiritual colocation is so important. Because even though we may be physically distant, we're on the same plane, we're connected from a shared purpose. There's a Surgeon I worked together a long, long time ago, so just it's been what, almost 15-16 years, since we worked at the same place. And yet I would say there's probably still a certain level of spiritual colocation, between us. because of this shared purposes that we've had in the past and what we've seen in the industry, and that's a really powerful tool to build on. So what do tools play as part of that, to the extent that tools make information available to build shared purpose on. To the extent that they enable communication so that we can build that spiritual colocation. To the extent that they reinforce the culture that we want to put in place. They can be incredibly valuable, especially when we don't have the luxury of physical colocation. Hope that makes sense.(chuckles) >> It does. I should have introduced this last segment as we're all spiritually colocated. All right. So Serge, clearly you're still spiritually colocated with Jeff. Talk to me about what your thoughts are about spiritual of colocation. The cultural impact and how technology can move it forward? >> Yes, so I think, while I'm going to sound very similar to Jeff in that respect. I think it starts with kind of shared purpose, and understanding how individuals teams contribute to kind of a business outcome. What is our shared goals our shared vision with what is it we're trying to achieve collectively. And keeping kind of the line to that. And so it really starts with it Now, the big challenge always is over the last 20 years, especially in large organization has been specialization of roles and functions. And so we all have started to basically measure which we do on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome. Or is it on purpose. We kind of revert that to Okay, what is my database uptime? What is my cycle time? Right. And I think which we can do or where we really should be focused as an industry is to start to basically provide a lens for these different stakeholders to look at what they're doing. In the context of benefiting this business outcomes. So, probably one of my theories experience was to actually witness at one of our large financial institution. Two stakeholders across development and operations staring at the same data. Like which was related to economy changes, test execution results, coverage, official liabilities, and all the overran direction of incidents. And when you start to put these things in context, and represent that in a way that these different stakeholders can look at from their different lens. And they can start to basically communicate, and understand how they jointly or complement to do that kind of common vision or objective. >> And Glyn, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part of technology impact? >> Yeah, I mean I agree with Jeffrey that, you know, the people and culture are the most important thing. Actually, that's why it's really important when you're transforming to have partners who have the same vision as you. Who you can work with have the same end goal in mind. And we would constantly found that with our continuing relationship with Broadcom. What it also does, are those tools can accelerate what you're doing and can drive consistency. You know, we've seen within simplify, which is BT's Flagship Transformation Program, where we're trying to as it says, simplify the number of system stacks that we have. The number of products that we have, actually at the moment we've got different value streams within that program. Who have got organizational silos who are trying to rewrite the wheel. Who are still doing things manually. So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT. Which is such a complex and very different environment, depending on what area BT you're in. Whether it's consumer, whether it's a mobile area, whether it's large global or government organizations. We found that we need tools that can drive that consistency. But also flex to Greenfield Brownfield kind of technologies as well. So it's really important that as it's a from a number of different aspects. That you have the right partner, and to drive the right culture here, and the same vision, but also who have the tool sets to help you accelerate, They can't do that on their own. But they can help accelerate what it is you're trying to do. And a really good example of that is we're trying to shift left, which is probably a quite a bit of a buzz phrase. And they're kind of testing well at the moment. But I could talk about things like Continuous Delivery Director to Broadcom tools. And it has many different features to it, but very simply on its own. It allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams around could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of Continuous Delivery Director. But it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >> Thanks, Glyn So we kind of sum that up autonomy, collaboration tools that facilitate that. So let's talk now about metrics. From your perspective, what are the metrics that matter Jeff? >> Well, I'm going to go right back to what Glyn said about data that provides visibility that enables us to to make decisions with shared purpose. And so business value has to be one of the first things that we looking at. How do we assess whether we have built something that is valuable? That could be sales revenue, it could be Net Promoter Score, if you're not selling what you've built, it could even be what the level of reuse is within your organization. Or other teams picking up the services that you've created. One of the things that I've begun to see organizations do is to align value streams with customer journeys. And then to align teams with those value streams. So that's one of the ways that you get to a shared purpose. 'Cause we're all trying to deliver around that customer journey. The value associated with it. And we're all measured on that. There are flow metrics, which are really important. How long does it take us to get a new feature out. From the time that we conceive it to the time that we can run our first experiments with it. There are quality metrics, some of the classics or maybe things like defect density or meantime to response. One of my favorites came from a company called Ultimate Software. Where they looked at the ratio of defects found in Production defects found in pre production. And their developers were in fact measured on that ratio and told them that guess what quality is your job too. Not just the test departments group. The fourth level that I think is really important in the current situation that we're in, is the level of engagement in your development organization. We used to joke that we measured this with the parking lot metric. How how full was the parking lot at 9, and how full was it at 5 o'clock. I can't do that anymore, since we're not physically colocated. But what you can do is you can look at how folks are delivering. You can look at your metrics in your SCCM environment, you can look at the relative rates of churn, you can look at things like well are our developers delivering during longer periods. Earlier in the morning, later in the evening? Are they delivering on the weekends as well. Are those signs that we might be heading toward burnout, because folks are still running at sprint levels instead of marathon levels. So all of those in combination, business value, flow, engagement and quality. I think form the backbone of any sort of metrics program. The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Unfortunately I see organizations where they weaponize the data. And that's completely the wrong way to look at it. What you need to do is you need to say. "How is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing? And then what do we do to remove those blockers to make sure that we're giving these autonomous teams, the context that they need to do their job in a way that creates the most value for the customers?" >> Great advice, Jeff. Glyn over to you metrics that matter to you that really make a big impact. And also how do you measure quality kind of following on to the advice that Jeff provided? >> I mean, Jeff provided some great advice. Actually, he talks about value, he talks about flow, both of those things are very much on my mind at the moment. But there was a time, listen to a speaker called Mia Kirsten, a couple of months ago, he talked very much around how important flow management is. And remove and using that to remove waste, to understand in terms of, making software changes. What is it that's causing us to do it longer than we need to? So where are those areas where it takes too long. So I think that's a very important thing. For us, it's even more basic than that at the moment. We're on a journey from moving from waterfall to agile. And the problem with moving from waterfall to agile is, with waterfall, the the business had a kind of comfort that everything was tested together, and therefore it's safer. And with agile, there's that kind of how do we make sure that you know, if we're doing things quick, and we're getting stuff out the door that we give that confidence, that that's ready to go? Or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence. And some of the metrics around that and how healthy those releases are and actually saying we spend a lot of money, in an investment setting up agile teams training agile teams. Are we actually seeing them deliver more quickly? And are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment. But I think it's also about, generally bringing it all together DevOps. We've got the kind of value ops, AI Ops. How do we actually bring that together to so we can make quick decisions, and making sure that we are delivering the biggest bang for our partners. >> Absolutely biggest bang for the partners. Serge your thoughts. >> Yes I think we all agree, right? It starts with business metrics, flow metrics. These are one of the most important metrics and ultimately, I mean, one of the things that's very common across I highly functional teams is engagements, right? When you see a team that's highly functional, and that's agile, that practices DevOps everyday. They are highly engaged. That definitely true. Now back to you, I think, Jeff's points on weaponization of metrics. One of the key challenges we see is that organizations traditionally have been kind of, setting up benchmarks. Right. So what is a good cycle time? What is a good mean time? What is a good mean time to repair? The problem is that this is very contextual, right? It's going to vary quite a bit, depending on the nature of application and system. And so one of the things that we really need to evolve as an industry. Is to understand that it's not so much about those flow metrics is about are these flow metrics ultimately contribute to the business metric. To the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood, is that when you have a bad cycle time or what you perceive as being a bad cycle time or bad quality. The problem is oftentimes like, how do you go and explore why, right? What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics. And not on the I type patterns, which are pretty common across the industry. If you look at for instance things like, lead time for instance. It's very common that organizational boundaries are going to be a key contributor to bad lead time. And so I think that there is reviewing the metrics, there is I think a lot of work that we need to do in terms of classifying this untied PaaS. Back to you, Jeff, I think you're one of the cool offers of Water-Scrum Fall as a key pattern in the industry or anti-patterns. >> Yeah >> But Water Scrum Fall, right. Is the key one right? And you will detect that through kind of a defect rival rates. That's right, that looks like an S curve. And so I think it's the output of the metrics is what do you do with those metrics. >> Right. I'll tell you Serge, one of the things that is really interesting to me in that space is. I think those of us had been in industry for a long time, we know the anti patterns, 'cause we've seen them in our career,(laughs) maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti patterns based on the telemetry that comes in. I think it would be a really interesting place to apply machine learning and reinforcement learning techniques. So hopefully something that we'd see in the future with DevOps tools. 'Cause as a manager that maybe only a 10 year veteran or a 15 year veteran. You may be seeing these anti patterns for the first time, and it would sure be nice to know what to do when they start to pop up.(chuckles) >> That would right? Insight, always helpful. All right guys, I would like to get your final thoughts on the fit one thing that you believe our audience really needs to be on the lookout for. and to put on our agendas. For the next 12 months. Jeff will be back to you. >> I would say, look for the opportunities that this disruption presents. And there are a couple that I see. First of all, as we shift to remote central working, we're unlocking new pools of talent. Where it's possible to implement more geographic diversity. So look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low code tools. To very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers. That agile and DevOps are the peanut butter and chocolate to support creative workers with algorithmic capabilities. >> Peanut butter and chocolate. Glyn where do we go from there? What's the one silver bullet that you think that needs to be on the look out for? >> (indistinct) out I certainly agree that low code is next year, we'll see much more low code. We've already started going moving towards more of a SaaS based world but low code also. I think as well for me, we've still got one foot in the kind of cloud camp. We'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last thing for me is, how do you really instill quality throughout the kind of the life cycle When I heard the word scrum for it kind of made me shut it. 'Cause I know that's a problem. That's where we're at with some of our things at the moment. So we need to get beyond that we need to be releasing changes more frequently into production. And actually being a bit more brave and having the confidence to actually do more testing in production and going straight to production itself. So expect to see much more of that next year. Yeah, thank you. I haven't got any food analogies unfortunately. (laughs) >> We all need some peanut butter and chocolate. All right Serge, Just take us on that sir. What's that nugget you think everyone needs to have on their agendas? >> That's interesting, right? So a couple of days ago, we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about velocity, right? It's all about we still are perceiving DevOps as being all about speed. And so to me the key advice is, in order to create kind of this spiritual colocation in order to foster engagement. We have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams. To understand how they're delivering value into allowing everything they do from a metrics to delivery to flow to those metrics. And only with data, I think, are we going to be able to actually start to to restart to align kind of all these roles across the organizations and drive not just speed, but business outcomes. >> All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me. Today and our guests, I think this was an incredibly valuable, fruitful conversation. And we appreciate all of you taking the time to spiritually colocate with us today. Guys, thank you. >> Thank you Lisa. >> Thank you. >> Thank you. >> For Jeff Hammond, Serge Lucio and Glyn Martin. I'm Lisa Martin. Thank you for watching the Broadcom DevOps virtual forum. (upbeat music)