>>from around the globe. It's the Cube presenting innovation for good. Brought to you by on shape. >>Okay, we're back. This is Dave Volonte and you're watching innovation for good. A program on Cuba 3 65 made possible by on shape of BTC company. We're live today really live TV, which is the heritage of the Cuban. Now we're gonna go to the sources and talkto on shape customers to find out how they're applying technology to create real world innovations that are changing the world. So let me introduce our panel members. Rafael Gomez Fribourg is with the Chan Zuckerberg bio hub. A very big idea. And collaborative nonprofit was initiative that was funded by Mark Zuckerberg and his wife, Priscilla Chan, and really around diagnosing and curing and better managing infectious diseases. So really timely topic. Philip Tabor is also joining us. He's with silver side detectors which develops neutron detective detection systems. Yet you want to know if early if neutrons and radiation or in places where you don't want them, so this should be really interesting. And last but not least, Matthew Shields is with the Charlottesville schools and is gonna educate us on how he and his team are educating students in the use of modern engineering tools and techniques. Gentlemen, welcome to the Cuban to the program. This should be really interesting. Thanks for coming on. >>Hi. Or pleasure >>for having us. >>You're very welcome. Okay, let me ask each of you because you're all doing such interesting and compelling work. Let's start with Rafael. Tell us more about the bio hub and your role there, please. >>Okay. Yes. As you said, the Bio Hope is a nonprofit research institution, um, funded by Mark Zuckerberg and his wife, Priscilla Chan. Um and our main mission is to develop new technologies to help advance medicine and help, hopefully cure and manage diseases. Um, we also have very close collaborations with Universe California, San Francisco, Stanford University and the University California Berkeley on. We tried to bring those universities together, so they collaborate more of biomedical topics. And I manage a team of engineers in by joining platform. Um, and we're tasked with creating instruments for the laboratory to help the scientist boats inside the organization and also in the partner universities do their experiments in better ways in ways that they couldn't do before >>in this edition was launched five years ago. It >>was announced at the end of 2016, and we actually started operations in the beginning of 2017, which is when I joined um, so this is our third year. >>And how's how's it going? How does it work? I mean, these things >>take time. It's been a fantastic experience. Uh, the organization works beautifully. Um, it was amazing to see it grow from the beginning. I was employee number 12, I think eso When I came in, it was just a nem p off his building and MP labs. And very quickly we had something running about from anything. Eso I'm very proud of the work that we have done to make that possible. Um And then, of course, that's you mentioned now, with co vid, um, we've been able to do a lot of very cool work, um, very being of the pandemic In March, when there was a deficit of testing, uh, capacity in California, we spun up a testing laboratory in record time in about a week. It was crazy. It was a crazy project. Um, but but incredibly satisfying. And we ended up running all the way until the beginning of November, when the lab was finally shut down, we could process about 3000 samples a day. I think at the end of it all, we were able to test about 100 on the road, 150,000 samples from all over the state. We were providing free testing toe all of the Department of Public Health Department of Public Health in California, which, at the media pandemic, had no way to do testing affordably and fast. So I think that was a great service to the state. Now the state has created a testing system that will serve those departments. So then we decided that it was unnecessary to keep going with testing in the other biopsy that would shut down, >>right? Thank you for that. Now, Now, Philip, you What you do is mind melting. You basically helped keep the world safe. Maybe you describe a little bit more about silver side detectors and what your role is there and how it all works. >>Tour. So we make a nuclear bomb detectors and we also make water detectors. So we try and do our part. Thio Keep the world from blowing up and make it a better place at the same time. Both of these applications use neutron radiation detectors. That's what we make. Put them out by a port border crossing Places like that they can help make sure that people aren't smuggling, shall we say, very bad things. Um, there's also a burgeoning field of research and application where you can use neutrons with some pretty cool physics to find water so you can do things like but a detector up in the mountains and measure snowpack. Put it out in the middle of the field and measure soil moisture content. And as you might imagine, there's some really cool applications in, uh, research and agronomy and public policy for this. >>All right, so it's OK, so it's It's much more than you know, whatever fighting terrorism, it's there's a riel edge, or I kind of i o t application for what you guys do. >>You do both Zito shares. You might >>say a mat. I I look at your role is kind of scaling the brain power for for the future. Maybe tell us more about Charlottesville schools and in the mission that you're pursuing and what you do. >>Thank you. Um, I've been in Charlottesville city schools for about 11 or 12 years. I started their teaching, Um, a handful of classes, math and science and things like that. But Thescore board and my administration had the crazy idea of starting an engineering program about seven years ago. My background is an engineering is an engineering. My masters is in mechanical and aerospace engineering. And, um, I basically spent a summer kind of coming up with what might be a fun engineering curriculum for our students. And it started with just me and 30 students about seven years ago, Um, kind of a home spun from scratch curriculum. One of my goals from the outside was to be a completely project based curriculum, and it's now grown. We probably have about six or 700 students, five or six full time teachers. We now have pre engineering going on at the 5th and 6th grade level. I now have students graduating. Uh, you know, graduating after senior year with, like, seven years of engineering under their belt and heading off to doing some pretty cool stuff. So it's It's been a lot of fun building up a program and, um, and learning a lot in the process. >>That's awesome. I mean, you know, Cuba's. We've been passionate about things like women in tech, uh, diversity stem. You know, not only do we need more more students in stem, we need mawr underrepresented women, minorities, etcetera. We were just talking to John her stock and integrate Grayson about this is do you do you feel is though you're I mean, first of all, the work that you do is awesome, but but I'll go one step further. Do you feel as though it's reaching, um, or, you know, diverse base and And how is that going? >>That's a great question. I think research shows that a lot of people get funneled into one kind of track or career path or set of interests really early on in their educational career. And sometimes that that funnels kind of artificial. And so that's one of the reasons we keep pushing back. Um, so our school systems introducing kindergartners to programming on DSO. We're trying to push back how we expose students to engineering and to stem fields as early as possible, and we've definitely seen the fruits of that in my program. In fact, my engineering program, uh, sprung out of an after school in Extracurricular Science Club that actually three girls started at our school. So I think that actually has helped that three girls started the club That eventually is what led our engineering programs that sort of baked into the DNA and also are a big public school. And we have about 50% of the students are under the poverty line, and we should I mean, Charlottesville, which is a big refugee town. And so I've been adamant from Day one that there are no barriers to entry into the program. There's no test you have to take. You don't have to have be taking a certain level of math or anything like that. That's been a lot of fun. To have a really diverse set of kids and or the program and be successful, >>that's phenomenal. That's great to hear. So, Philip, I wanna come back to you. You know, I think about maybe some day we'll be able to go back to a sporting events, and I know when I when I'm in there, there's somebody up on the roof looking out for me, you know, watching the crowd. And they have my back. And I think in many ways, the products that you build, you know, our similar I may not know they're there, but they're keeping us safe or they're measuring things that that that I don't necessarily see. But I wonder if you could talk about a little bit more detail about the products you build and how they're impacting society. >>Sure, So there are certainly a lot of people who are who are watching, trying to make sure things were going well in keeping you safe that you may or may not be aware of. And we try and support ah lot of them. So we have detectors that are that are deployed in a variety of variety of uses with a number of agencies and governments that dio like I was saying, ports and border crossing some other interesting applications that are looking for looking for signals that should not be there and working closely to fit into the operations these folks do Onda. We also have ah lot of outreach to researchers and scientists trying to help them support the work they're doing, um, using neutron detection for soil moisture monitoring is a some really cool opportunities for doing it at large scale and with much less, um, expense or complication then would have been done previous technologies. Mhm. You know, they were talking about collaboration in the previous segment. We've been able to join a number of conferences for that, virtually including one that was supposed to be held in Boston. But another one that was held, uh, of the University of Heidelberg in Germany. And, uh, this is sort of things that in some ways, the pandemic is pushing people towards greater collaboration than there would have been able to do. Had it all but in person. >>Yeah, we did. Uh, the cube did live works a couple years ago in Boston. It was awesome show. And I think, you know, with this whole trend toward digit, I call it the forced march to digital. Thanks to cove it I think that's just gonna continue. Thio grow Raphael one. If you could describe the process that you used to better understand diseases and what's your organization's involvement? Been in more detail, addressing the cove in pandemic. >>Um, so so we have the bio be structured in, Um um, in a way that foster So the combination of technology and science. So we have to scientific tracks, one about infectious diseases and the other one about understanding just basic human biology how the human body functions and especially how the cells in the human body function on how they're organized to create teachers in the body. Um, and then it has the set of platforms. Um, mind is one of them by engineering that are all technology. Read it. So we have data science platform, all about data analysis, machine learning, things like that. Um, we have a mass spectrometry platform is all about mass spectrometry technologies to, um, exploit those ones in service for the scientists on. We have a genomics platform. That is all about sequencing DNA in our DNA. Um, and then an advanced microscopy. It's all about developing technologies, uh, to look at things with advanced microscopes and the little technologies to marry computation on microscope. So, um, the scientists said the agenda and the platforms we just serve their needs, support their needs, and hopefully develop technologies that help them do their experiments better, faster, or allow them to the experiment that they couldn't do in any other way before. Um And so with cove, it because we have that very strong group of scientists that work on. I have been working on infectious disease before, and especially in viruses, we've been able to very quickly pivot to working on that s O, for example, my team was able to build pretty quickly a machine to automatically purified proteins, and it's being used to purify all these different important proteins in the cove. It virus the SARS cov to virus on Dwyer, sending some of those purified proteins all over the world. Two scientists that are researching the virus and trying to figure out how to develop vaccines, understand how the virus affects the body and all that. So some of the machines we built are having a very direct impact on this. Um, Also for the copy testing lab, we were able to very quickly develop some very simple machines that allowed the lab to function sort of faster and more efficiently. Sort of had a little bit of automation in places where we couldn't find commercial machines that would do it. >>Um, God s o mat. I mean, you gotta be listening to this in thinking about, Okay? Some. Someday your students are gonna be working at organizations like Like like Bio Hub and Silver Side. And you know, a lot of young people that just have I don't know about you guys, but like my kids, they're really passionate about changing the world. You know, there's way more important than, you know, the financial angles and that z e I gotta believe you're seeing that you're right in the front lines there. >>Really? Um, in fact, when I started the curriculum six or seven years ago, one of the first bits of feedback I got from my students is they said Okay, this is a lot of fun. So I had my students designing projects and programming microcontrollers raspberry, PiS and order We nose and things like that. The first bit of feedback I got from students was they said Okay, when do we get to impact the world? I've heard engineering is about making the world a better place, and robots are fun and all, but, you know, where is the real impact? And so, um do Yeah, thanks to the guidance of my students, I'm baking that Maurin. Now I'm like Day one of engineering one. We talk about how the things that the tools they're learning and the skills they're gaining eventually you know, very soon could be could be used to make the world a better place. >>You know, we all probably heard that famous line By Jeff Hammond Barker. The greatest minds of my generation are trying to figure out how to get people to click on ads. E. I think we're really generally generationally finally, at the point where you know young students and engineering and really you know it passionate about affecting society. I wanna get into the product, you know, side and understand how each of you are using on shape and and the value that that it brings. Maybe Raphael, you could start how long you've been using it. You know, what's your experience with it? Let's let's start there. >>I begin for about two years, and I switched to it with some trepidation. You know, I was used to always using the traditional product that you have to install on your computer, that everybody uses that. So I was kind of locked into that, but I started being very frustrated with the way it worked, um, and decided to give on ship chance. Which reputation? Because any change always, you know, causes anxiety. But very quickly my engineers started loving it. Uh, just because it's it's first of all, the learning curve wasn't very difficult at all. You can transfer from one from the traditional product to entree very quickly and easily. You can learn all the concepts very, very fast. It has all the functionality that we needed, and and what's best is that it allows to do things that we couldn't do before or we couldn't do easily. Um, now we can access the our cat documents from anywhere in the world. Um, so when we're in the lab fabricating something or testing a machine, any computer we have next to us or a tablet or on iPhone, we can pull it up and look at the cad and check things or make changes that something that couldn't do before because before you had to pay for every installation off the software for the computer, and I couldn't afford to have 20 installations to have some computers with the cat ready to use them like once every six months would have been very inefficient. So we love that part. And the collaboration features are fantastic. Especially now with Kobe, that we have to have all the remote meetings, eyes fantastic, that you can have another person drive the cad while the whole team is watching that person change the model and do things and point to things that is absolutely revolutionary. We love it. The fact that you have very, very sophisticated version control before it was always a challenge asking people, please, if you create anniversary and apart, how do we name it so that people find it? And then you end up with all these collection of files with names that nobody remembers, what they are, the person left and now nobody knows which version is the right one m s with on shape on the version ING system it has, and the fact that you can go back in history off the document and go back to previous version so easily and then go back to the press and version and explore the history of the part that is truly, um, just world changing for us, that we can do that so easily on for me as a manager to manage this collection of information that is critical for our operations. It makes it so much easier because everything is in one place. I don't have to worry about file servers that go down that I have to administer that have to have I t taken care off that have to figure how to keep access to people to those servers when they're at home. And they need a virtual private network and all of that mess disappears. I just simply give give a personal account on shape. And then, magically, they have access to everything in the way I want. And we can manage the lower documents and everything in a way, that is absolutely fantastic. >>Rafael, what was your what? What were some of the concerns you had mentioned? You had some trepidation. Was it a performance? Was it security? You know, some of the traditional cloud stuff and I'm curious as to how How whether any of those act manifested were they really that you had to manage? What were your concerns? >>Look, the main concern is how long is it going to take for everybody in the team? to learn to use the system like it and buy into it because I don't want to have my engineers using tools against their will write. I want everybody to be happy because that's how they're productive. They're happy and they enjoyed the tools they have. That was my main concern. I was a little bit worried about the whole concept of not having the files in a place where I couldn't quote unquote seat in some serving on site, but that that's kind of an outdated concept, right? So that took a little bit of a mind shift. But very quickly. Then I started thinking, Look, I have a lot of documents on Google Drive like I don't worry about that. Why would I worry about my cat on on shape? Right is the same thing. So I just needed to sort of put things in perspective that way. Um, the other, um, you know, their concern was the learning curve right is like how is he will be for everybody to and for me to learn it on whether it had all of the features that we needed and there were a few features that I actually discussed with, um uh, Cody at on shape on. They were actually awesome about using their scripting language in on shape to sort of mimic some of the features of the old cat, uh, in on shaped in a way that actually works even better than the old system. So it was It was amazing. Yeah. >>Great. Thank you for that, Phillip. What's your experience been? Maybe you could take us through your journey with on shape? >>Sure. So we've been we've been using on shaped Silver Side for coming up on about four years now, and we love it. We're very happy with it. We have a very modular product line, so and we make anything from detectors that would go into backpacks? Two vehicles, two very large things that a shipping container would go through and saw. Excuse me. Shape helps us to track and collaborate faster on the design, have multiple people working a same time on a project. And it also helps us to figure out if somebody else comes to us and say, Hey, I want something new. How we congrats modules from things that we already have. Put them together and then keep track of the design development and the different branches and ideas that we have, how they all fit together. A za design comes together and it's just been fantastic from a mechanical engineering background. I will also say that having used a number of different systems and solid works was the greatest thing since sliced bread. Before I got using on shape, I went, Wow, this is amazing. And I really don't want to design in any other platform after after getting on Lee a little bit familiar with it. >>You know, it's funny, right? I will have the speed of technology progression. I was explaining to some young guns the other day how e used to have a daytime er and that was my life. And if I lost that day, timer, I was dead. And I don't know how we weigh existed without, you know, Google Maps. Eso did we get anywhere? I don't know, but, uh, but so So, Matt, you know, it's interesting to think about, um, you know, some of the concerns that Raphael brought up, you hear? For instance, you know, all the time. Wow. You know, I get my Amazon bill at the end of the month It's through the roof in. But the reality is that Yeah, well, maybe you are doing more, but you're doing things that you couldn't have done before. And I think about your experience in teaching and educating. I mean, you so much more limited in terms of the resource is that you would have had to be able to educate people. So what's your experience been with With on shape and what is it enabled? >>Um, yeah, it was actually talking before we went with on shape. We had a previous CAD program and I was talking to my vendor about it, and he let me know that we were actually one of the biggest CAD shops in the state. Because if you think about it a really big program, you know, really big company might employ 5, 10, 15, 20 cad guys, right? I mean, when I worked for a large defense contractor, I think there were probably 20 of us as the cad guys. I now have about 300 students doing cat. So there's probably more students with more hours of cat under their belt in my building than there were when I worked for the big defense contractor. Um, but like you mentioned, uh, probably our biggest hurdle is just re sources. And so we want We want one of things I've always prided myself and trying to do in this programs provide students with access two tools and skills that they're going to see either in college or in the real world. So it's one of the reason we went with a big professional cad program. There are, you know, sort of k 12 oriented software and programs and things. But, you know, I want my kids coding and python and using slack and using professional type of tools on DSO when it comes to cat. That's just that that was a really hurt. I mean, you know, you could spend $30,000 on one seat of, you know, professional level cad program, and then you need a $30,000 computer to run it on if you're doing a heavy assemblies, Um, and so one of my dreams and it was always just a crazy dream. And I was the way I would always pitcher in my school system and say someday I'm gonna have a kid on a school issued chromebook in subsidized housing on public WiFi doing professional level bad and that that was a crazy statement until a couple of years ago. So we're really excited that I literally and, you know, march in, um, you said the forced march the forced march into, you know, modernity, March 13th kids sitting in my engineering lab that we spent a lot of money on doing. Cad March 14th. Those kids were at home on their school shoot chromebooks on public WiFi, uh, keeping their designs going and collaborating. And then, yeah, I could go on and on about some of the things you know, the features that we've learned since then they're even better. So it's not like this is some inferior, diminished version of the cat. And there's so much about it, E >>wanna I wanna ask you that I may be over my skis on this, but we're seeing we're starting to see the early days of the democratization of CAD and product design. It is the the citizen engineer. I mean, maybe insulting to the engineers in the room, but but is that we're beginning to see that >>I have to believe that everything moves into the cloud. Part of that is democratization that I don't need. I can whether you know, I think artists, you know, I could have a music studio in my basement with a nice enough software package. And Aiken, I could be a professional for now. My wife's a photographer. I'm not allowed to say that I could be a professional photographer with, you know, some cloud based software. And so, yeah, I do think that's part of what we're seeing is more and more technology is moving to the cloud >>Philip or Rafael anything. Your dad, >>I think I mean yeah, that that that combination of cloud based cat and then three D printing that is becoming more and more affordable on ubiquitous It's truly transformative, and I think for education is fantastic. I wish when I was a kid I had the opportunity to play with those kinds of things because I was always the late things. But, you know, the in a very primitive way. So, um, I think there's a dream for kids Thio to be able to do this. And, um, yeah, there's so many other technologies coming on, like Arduino and all of these electronic things that live. Kids play at home very cheaply with things that back in my day would have been unthinkable. >>So we know there's a go ahead. Philip Way >>had a pandemic and silver site moved to a new manufacturing facility this year. I was just on the shop floor, talking with contractors, standing 6 ft apart, pointing at things. But through it all, our CAD system was completely unruffled. Nothing stopped in our development work. Nothing stopped in our support for existing systems in the field. We didn't have to think about it. We had other server issues, but none with our, you know, engineering cad, platform and product development and support world right ahead, which was cool, but also a That's point. I think it's just really cool what you're doing with the kids. The most interesting secondary and college level engineering work that I did was project based. It's an important problem to the world. Go solve it and that is what we do here. That is what my entire career has been. And I'm super excited to see See what your students are gonna be doing, uh, in there home classrooms on their chromebooks now and what they do. Building on that. >>Yeah, I'm super excited to see your kids coming out of college with engineering degrees because yeah, I think that project based experience is so much better than just sitting in a classroom, taking notes and doing math problems on. And I think he will give the kids a much better flavor What engineering is really about. Think a lot of kids get turned off by engineering because they think it's kind of dry because it's just about the math for some very abstract abstract concept, and they are there. But I think the most important thing is just that. Hands on a building and the creativity off, making things that you can touch that you can see that you can see functioning. >>Great. So you know, we all know the relentless pace of technology progression. So when you think about when you're sitting down with the folks that on shape and there the customer advisor for one of the things that you want on shape to do that it doesn't do today >>I could start by saying, I just love some of the things that does do because it's such a modern platform and I think some of these, uh, some some platforms that have a lot of legacy and a lot of history behind them. I think we're dragging some of that behind them. So it's cool to see a platform that seemed to be developed in a modern era. And so that's, you know, it is the Google docks. And so the fact that collaboration and version ing and link sharing is, and, like, platform agnostic abilities the fact that that seems to be just built into the nature of the thing so far, that's super exciting as far as things that it to go from there, Um, I don't know. >>Other than price, >>you can't say I >>can't say lower price. >>Yeah, so far on a PTC s that worked with us. Really well, so I'm not complaining. There. You there? >>Yeah. Yeah. No Gaps, guys. Whitespace, Come on. >>We've been really enjoying the three week update Cadence. You know, there's a new version every three weeks and we don't have to install it. We just get all the latest and greatest goodies. One of the trends that we've been following and enjoying is the the help with a revision management and release work flows. Um, and I know that there's more than on shape is working on that we're very excited for, because that's a big important part about making real hardware and supporting it in the field. Um, something that was cool. They just integrated Cem markup capability In the last release that took, we were doing that anyway, but we were doing it outside of on shapes, and now we get to streamline our workflow and put it in the CAD system where we're making those changes anyway, when we're reviewing drawings and doing this kind of collaboration. And so I think from our perspective, we continue to look forward toa further progress on that. There's a lot of capability in the cloud that I think they're just kind of scratching the surface on you. >>I would. I mean, you're you're asking to knit. Pick. I would say one of the things that I would like to see is is faster regeneration speed. There are a few times with comics necessities that regenerating the document takes a little longer than I would like to. It's not a serious issue, but anyway, I'm being spoiled, >>you know. That's good. I've been doing this a long time and I like toe Ask that question of practitioners and to me, it it's a signal like when you're nit picking and that you're struggling to knit. Pick that to me is a sign of a successful product. And And I wonder, I don't know, uh, have the deep dive into the architecture, But are things like alternative processors? You're seeing them hit the market in a big way. Uh, you know, maybe a helping address the challenge, But I'm gonna ask you the big, chewy question now, then would maybe go to some audience questions when you think about the world's biggest problems. I mean, we're global pandemics. Obviously top of mind. You think about nutrition, you know, feeding the global community. We've actually done a pretty good job of that. But it's not necessarily with the greatest nutrition climate change, alternative energy, the economic divides. You've got geopolitical threats and social unrest. Health care is a continuing problem. What's your vision for changing the world and how product innovation for good can be applied to some of the the problems that that you all are passionate about? Big question. But who wants toe start >>not biased. But for years I've been saying that if you want to solve the economy, the environment, uh, global unrest, pandemics education is the case If you wanna if you want to, um, make progress in those in those realms, I think funding funding education is probably gonna pay off pretty well. >>Absolutely. And I think stem is key to that. I mean, all of the, ah lot of the well being that we have today and then industrialized countries, thanks to science and technology, right, improvements in health care, improvements in communication, transportation, air conditioning. Um, every aspect of life is touched by science and technology. So I think having more kids studying and understanding that is absolutely key. Yeah, I agree, >>Philip, you got anything they had? >>I think there's some big technical problems in the world today, Raphael and ourselves there certainly working on a couple of them. Think they're also collaboration problems and getting everybody doing ableto pull together instead of pulling, pulling separately and to be able to spur the idea is onwards. So that's where I think the education side is really exciting. What Matt is doing and and it just kind of collaboration in general when we could do provide tools to help people do good work? Uh, that is, I think, valuable. >>Yeah, I think that's a very good point. And along those lines, we have some projects that are about creating very low cost instruments for low research settings places in Africa, Southeast Asia, South America so that they can do, um, um, biomedical research that it's difficult to do in those place because they don't have the money to buy the fancy lab machines that cost $30,000 an hour. Um, so we're trying to sort of democratize some of those instruments. And I think thanks to tools like Kahn shaped and is easier, for example, to have a conversation with somebody in Africa and show them the design that we have and discuss the details of it with them. Andi, that's amazing. Right? To have somebody you know, 10 time zones away, Um, looking really life in real time with you about your design and discussing the details or teaching them how to build a machine. Right? Because, um, you know, they have a three d printer. You can you just give them the design and say, like, you build it yourself, uh, even cheaper than and, you know, also billing and shipping it there. Um, so all that that that aspect of it is also so super important, I think, for any of these efforts to improve, um, some of the hardest part was in the world from climate change. Do you say, as you say, poverty, nutrition issues? Um, you know, availability of water. You have that project at about finding water. Um, if we can also help deploy technologies that teach people remotely how to create their own technologies or how to build their own systems that will help them solve those forms locally. I think that's very powerful. >>Yeah, that point about education is right on. I think some people in the audience may be familiar with the work of Erik Brynjolfsson and Andrew McAfee, the second machine age where they sort of put forth the premise that, uh, is it laid it out. Look, for the first time in history, machines air replacing humans from a cognitive perspective. Machines have always replaced humans, but that's gonna have an impact on jobs. But the answer is not toe protect the past from the future. Uh, the answer is education and public policy. That really supports that. So I couldn't agree more. I think it's a really great point. Um, we have We do have some questions from the audience. If if we can. If I can ask you guys, um, you know, this one kind of stands out. How do you see artificial intelligence? I was just talking about machine intelligence. Um, how do you see that? Impacting the design space guys trying to infuse a I into your product development. What can you tell me? >>Um, absolutely. Like, we're using AI for some things, including some of these very low cost instruments that will hopefully help us diagnose certain diseases, especially this is that are very prevalent in the Third World. Um, and some of those diagnostics are these days done by thes armies of technicians that are trained to look under the microscope. But, um, that's a very slow process. Is very error prone and having machine learning systems that can, to the same diagnosis faster, cheaper and also little machines that can be taken to very remote places to these villages that have no access to a fancy microscope to look at a sample from a patient that's very powerful, and I we don't do this. But I have read quite a bit about how certain places air, using a Tribune attorneys to actually help them optimize designs for parts. So you get these very interesting looking parts that you would have never thought off. A person would have never thought off, but that are incredibly light ink earlier strong and I have all sort of properties that are interesting thanks to artificial intelligence machine learning in particular, >>yet another, uh, advantage you get when when your work is in the cloud I've seen. I mean, there's just so many applications that so if the radiology scan is in the cloud and the radiologist is goes to bed at night, radiologist could come in in the morning and and say, Oh, the machine while you were sleeping was using artificial intelligence to scan these 40,000 images. And here's the five that we picked out that we think you should take a closer look at or like Raphael said. I can design my part. My, my, my, my, my you know, mount or bracket or whatever and go to sleep. And then I wake up in the morning. The machine has improved. It for me has made it strider strider stronger and lighter. Um And so just when your when your work is in the cloud, that's just that's a really cool advantage that you get that you can have machines doing some of your design work for you. >>Yeah, we've been watching, uh, you know, this week is this month, I guess is aws re invent and it's just amazing to see how much effort is coming around machine learning machine intelligence. You know, Amazon has sage maker Google's got, you know, embedded you no ML and big query. Certainly Microsoft with Azure is doing tons of stuff and machine learning. I think the point there is that that these things will be infused in tow R and D and in tow software products by the vendor community. And you all will apply that to your business and and build value through the unique data that your collecting you know, in your ecosystems. And and that's how you add value. You don't have to be necessarily, you know, developers of artificial intelligence, but you have to be practitioners to apply that. Does that make sense to you, Philip? >>Yeah, absolutely. And I think your point about value is really well chosen. We see AI involved from the physics simulations all the way up to interpreting radiation data, and that's where the value question, I think, is really important because it's is the output of the AI giving helpful information that the people that need to be looking at it. So if it's curating a serious of radiation alert, saying, Hey, like these are the anomalies you need to look at eyes it, doing that in a way that's going to help a good response on. In some cases, the II is only as good as the people. That sort of gave it a direction and turn it loose. And you want to make sure that you don't have biases or things like that underlying your AI that air going to result in, uh in less than helpful outcomes coming from it. So we spend quite a lot of time thinking about how do we provide the right outcomes to people who are who are relying on our systems? >>That's a great point, right? Humans, air biased and humans build models, so models are inherently biased. But then software is hitting the market. That's gonna help us identify those biases and help us, you know? Of course. Correct. So we're entering Cem some very exciting times, guys. Great conversation. I can't thank you enough for spending the time with us and sharing with our audience the innovations that you're bringing to help the world. So thanks again. >>Thank you so much. >>Thank you. >>Okay. You're welcome. Okay. When we come back, John McElheny is gonna join me. He's on shape. Co founder. And he's currently the VP of strategy at PTC. He's gonna join the program. We're gonna take a look at what's next and product innovation. I'm Dave Volonte and you're watching innovation for good on the Cube, the global leader. Digital technology event coverage. We'll be right back

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of DevOps and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also got Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago, surgery it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen, uh, in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate, physical colocation. Hope. That makes sense. >>It does. I should have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with junk. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other, I, Oh, individuals teams, uh, contributed to kind of a business outcome. What is our shared goal or shared vision? What's what is it we're trying to achieve collectively and, uh, keeping it aligned to that. Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organizations, there's the specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of revert back to, okay, what is my database all the time? What is my cycle time like? >>And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens for these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, theories of experience was to actually weakness at one of a large financial institution, um, you know, to stakeholders and quote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, testing, execution results, you know, covert coverage, um, official liabilities and all the all ran. It could have a direction leveling. So that's when you start to put these things in context and represent that in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and understand of they jointly are competing to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your on spiritual co-location and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it says, simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos who were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very different environments. But in all areas, BT you're in whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in there kind of testing world at the moment. >>But, you know, I could talk about things like continuous delivery director, one of Broadcom's tools, and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum that up. Autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter, Jeff? >>Well, I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value associated with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test? Uh, department's a group. The fourth level that I think is really important, uh, in, in the current, uh, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric. How full was the parking lot at nine? And how full was it at five o'clock? I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customer. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big, and also, >>How do you measure quality kind of following onto the advice that Jeff provided? I mean, Jeff provided some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker called me Kirsten a couple of months ago. It talked very much about how important flight management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes too long? So I think that's a very important thing for us. It's, um, even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up, Pat, our teams training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, GemCis point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about, are these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing, the second aspect, I think that's oftentimes misunderstood. >>Yeah. >>So that cycle time, or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, you look at, for instance, things like, you know, lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. And so I think that there is, you know, the metrics there is, I think a lot of, uh, work that we need to do in terms of classifying this antibiograms, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as a, as a key patterning industry or anti-fat. Um, but what our scrum fall right, is a key one, right. And you will detect that through defect, arrival rates. That's where that looks like an escort. And so I think it's beyond kind of the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on the fit. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff, we'll go back to you. >>I would say, look for the opportunities that this disruption presents. And there are a couple that I see, first of all, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks should be on the lookout for? >>I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but Loco also, um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and, and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the life cycle, um, where, when I heard the word scrum for it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment. So we need to get beyond that. We need to be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production in going straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately >>We all need some peanut butter and chocolate. All right. It starts to take us on that's what's that nugget you think everyone needs to have on their agendas. >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's, it's all about the lost city, right? It's all about, we still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of that spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value and to align everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi, Lisa Martin here covering the Broadcom dev ops virtual forum. I'm very pleased to be joined today by a cube alumni, Jeffrey Hammond, the vice president and principal analyst serving CIO is at Forester. Jeffrey. Nice to talk with you today. >>Good morning. It's good to be here. Yeah. >>So a virtual forum, great opportunity to engage with our audiences so much has changed in the last it's an understatement, right? Or it's an overstated thing, but it's an obvious, so much has changed when we think of dev ops. One of the things that we think of is speed, you know, enabling organizations to be able to better serve customers or adapt to changing markets like we're in now, speaking of the need to adapt, talk to us about what you're seeing with respect to dev ops and agile in the age of COVID, what are things looking like? >>Yeah, I think that, um, for most organizations, we're in a, uh, a period of adjustment, uh, when we initially started, it was essentially a sprint, you know, you run as hard as you can for as fast as you can for as long as you can and you just kind of power through it. And, and that's actually what, um, the folks that get hub saw in may when they ran an analysis of how developers, uh, commit times and a level of work that they were committing and how they were working, uh, in the first couple of months of COVID was, was progressing. They found that developers, at least in the Pacific time zone were actually increasing their work volume, maybe because they didn't have two hour commutes or maybe because they were stuck away in their homes, but for whatever reason, they were doing more work. >>And it's almost like, you know, if you've ever run a marathon the first mile or two in the marathon, you feel great and you just want to run and you want to power through it and you want to go hard. And if you do that by the time you get to mile 18 or 19, you're going to be gassed. It's sucking for wind. Uh, and, and that's, I think where we're starting to hit. So as we start to, um, gear our development chops out for the reality that most of us won't be returning into an office until 2021 at the earliest and many organizations will, will be fundamentally changing, uh, their remote workforce, uh, policies. We have to make sure that the agile processes that we use and the dev ops processes and tools that we use to support these teams are essentially aligned to help developers run that marathon instead of just kind of power through. >>So, um, let me give you a couple of specifics for many organizations, they have been in an environment where they will, um, tolerate Rover remote work and what I would call remote work around the edges like developers can be remote, but product managers and, um, you know, essentially scrum masters and all the administrators that are running the, uh, uh, the SCM repositories and, and the dev ops pipelines are all in the office. And it's essentially centralized work. That's not, we are anymore. We're moving from remote workers at the edge to remote workers at the center of what we do. And so one of the implications of that is that, um, we have to think about all the activities that you need to do from a dev ops perspective or from an agile perspective, they have to be remote people. One of the things I found with some of the organizations I talked to early on was there were things that administrators had to do that required them to go into the office to reboot the SCM server as an example, or to make sure that the final approvals for production, uh, were made. >>And so the code could be moved into the production environment. And so it actually was a little bit difficult because they had to get specific approval from the HR organizations to actually be allowed to go into the office in some States. And so one of the, the results of that is that while we've traditionally said, you know, tools are important, but they're not as important as culture as structure as organization as process. I think we have to rethink that a little bit because to the extent that tools enable us to be more digitally organized and to hiring, you know, achieve higher levels of digitization in our processes and be able to support the idea of remote workers in the center. They're now on an equal footing with so many of the other levers, uh, that, that, um, uh, that organizations have at their disposal. Um, I'll give you another example for years. >>We've said that the key to success with agile at the team level is cross-functional co located teams that are working together physically co located. It's the easiest way to show agile success. We can't do that anymore. We can't be physically located at least for the foreseeable future. So, you know, how do you take the low hanging fruits of an agile transformation and apply it in, in, in, in the time of COVID? Well, I think what you have to do is that you have to look at what physical co-location has enabled in the past and understand that it's not so much the fact that we're together looking at each other across the table. It's the fact that we're able to get into a shared mindspace, uh, from, um, uh, from a measurement perspective, we can have shared purpose. We can engage in high bandwidth communications. It's the spiritual aspect of that physical co-location that is actually important. So one of the biggest things that organizations need to start to ask themselves is how do we achieve spiritual colocation with our agile teams? Because we don't have the, the ease of physical co-location available to us anymore? >>Well, the spiritual co-location is such an interesting kind of provocative phrase there, but something that probably was a challenge here, we are seven, eight months in for many organizations, as you say, going from, you know, physical workspaces, co-location being able to collaborate face to face to a, a light switch flip overnight. And this undefined period of time where all we were living with with was uncertainty, how does spiritual, what do you, when you talk about spiritual co-location in terms of collaboration and processes and technology help us unpack that, and how are you seeing organizations adopted? >>Yeah, it's, it's, um, it's a great question. And, and I think it goes to the very root of how organizations are trying to transform themselves to be more agile and to embrace dev ops. Um, if you go all the way back to the, to the original, uh, agile manifesto, you know, there were four principles that were espoused individuals and interactions over processes and tools. That's still important. Individuals and interactions are at the core of software development, processes and tools that support those individual and interact. Uh, those individuals in those interactions are more important than ever working software over comprehensive documentation. Working software is still more important, but when you are trying to onboard employees and they can't come into the office and they can't do the two day training session and kind of understand how things work and they can't just holler over the cube, uh, to ask a question, you may need to invest a little bit more in documentation to help that onboarding process be successful in a remote context, uh, customer collaboration over contract negotiation. >>Absolutely still important, but employee collaboration is equally as important if you want to be spiritually, spiritually co-located. And if you want to have a shared purpose and then, um, responding to change over following a plan. I think one of the things that's happened in a lot of organizations is we have focused so much of our dev ops effort around velocity getting faster. We need to run as fast as we can like that sprinter. Okay. You know, trying to just power through it as quickly as possible. But as we shift to, to the, to the marathon way of thinking, um, velocity is still important, but agility becomes even more important. So when you have to create an application in three weeks to do track and trace for your employees, agility is more important. Um, and then just flat out velocity. Um, and so changing some of the ways that we think about dev ops practices, um, is, is important to make sure that that agility is there for one thing, you have to defer decisions as far down the chain to the team level as possible. >>So those teams have to be empowered to make decisions because you can't have a program level meeting of six or seven teams and one large hall and say, here's the lay of the land. Here's what we're going to do here are our processes. And here are our guardrails. Those teams have to make decisions much more quickly that developers are actually developing code in smaller chunks of flow. They have to be able to take two hours here or 50 minutes there and do something useful. And so the tools that support us have to become tolerant of the reality of, of, of, of how we're working. So if they work in a way that it allows the team together to take as much autonomy as they can handle, um, to, uh, allow them to communicate in a way that, that, that delivers shared purpose and allows them to adapt and master new technologies, then they're in the zone in their spiritual, they'll get spiritually connected. I hope that makes sense. >>It does. I think we all could use some of that, but, you know, you talked about in the beginning and I've, I've talked to numerous companies during the pandemic on the cube about the productivity, or rather the number of hours of work has gone way up for many roles, you know, and, and, and times that they normally late at night on the weekends. So, but it's a cultural, it's a mind shift to your point about dev ops focused on velocity, sprints, sprints, sprints, and now we have to, so that cultural shift is not an easy one for developers. And even at this folks to flip so quickly, what have you seen in terms of the velocity at which businesses are able to get more of that balance between the velocity, the sprint and the agility? >>I think, I think at the core, this really comes down to management sensitivity. Um, when everybody was in the office, you could kind of see the mental health of development teams by, by watching how they work. You know, you call it management by walking around, right. We can't do that. Managers have to, um, to, to be more aware of what their teams are doing, because they're not going to see that, that developer doing a check-in at 9:00 PM on a Friday, uh, because that's what they had to do, uh, to meet the objectives. And, um, and, and they're going to have to, to, um, to find new ways to measure engagement and also potential burnout. Um, friend of mine once had, uh, had a great metric that he called the parking lot metric. It was helpful as the parking lot at nine. And how full was it at five? >>And that gives you an indication of how engaged your developers are. Um, what's the digital equivalent equivalent to the parking lot metric in the time of COVID it's commit stats, it's commit rates. It's, um, you know, the, uh, the turn rate, uh, that we have in our code. So we have this information, we may not be collecting it, but then the next question becomes, how do we use that information? Do we use that information to say, well, this team isn't delivering as at the same level of productivity as another team, do we weaponize that data or do we use that data to identify impedances in the process? Um, why isn't a team working effectively? Is it because they have higher levels of family obligations and they've got kids that, that are at home? Um, is it because they're working with, um, you know, hardware technology, and guess what, they, it's not easy to get the hardware technology into their home office because it's in the lab at the, uh, at the corporate office, uh, or they're trying to communicate, uh, you know, halfway around the world. >>And, uh, they're communicating with a, with an office lab that is also shut down and, and, and the bandwidth just doesn't enable the, the level of high bandwidth communications. So from a dev ops perspective, managers have to get much more sensitive to the, the exhaust that the dev ops tools are throwing off, but also how they're going to use that in a constructive way to, to prevent burnout. And then they also need to, if they're not already managing or monitoring or measuring the level of developer engagement, they have, they really need to start whether that's surveys around developer satisfaction, um, whether it's, you know, more regular social events, uh, where developers can kind of just get together and drink a beer and talk about what's going on in the project, uh, and monitoring who checks in and who doesn't, uh, they have to, to, um, work harder, I think, than they ever have before. >>Well, and you mentioned burnout, and that's something that I think we've all faced in this time at varying levels and it changes. And it's a real, there's a tension in the air, regardless of where you are. There's a challenge, as you mentioned, people having, you know, coworker, their kids as coworkers and fighting for bandwidth, because everyone is forced in this situation. I'd love to get your perspective on some businesses that are, that have done this well, this adaptation, what can you share in terms of some real-world examples that might inspire the audience? >>Yeah. Uh, I'll start with, uh, stack overflow. Uh, they recently published a piece in the journal of the ACM around some of the things that they had discovered. Um, you know, first of all, just a cultural philosophy. If one person is remote, everybody is remote. And you just think that way from an executive level, um, social spaces. One of the things that they talk about doing is leaving a video conference room open at a team level all day long, and the team members, you know, we'll go on mute, you know, so that they don't have to, that they don't necessarily have to be there with somebody else listening to them. But if they have a question, they can just pop off mute really quickly and ask the question. And if anybody else knows the answer, it's kind of like being in that virtual pod. Uh, if you, uh, if you will, um, even here at Forrester, one of the things that we've done is we've invested in social ceremonies. >>We've actually moved our to our team meetings on, on my analyst team from, from once every two weeks to weekly. And we have built more time in for social Ajay socialization, just so we can see, uh, how, how, how we're doing. Um, I think Microsoft has really made some good, uh, information available in how they've managed things like the onboarding process. I think I'm Amanda silver over there mentioned that a couple of weeks ago when, uh, uh, a presentation they did that, uh, uh, Microsoft onboarded over 150,000 people since the start of COVID, if you don't have good remote onboarding processes, that's going to be a disaster. Now they're not all developers, but if you think about it, um, everything from how you do the interviewing process, uh, to how you get people, their badges, to how they get their equipment. Um, security is a, is another issue that they called out typically, uh, it security, um, the security of, of developers machines ends at, at, at the corporate desktop. >>But, you know, since we're increasingly using our own machines, our own hardware, um, security organizations kind of have to extend their security policies to cover, uh, employee devices, and that's caused them to scramble a little bit. Uh, so, so the examples are out there. It's not a lot of, like, we have to do everything completely differently, but it's a lot of subtle changes that, that have to be made. Um, I'll give you another example. Um, one of the things that, that we are seeing is that, um, more and more organizations to deal with the challenges around agility, with respect to delivering software, embracing low-code tools. In fact, uh, we see about 50% of firms are using low-code tools right now. We predict it's going to be 75% by the end of next year. So figuring out how your dev ops processes support an organization that might be using Mendix or OutSystems, or, you know, the power platform building the front end of an application, like a track and trace application really, really quickly, but then hooking it up to your backend infrastructure. Does that happen completely outside the dev ops investments that you're making and the agile processes that you're making, or do you adapt your organization? Um, our hybrid teams now teams that not just have professional developers, but also have business users that are doing some development with a low-code tool. Those are the kinds of things that we have to be, um, willing to, um, to entertain in order to shift the focus a little bit more toward the agility side, I think >>Lot of obstacles, but also a lot of opportunities for businesses to really learn, pay attention here, pivot and grow, and hopefully some good opportunities for the developers and the business folks to just get better at what they're doing and learning to embrace spiritual co-location Jeffrey, thank you so much for joining us on the program today. Very insightful conversation. >>My pleasure. It's it's, it's an important thing. Just remember if you're going to run that marathon, break it into 26, 10 minute runs, take a walk break in between each and you'll find that you'll get there. >>Digestible components, wise advice. Jeffery Hammond. Thank you so much for joining for Jeffrey I'm Lisa Martin, you're watching Broadcom's dev ops virtual forum >>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom, >>Continuing our conversations here at Broadcom's dev ops virtual forum. Lisa Martin here, please. To welcome back to the program, Serge Lucio, the general manager of the enterprise software division at Broadcom. Hey, Serge. Welcome. Thank you. Good to be here. So I know you were just, uh, participating with the biz ops manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept, but I wanted to get your thoughts on spiritual co-location as really a necessity for biz ops to succeed in this unusual time in which we're living. What are your thoughts on spiritual colocation in terms of cultural change versus adoption of technologies? >>Yeah, it's a, it's, it's quite interesting, right? When we, when we think about the major impediments for, uh, for dev ops implementation, it's all about culture, right? And swore over the last 20 years, we've been talking about silos. We'd be talking about the paradox for these teams to when it went to align in many ways, it's not so much about these teams aligning, but about being in the same car in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me, the, this, this is really about kind of changing this culture where people start to look at a kind of OKR is instead of the key objective, um, that, that drives the entire team. Now, what it means in practice is really that's, uh, we need to change a lot of behaviors, right? It's not about the Yarki, it's not about roles. It's about, you know, who can do what and when, and, uh, you know, driving a bias towards action. It also means that we need, I mean, especially in this school times, it becomes very difficult, right? To drive kind of a kind of collaboration between these teams. And so I think there there's a significant role that especially tools can play in terms of providing this complex feedback from teams to, uh, to be in that preface spiritual qualification. >>Well, and it talked about culture being, it's something that, you know, we're so used to talking about dev ops with respect to velocity, all about speed here. But of course this time everything changed so quickly, but going from the physical spaces to everybody being remote really does take it. It's very different than you can't replicate it digitally, but there are collaboration tools that can kind of really be essential to help that cultural shift. Right? >>Yeah. So 2020, we, we touch to talk about collaboration in a very mundane way. Like, of course we can use zoom. We can all get into, into the same room. But the point when I think when Jeff says spiritual, co-location, it's really about, we all share the same objective. Do we, do we have a niece who, for instance, our pipeline, right? When you talk about dev ops, probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team, but just thinking about a pipeline, right, at the end of the day, it's all about what is the meantime to beat back to these teams. If I'm a developer and a commit code, I don't, does it take where, you know, that code to be processed through pipeline pushy? Can I get feedback if I am a finance person who is funding a product or a project, what is my meantime to beat back? >>And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of the Dora metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration, it's really a balance. How do you provide the feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this, this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when I checked in code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allegation, how can the system help me reduce my change pillar rates? Because it's, it's able to predict that some issue was introduced in the outpatient or work product. Um, so I think there's, there's a great role of technology and AI candidate Lynch to, to actually provide that new level of collaboration. >>So we'll get to AI in a second, but I'm curious, what are some of the, of the metrics you think that really matter right now is organizations are still in some form of transformation to this new almost 100% remote workforce. >>So I'll just say first, I'm not a big fan of metrics. Um, and the reason being that, you know, you can look at a change killer rate, right, or a lead time or cycle time. And those are, those are interesting metrics, right? The trend on metric is absolutely critical, but what's more important is you get to the root cause what is taught to you lean to that metric to degrade or improve or time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll give you a very mundane example. You know, we know that cycle time is heavily influenced by, um, organizational boundaries. So, you know, we talk a lot about silos, but, uh, we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are a pretty, you know, valid set metrics, but what's way more important is to understand what are the antiperspirants, what are the things that we can detect through the data that actually are affecting those metrics. And, uh, I mean, over the last 10, 20 years, we've learned a lot about kind of what are, what are the antiperspirants within our large enterprise customers. And there are plenty of them. >>What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >>So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle, where DevOps is more and more embracing value stream management value stream mapping. Um, and I think what's interesting is that in many ways the product is becoming the new silo. Uh, the notion of a product is very difficult by itself to actually define people are starting to recognize that a value stream is not its own little kind of Island. That in reality, when I define a product, this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So, so even on that, and there is clearly kind of a new sets, if you will, of anti-patterns where products are being defined as a set of OTRs, they have interdependencies and you have have a new set of silos on the operands, uh, the Abra key movement to Israel and the SRE space where, um, I think there is a cultural clash while the dev ops side is very much embracing this notion of OTRs and value stream mapping and Belgium management. >>On the other end, you have the it operations teams. We still think business services, right? For them, they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so the, the, there is there's this boundary where, um, I think, well, SRE is being put in place. And there's lots of thinking about what kind of metrics can be fined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for true operations team. >>And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You talked about, uh, AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate, maybe see eye to eye with the SRS? >>Yeah. Um, so th you're kind of too. So even for myself, as a leader of a, you know, 1500 people organization, there's a number of things I don't see right. On a daily basis. And, um, I think the, the, the, the technologies that we have at our disposal today from the AI are able to mind a lot of data and expose a lot of, uh, issues that's as leaders we may not be aware of. And some of the, some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you, when you start to understand, for instance, uh, what is the, what is the working progress right to during the sprint? Um, when you start to analyze the data you can detect, for instance, that maybe the teams are over committed, that there is too much work in progress. >>You can start to identify kind of, interdepencies either from a technology, from a people point of view, which were hidden, uh, you can start to understand maybe the change filler rates he's he is dragging. So I believe that there is a, there's a fundamental role to be played by the tools to, to expose again, these anti parents, to, to make these things visible to the teams, to be able to even compare teams. Right. One of the things that's, that's, uh, that's amazing is now we have access to tons of data, not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate, and what's working, what's not working >>Thoughts on AI and automation as, as a facilitator of spiritual co-location. >>Yeah, absolutely. Absolutely. It's um, you know, th there's, uh, the problem we all face is the unknown, right? The, the law city, but volume variety of the data, uh, everyday we don't really necessarily completely appreciate what is the impact of our actions, right? And so, um, AI can really act as a safety net that enables us to, to understand what is the impact of our actions. Um, and so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data, um, and collaborate on the data. And the actual matter, I think is, is a, is a very powerful enabler, uh, on, in that respect. I mean, I, I've seen, um, I've seen countless of times that, uh, for instance, at the SRE boundary, um, to basically show that we'll turn the quality attributes, so an incoming release, right. And exposing that to, uh, an operations person and a sorry person, and enabling that collaboration dialogue through data is a very, very powerful tool. >>Do you have any recommendations for how teams can use, you know, the SRE folks, the dev ops says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be nonproductive. >>Yeah. So to me, the th there, there's a part of the question really is when, when we talk about data, there are there different ways you can use data, right? Um, so you can, you can do a lot of an analytics, predictive analytics. So I think there is a, there's a tendency, uh, to look at, let's say a, um, a specific KPI, like a, an availability KPI, or change filler rate, and to basically do a regression analysis and projecting all these things, going to happen in the future. To me, that that's, that's a, that's a bad approach. The reason why I fundamentally think it's a better approach is because we are systems. The way we develop software is, is a, is a non-leader kind of system, right? Software development is not linear nature. And so I think there's a D this is probably the worst approach is to actually focus on metrics on the other end. >>Um, if you, if you start to actually understand at a more granular level, what har, uh, which are the things which are contributing to this, right? So if you start to understand, for instance, that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we, we have, I've actually, uh, a customer who, uh, identified that, uh, over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed, this resulted in these plant outages. So if you start to be able to basically establish causality, right, cause an effect between kind of data across the last cycle. I think, I think this is the right way to, uh, to, to use AI. And so pharma to be, I think it's way more God could have a classification problem. What are the classes of problems that do exist and affect things as opposed to analytics, predictive, which I don't think is as powerful. >>So I mentioned in the beginning of our conversation, that just came off the biz ops manifesto. You're one of the authors of that. I want to get your thoughts on dev ops and biz ops overlapping, complimenting each other, what, from a, the biz ops perspective, what does it mean to the future of dev ops? >>Yeah, so, so it's interesting, right? If you think about DevOps, um, there's no felony document, right? Can we, we can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways, there's no clear definition of what dev ops is. Uh, if you go to the dev ops Institute today, you'll see that they are specific, um, trainings for instance, on value management on SRE. And so in many ways, the problem we have as an industry is that, um, there are set practices between agile dev ops, SRE Valley should management. I told, right. And we all basically talk about the same things, right. We all talk about essentially, um, accelerating in the meantime fee to feedback, but yet we don't have the common framework to talk about that. The other key thing is that we add to wait, uh, for, uh, for jeans, Jean Kim's Lascaux, um, to, uh, to really start to get into the business aspect, right? >>And for value stream mapping to start to emerge for us to start as an industry, right. It, to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about driving these business outcomes. And so to me, these ops is really about kind of, uh, putting a lens on this critical element that it's not business and it, that we in fact need to fuse business 19 that I need needs to transform itself to recognize that it's, it's this value generator, right. It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this Oliver or kind of framework, if you will. That set the context for what is the reason, uh, for it to exist. What's part of the core values and principles that it needs to embrace to, again, change from a cost center to a value center. And then we need to start to use this as a way to start to unify some of the, again, the core practices, whether it's agile, DevOps value, stream mapping SRE. Um, so, so I think over time, my hope is that we start to optimize a lot of our practices, language, um, and, uh, and cultural elements. >>Last question surgeon, the last few seconds we have here talking about this, the relation between biz ops and dev ops, um, what do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >>So to me, the key, the key, um, challenge for, for the industry is really around. So we were seeing a very rapid shift towards kind of, uh, product to product, right. Which we don't want to do is to recreate kind of these new silos, these hard silos. Um, so that, that's one of the big changes, uh, that I think we need to be, uh, to be really careful about, um, because it is ultimately, it is about culture. It's not about, uh, it's not about, um, kind of how we segment the work, right. And, uh, any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of, um, kind of the spiritual co-location, I think it's, it's really about that too. It's really about kind of, uh, uh, focusing on the business outcomes on kind of aligning on driving engagement across the teams, but, but not for create a, kind of a new set of silos, which instead of being vertical are going to be these horizontal products >>Crazy by surge that looking at culture as kind of a way of really, uh, uh, addressing and helping to, uh, re re reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps virtual forum. >>Thank you. Thanks for your time. >>I'll be right back >>From around the globe it's the cube with digital coverage of devops virtual forum brought to you by Broadcom. >>Welcome to Broadcom's DevOps virtual forum, I'm Lisa Martin, and I'm joined by another Martin, very socially distanced from me all the way coming from Birmingham, England is Glynn Martin, the head of QA transformation at BT. Glynn, it's great to have you on the program. Thank you, Lisa. I'm looking forward to it. As we said before, we went live to Martins for the person one in one segment. So this is going to be an interesting segment guys, what we're going to do is Glynn's going to give us a really kind of deep inside out view of devops from an evolution perspective. So Glynn, let's start. Transformation is at the heart of what you do. It's obviously been a very transformative year. How have the events of this year affected the >> transformation that you are still responsible for driving? Yeah. Thank you, Lisa. I mean, yeah, it has been a difficult year. >>Um, and although working for BT, which is a global telecommunications company, um, I'm relatively resilient, I suppose, as a, an industry, um, through COVID obviously still has been affected and has got its challenges. And if anything, it's actually caused us to accelerate our transformation journey. Um, you know, we had to do some great things during this time around, um, you know, in the UK for our emergency and, um, health workers give them unlimited data and for vulnerable people to support them. And that's spent that we've had to deliver changes quickly. Um, but what we want to be able to do is deliver those kinds of changes quickly, but sustainably for everything that we do, not just because there's an emergency. Um, so we were already on the kind of journey to agile, but ever more important now that we are, we are able to do those, that kind of work, do it more quickly. >>Um, and that it works because the, the implications of it not working is, can be terrible in terms of you know, we've been supporting testing centers,  new hospitals to treat COVID patients. So we need to get it right. And then therefore the coverage of what we do, the quality of what we do and how quickly we do it really has taken on a new scale and what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that, you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously, um, deal with the fact that, you know, COVID 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less costs, but having to deliver more value quicker and  to higher quality. So yeah, certainly the finances is, um, on our minds and that's why we need flexible models, cost models that allow us to kind of do growth, but we get that growth by showing that we're delivering value. Um, especially in these times when there are financial challenges on companies. So one of the things that I want to ask you about, I'm again, looking at DevOps from the inside >>Out and the evolution that you've seen, you talked about the speed of things really accelerating in this last nine months or so. When we think dev ops, we think speed. But one of the things I'd love to get your perspective on is we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that you've seen there as, as needing to get, as you said, get things right, but done so quickly to support essential businesses, essential workers. How have you seen that cultural shift? >>Yeah, I think, you know, before test teams for themselves at this part of the software delivery cycle, um, and actually now really our customers are expecting that quality and to deliver for our customers what they want, quality has to be ingrained throughout the life cycle. Obviously, you know, there's lots of buzzwords like shift left. Um, how do we do shift left testing? Um, but for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle that drive automation, drive improvements. I always say that, you know, you're only as good as your lowest common denominator. And one thing that we were finding on our dev ops journey was that we  would be trying to do certain things quick, we had automated build, automated tests. But if we were taking a weeks to create test scripts, or we were taking weeks to manually craft data, and even then when we had taken so long to do it, that the coverage was quite poor and that led to lots of defects later on in the life cycle, or even in our production environment, we just couldn't afford to do that. >>And actually, focusing on continuous testing over the last nine to 12 months has really given us the ability to deliver quickly across the whole life cycle. And therefore actually go from doing a kind of semi agile kind of thing, where we did the user stories, we did a few of the kind of agile ceremonies, but we weren't really deploying any quicker into production because our stakeholders were scared that we didn't have the same control that we had when we had more waterfall releases. And, you know, when we didn't think of ourselves. So we've done a lot of work on every aspect, um, especially from a testing point of view, every aspect of every activity, rather than just looking at automated tests, you know, whether it is actually creating the test in the first place, whether it's doing security testing earlier in the lot and performance testing in the life cycle, et cetera. So, yeah,  it's been a real key thing that for CT, for us to drive DevOps, >>Talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this. Um, you know, there's a thing that I think people will probably call it a customer experience gap, and it reminds me of a Gilbert cartoon, where we start with the requirements here and you're almost like a Chinese whisper effects and what we deliver is completely different. So we think the testing team or the delivery teams, um, know in our teeth has done a great job. This is what it said in the acceptance criteria, but then our customers are saying, well, actually that's not working this isn't working and there's this kind of gap. Um, we had a great launch this year of agile requirements, it's one of the Broadcom tools. And that was the first time in, ever since I remember actually working within BT, I had customers saying to me, wow, you know, we want more of this. >>We want more projects to have extra requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do we actually, you know, do that and have something that both the business and technical people can understand. And we've actually been working with the business , using agile requirements designer to really look at what the requirements are, tease out requirements we hadn't even thought of and making sure that we've got high levels of test coverage. And what we actually deliver at the end of it, not only have we been able to generate tests more quickly, but we've got much higher test coverage and also can more smartly, using the kind of AI within the tool and then some of the other kinds of pipeline tools, actually deliver to choose the right tasks, and actually doing a risk based testing approach. So that's been a great launch this year, but just the start of many kinds of things that we're doing >>Well, what I hear in that, Glynn is a lot of positives that have come out of a very challenging situation. Talk to me about it. And I liked that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration perspective you're right, we talk about that a lot critical with devops. But those challenges there, you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pivot so fast? >>I mean, you talked about culture. You know, BT is like most companies  So it's very siloed. You know we're still trying to work to become closer as a company. So I think there's a lot of challenges around how would you integrate with other tools? How would you integrate with the various different technologies. And BT, we have 58 different IT stacks. That's not systems, that's stacks, all of those stacks can have hundreds of systems. And we're trying to, we've got a drive at the moment, a simplified program where we're trying to you know, reduce that number to 14 stacks. And even then there'll be complexity behind the scenes that we will be challenged more and more as we go forward. How do we actually highlight that to our users? And as an it organization, how do we make ourselves leaner, so that even when we've still got some of that legacy, and we'll never fully get rid of it and that's the kind of trade off that we have to make, how do we actually deal with that and hide that from our users and drive those programs, so we can, as I say, accelerate change,  reduce that kind of waste and that kind of legacy costs out of our business. You know, the other thing as well, I'm sure telecoms is probably no different to insurance or finance. When you take the number of products that we do, and then you combine them, the permutations are tens and hundreds of thousands of products. So we, as a business are trying to simplify, we are trying to do that in an agile way. >>And haven't tried to do agile in the proper way and really actually work at pace, really deliver value. So I think what we're looking more and more at the moment is actually  more value focused. Before we used to deliver changes sometimes into production. Someone had a great idea, or it was a great idea nine months ago or 12 months ago, but actually then we ended up deploying it and then we'd look at the users, the usage of that product or that application or whatever it is, and it's not being used for six months. So we haven't got, you know, the cost of the last 12 months. We certainly haven't gotten room for that kind of waste and, you know, for not really understanding the value of changes that we are doing. So I think that's the most important thing of the moment, it's really taking that waste out. You know, there's lots of focus on things like flow management, what bits of our process are actually taking too long. And we've started on that journey, but we've got a hell of a long way to go. But that involves looking at every aspect of the software delivery cycle. >> Going from, what 58 IT stacks down to 14 or whatever it's going to be, simplifying sounds magical to everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we were started on a continuous testing journey, and I think that's just the start. I mean as I say, looking at every aspect of, you know, from a QA point of view is every aspect of what we do. And it's also looking at, you know, we've started to branch into more like AI, uh, AI ops and, you know, really the full life cycle. Um, and you know, that's just a stepping stone to, you know, I think autonomics is the way forward, right. You know, all of this kind of stuff that happens, um, you know, monitoring, uh, you know, watching the systems what's happening in production, how do we feed that back? How'd you get to a point where actually we think about change and then suddenly it's in production safely, or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey, but if we want to, you know, in a world where the pace is in ever-increasing and the demands for the team, and, you know, with the pressures on, at the moment where we're being asked to do things, uh, you know, more efficiently and as lean as possible, we need to be thinking about every part of the process and how we put the kind of stepping stones in place to lead us to a more automated kind of, um, you know, um, the future. >>Do you feel that that planned outcomes are starting to align with what's delivered, given this massive shift that you're experiencing? >>I think it's starting to, and I think, you know, as I say, as we look at more of a value based approach, um, and, um, you know, as I say, print, this was a kind of flow management. I think that that will become ever, uh, ever more important. So, um, I think it starting to people certainly realize that, you know, teams need to work together, you know, the kind of the cousin between business and it, especially as we go to more kind of SAS based solutions, low code solutions, you know, there's not such a gap anymore, actually, some of our business partners that expense to be much more tech savvy. Um, so I think, you know, this is what we have to kind of appreciate what is its role, how do we give the capabilities, um, become more of a centers of excellence rather than actually doing mounds amounts of work. And for me, and from a testing point of view, you know, mounds and mounds of testing, actually, how do we automate that? How do we actually generate that instead of, um, create it? I think that's the kind of challenge going forward. >>What are some, as we look forward, what are some of the things that you would like to see implemented or deployed in the next, say six to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think, um, you know, certainly for, for where we are as a company from a QA perspective, we are, um, you let's start in bits that we do well, you know, we've started creating, um, continuous delivery and DevOps pipelines. Um, there's still manual aspects of that. So, you know, certainly for me, I I've challenged my team with saying how do we do an automated journey? So if I put a requirement in JIRA or rally or wherever it is and why then click a button and, you know, with either zero touch for one such, then put that into production and have confidence that, that has been done safely and that it works and what happens if it doesn't work. So, you know, that's, that's the next, um, the next few months, that's what our concentration, um, is, is about. But it's also about decision-making, you know, how do you actually understand those value judgments? >>And I think there's lots of the things dev ops, AI ops, kind of that always ask aspects of business operations. I think it's about having the information in one place to make those kinds of decisions. How does it all try and tie it together? As I say, even still with kind of dev ops, we've still got elements within my company where we've got lots of different organizations doing some, doing similar kinds of things, but they're all kind of working in silos. So I think having AI ops as it comes more and more to the fore as we go to cloud, and that's what we need to, you know, we're still very early on in our cloud journey, you know, so we need to make sure the technologies work with cloud as well as you can have, um, legacy systems, but it's about bringing that all together and having a full, visible pipeline, um, that everybody can see and make decisions. >>You said the word confidence, which jumped out at me right away, because absolutely you've got to have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question then for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops, to be able to gain the confidence that they're making the right decisions for their business? >>I think the, the, the, the, the approach that we've taken actually is not started with technology. Um, we've actually taken a human centered design, uh, as a core principle of what we do, um, within the it part of BT. So by using human centered design, that means we talk to our customers, we understand their pain points, we map out their current processes. Um, and then when we mapped out what this process does, it also understand their aspirations as well, you know? Um, and where do they want to be in six months? You know, do they want it to be, um, more agile and, you know, or do they want to, you know, is, is this a part of their business that they want to do one better? We actually then looked at why that's not running well, and then see what, what solutions are out there. >>We've been lucky that, you know, with our partnership, with Broadcom within the payer line, lots of the tools and the PLA have directly answered some of the business's problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which in, in, uh, you know, in some companies include not as they do there is that kind of, you know, almost by understanding their, their pain points and then starting, this is how we can solve your problem. Um, is we've, we've tended to be much more successful than trying to impose something and say, well, here's the technology that they don't quite understand. It doesn't really understand how it kind of resonates with their problems. So I think that's the heart of it. It's really about, you know, getting, looking at the data, looking at the processes, looking at where the kind of waste is. >>And then actually then looking at the right solutions. Then, as I say, continuous testing is massive for us. We've also got a good relationship with Apple towards looking at visual AI. And actually there's a common theme through that. And I mean, AI is becoming more and more prevalent. And I know, you know, sometimes what is AI and people have kind of this semantics of, is it true AI or not, but it's certainly, you know, AI machine learning is becoming more and more prevalent in the way that we work. And it's allowing us to be much more effective, be quicker in what we do and be more accurate. And, you know, whether it's finding defects running the right tests or, um, you know, being able to anticipate problems before they're happening in a production environment. >>Well, thank you so much for giving us this sort of insight outlook at dev ops sharing the successes that you're having, taking those challenges, converting them to opportunities and forgiving folks who might be in your shoes, or maybe slightly behind advice enter. They appreciate it. We appreciate your time. >>Well, it's been an absolute pleasure, really. Thank you for inviting me. I have a extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glenn Martin. I'm Lisa Martin. You're watching the cube >>Driving revenue today means getting better, more valuable software features into the hands of your customers. If you don't do it quickly, your competitors as well, but going faster without quality creates risks that can damage your brand destroy customer loyalty and cost millions to fix dev ops from Broadcom is a complete solution for balancing speed and risk, allowing you to accelerate the flow of value while minimizing the risk and severity of critical issues with Broadcom quality becomes integrated across the entire DevOps pipeline from planning to production, actionable insights, including our unique readiness score, provide a three 60 degree view of software quality giving you visibility into potential issues before they become disasters. Dev ops leaders can manage these risks with tools like Canary deployments tested on a small subset of users, or immediately roll back to limit the impact of defects for subsequent cycles. Dev ops from Broadcom makes innovation improvement easier with integrated planning and continuous testing tools that accelerate the flow of value product requirements are used to automatically generate tests to ensure complete quality coverage and tests are easily updated. >>As requirements change developers can perform unit testing without ever leaving their preferred environment, improving efficiency and productivity for the ultimate in shift left testing the platform also integrates virtual services and test data on demand. Eliminating two common roadblocks to fast and complete continuous testing. When software is ready for the CIC CD pipeline, only DevOps from Broadcom uses AI to prioritize the most critical and relevant tests dramatically improving feedback speed with no decrease in quality. This release is ready to go wherever you are in your DevOps journey. Broadcom helps maximize innovation velocity while managing risk. So you can deploy ideas into production faster and release with more confidence from around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of dev ops and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high-performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago. So it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate physical co-location. Okay. That makes sense. >>It does. I shouldn't have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with jump. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other understanding, Oh, individuals teams, uh, contributed to kind of a business outcome, what is our shared goal or shared vision? What's what is it we're trying to achieve collectively and keeping it kind of aligned to that? Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organization, there's been specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of reverted back to, okay, what is my database all the time? What is my cycle time? >>Right. And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens or these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, favorites experience was to actually weakness at one of a large financial institution. Um, you know, Tuesday Golder's unquote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, test execution results, you know, Coverity coverage, um, official liabilities and all the all ran. It could have a direction level links. And that's when you start to put these things in context and represent that to you in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and, and understand have they joined our company to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And w I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it can, it says simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos. We were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very dev, uh, different environments, depending on what area of BT you're in, whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can, um, drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. >>And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in their kind of testing world at the moment. But, you know, I could talk about things like continuous delivery direct to when a ball comes tools and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum it up, autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter? Jeff, >>I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we look at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test, uh, departments, a group, the fourth level that I think is really important, uh, in, in the current, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric helpful was the parking lot at nine. And how full was it at five o'clock. I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customers. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big impact. And, and, and also how do you measure quality kind of following onto the advice that Jeff provided? >>That's some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker, uh, called me Kirsten a couple of months ago. It taught very much around how important flow management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes long? So I think that's a very important thing for us. It's even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, you know, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how, how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up our teams, training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are, um, delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, Jeff's point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. >>And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about our, these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood is that, you know, when you have a bad cycle time or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, if you look at, for instance, things like lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. >>And so I think that there is, you know, the only the metrics there is, I think a lot of work that we need to do in terms of classifying, descend type patterns, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as, as a key pattern, the industry or anti-spatter. Um, but waterscrumfall right is a key one, right? And you will detect that through kind of a defect arrival rates. That's where that looks like an S-curve. And so I think it's beyond kind of the, the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on this. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff will go back to you. Okay. >>I would say look for the opportunities that this disruption presents. And there are a couple that I see, first of all, uh, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks to be on the lookout for now? I, I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but low code also. Um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the, the life cycle, um, where, when I heard the word scrum fall, it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment we need to get beyond that. We need >>To be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production and go straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately we all need some peanut butter and chocolate. All right. It starts to take us home. That's what's that nugget you think everyone needs to have on their agendas? >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about the lost city, but it's all about sweet. We still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of a spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value into aligning everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>> Announcer: From around the globe, it's theCUBE with digital coverage of DevOps virtual forum. Brought to you by Broadcom. >> Hi guys, welcome back. So we have discussed the current state and the near future state of DevOps and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where DevOps needs to go. In order to be successful next year. So our guests today are you've seen them all before. Jeffrey Hammond is here the VP and Principal Analyst serving CIO at Forrester. We've also got Serge Lucio, the GM of Broadcom Enterprise Software Division. And Glyn Martin, the head of QA Transformation at BT. Guys welcome back. Great to have you all three together. >> Hi Lisa. (Serge speaks faintly) >> Good to be here. >> All right. So we're all very socially distanced as we talked about before. Great to have this conversation. So let's start with one of the topics that we kicked off the forum with. Jeff, we're going to start with you spiritual colocation. That's a really interesting topic that we've uncovered. But how much of the challenge is truly cultural? And what can we solve through technology? Jeff, we'll start with you, then Serge, then Glyn, Jeff take it away. >> Yeah I think fundamentally, you can have all the technology in the world. And if you don't make the right investments in the cultural practices in your development organization. You still won't be effective. Almost 10 years ago, I wrote a piece. Where I did a bunch of research around what made high performance teams software delivery teams high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the Agile Manifesto. Let's take that today. Where developers are on their own in their own offices. if you've got teams where the team itself had a high level of autonomy. And they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. And so what we have seen is that organizations that embraced autonomy, and got their teams in the right place. And their teams had the information that they needed to make the right decisions. Have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like well, how do we actually push the software that we've created into production that have become the challenge is not. Are we writing the right software? And that's why I think the term spiritual colocation is so important. Because even though we may be physically distant, we're on the same plane, we're connected from a shared purpose. There's a Surgeon I worked together a long, long time ago, so just it's been what, almost 15-16 years, since we worked at the same place. And yet I would say there's probably still a certain level of spiritual colocation, between us. because of this shared purposes that we've had in the past and what we've seen in the industry, and that's a really powerful tool to build on. So what do tools play as part of that, to the extent that tools make information available to build shared purpose on. To the extent that they enable communication so that we can build that spiritual colocation. To the extent that they reinforce the culture that we want to put in place. They can be incredibly valuable, especially when we don't have the luxury of physical colocation. Hope that makes sense.(chuckles) >> It does. I should have introduced this last segment as we're all spiritually colocated. All right. So Serge, clearly you're still spiritually colocated with Jeff. Talk to me about what your thoughts are about spiritual of colocation. The cultural impact and how technology can move it forward? >> Yes, so I think, while I'm going to sound very similar to Jeff in that respect. I think it starts with kind of shared purpose, and understanding how individuals teams contribute to kind of a business outcome. What is our shared goals our shared vision with what is it we're trying to achieve collectively. And keeping kind of the line to that. And so it really starts with it Now, the big challenge always is over the last 20 years, especially in large organization has been specialization of roles and functions. And so we all have started to basically measure which we do on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome. Or is it on purpose. We kind of revert that to Okay, what is my database uptime? What is my cycle time? Right. And I think which we can do or where we really should be focused as an industry is to start to basically provide a lens for these different stakeholders to look at what they're doing. In the context of benefiting this business outcomes. So, probably one of my theories experience was to actually witness at one of our large financial institution. Two stakeholders across development and operations staring at the same data. Like which was related to economy changes, test execution results, coverage, official liabilities, and all the overran direction of incidents. And when you start to put these things in context, and represent that in a way that these different stakeholders can look at from their different lens. And they can start to basically communicate, and understand how they jointly or complement to do that kind of common vision or objective. >> And Glyn, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part of technology impact? >> Yeah, I mean I agree with Jeffrey that, you know, the people and culture are the most important thing. Actually, that's why it's really important when you're transforming to have partners who have the same vision as you. Who you can work with have the same end goal in mind. And we would constantly found that with our continuing relationship with Broadcom. What it also does, are those tools can accelerate what you're doing and can drive consistency. You know, we've seen within simplify, which is BT's Flagship Transformation Program, where we're trying to as it says, simplify the number of system stacks that we have. The number of products that we have, actually at the moment we've got different value streams within that program. Who have got organizational silos who are trying to rewrite the wheel. Who are still doing things manually. So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT. Which is such a complex and very different environment, depending on what area BT you're in. Whether it's consumer, whether it's a mobile area, whether it's large global or government organizations. We found that we need tools that can drive that consistency. But also flex to Greenfield Brownfield kind of technologies as well. So it's really important that as it's a from a number of different aspects. That you have the right partner, and to drive the right culture here, and the same vision, but also who have the tool sets to help you accelerate, They can't do that on their own. But they can help accelerate what it is you're trying to do. And a really good example of that is we're trying to shift left, which is probably a quite a bit of a buzz phrase. And they're kind of testing well at the moment. But I could talk about things like Continuous Delivery Director to Broadcom tools. And it has many different features to it, but very simply on its own. It allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams around could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of Continuous Delivery Director. But it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >> Thanks, Glyn So we kind of sum that up autonomy, collaboration tools that facilitate that. So let's talk now about metrics. From your perspective, what are the metrics that matter Jeff? >> Well, I'm going to go right back to what Glyn said about data that provides visibility that enables us to to make decisions with shared purpose. And so business value has to be one of the first things that we looking at. How do we assess whether we have built something that is valuable? That could be sales revenue, it could be Net Promoter Score, if you're not selling what you've built, it could even be what the level of reuse is within your organization. Or other teams picking up the services that you've created. One of the things that I've begun to see organizations do is to align value streams with customer journeys. And then to align teams with those value streams. So that's one of the ways that you get to a shared purpose. 'Cause we're all trying to deliver around that customer journey. The value associated with it. And we're all measured on that. There are flow metrics, which are really important. How long does it take us to get a new feature out. From the time that we conceive it to the time that we can run our first experiments with it. There are quality metrics, some of the classics or maybe things like defect density or meantime to response. One of my favorites came from a company called Ultimate Software. Where they looked at the ratio of defects found in Production defects found in pre production. And their developers were in fact measured on that ratio and told them that guess what quality is your job too. Not just the test departments group. The fourth level that I think is really important in the current situation that we're in, is the level of engagement in your development organization. We used to joke that we measured this with the parking lot metric. How how full was the parking lot at 9, and how full was it at 5 o'clock. I can't do that anymore, since we're not physically colocated. But what you can do is you can look at how folks are delivering. You can look at your metrics in your SCCM environment, you can look at the relative rates of churn, you can look at things like well are our developers delivering during longer periods. Earlier in the morning, later in the evening? Are they delivering on the weekends as well. Are those signs that we might be heading toward burnout, because folks are still running at sprint levels instead of marathon levels. So all of those in combination, business value, flow, engagement and quality. I think form the backbone of any sort of metrics program. The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Unfortunately I see organizations where they weaponize the data. And that's completely the wrong way to look at it. What you need to do is you need to say. "How is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing? And then what do we do to remove those blockers to make sure that we're giving these autonomous teams, the context that they need to do their job in a way that creates the most value for the customers?" >> Great advice, Jeff. Glyn over to you metrics that matter to you that really make a big impact. And also how do you measure quality kind of following on to the advice that Jeff provided? >> I mean, Jeff provided some great advice. Actually, he talks about value, he talks about flow, both of those things are very much on my mind at the moment. But there was a time, listen to a speaker called Mia Kirsten, a couple of months ago, he talked very much around how important flow management is. And remove and using that to remove waste, to understand in terms of, making software changes. What is it that's causing us to do it longer than we need to? So where are those areas where it takes too long. So I think that's a very important thing. For us, it's even more basic than that at the moment. We're on a journey from moving from waterfall to agile. And the problem with moving from waterfall to agile is, with waterfall, the the business had a kind of comfort that everything was tested together, and therefore it's safer. And with agile, there's that kind of how do we make sure that you know, if we're doing things quick, and we're getting stuff out the door that we give that confidence, that that's ready to go? Or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence. And some of the metrics around that and how healthy those releases are and actually saying we spend a lot of money, in an investment setting up agile teams training agile teams. Are we actually seeing them deliver more quickly? And are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment. But I think it's also about, generally bringing it all together DevOps. We've got the kind of value ops, AI Ops. How do we actually bring that together to so we can make quick decisions, and making sure that we are delivering the biggest bang for our partners. >> Absolutely biggest bang for the partners. Serge your thoughts. >> Yes I think we all agree, right? It starts with business metrics, flow metrics. These are one of the most important metrics and ultimately, I mean, one of the things that's very common across I highly functional teams is engagements, right? When you see a team that's highly functional, and that's agile, that practices DevOps everyday. They are highly engaged. That definitely true. Now back to you, I think, Jeff's points on weaponization of metrics. One of the key challenges we see is that organizations traditionally have been kind of, setting up benchmarks. Right. So what is a good cycle time? What is a good mean time? What is a good mean time to repair? The problem is that this is very contextual, right? It's going to vary quite a bit, depending on the nature of application and system. And so one of the things that we really need to evolve as an industry. Is to understand that it's not so much about those flow metrics is about are these flow metrics ultimately contribute to the business metric. To the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood, is that when you have a bad cycle time or what you perceive as being a bad cycle time or bad quality. The problem is oftentimes like, how do you go and explore why, right? What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics. And not on the I type patterns, which are pretty common across the industry. If you look at for instance things like, lead time for instance. It's very common that organizational boundaries are going to be a key contributor to bad lead time. And so I think that there is reviewing the metrics, there is I think a lot of work that we need to do in terms of classifying this untied PaaS. Back to you, Jeff, I think you're one of the cool offers of Water-Scrum Fall as a key pattern in the industry or anti-patterns. >> Yeah >> But Water Scrum Fall, right. Is the key one right? And you will detect that through kind of a defect rival rates. That's right, that looks like an S curve. And so I think it's the output of the metrics is what do you do with those metrics. >> Right. I'll tell you Serge, one of the things that is really interesting to me in that space is. I think those of us had been in industry for a long time, we know the anti patterns, 'cause we've seen them in our career,(laughs) maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti patterns based on the telemetry that comes in. I think it would be a really interesting place to apply machine learning and reinforcement learning techniques. So hopefully something that we'd see in the future with DevOps tools. 'Cause as a manager that maybe only a 10 year veteran or a 15 year veteran. You may be seeing these anti patterns for the first time, and it would sure be nice to know what to do when they start to pop up.(chuckles) >> That would right? Insight, always helpful. All right guys, I would like to get your final thoughts on the fit one thing that you believe our audience really needs to be on the lookout for. and to put on our agendas. For the next 12 months. Jeff will be back to you. >> I would say, look for the opportunities that this disruption presents. And there are a couple that I see. First of all, as we shift to remote central working, we're unlocking new pools of talent. Where it's possible to implement more geographic diversity. So look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low code tools. To very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers. That agile and DevOps are the peanut butter and chocolate to support creative workers with algorithmic capabilities. >> Peanut butter and chocolate. Glyn where do we go from there? What's the one silver bullet that you think that needs to be on the look out for? >> (indistinct) out I certainly agree that low code is next year, we'll see much more low code. We've already started going moving towards more of a SaaS based world but low code also. I think as well for me, we've still got one foot in the kind of cloud camp. We'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last thing for me is, how do you really instill quality throughout the kind of the life cycle When I heard the word scrum for it kind of made me shut it. 'Cause I know that's a problem. That's where we're at with some of our things at the moment. So we need to get beyond that we need to be releasing changes more frequently into production. And actually being a bit more brave and having the confidence to actually do more testing in production and going straight to production itself. So expect to see much more of that next year. Yeah, thank you. I haven't got any food analogies unfortunately. (laughs) >> We all need some peanut butter and chocolate. All right Serge, Just take us on that sir. What's that nugget you think everyone needs to have on their agendas? >> That's interesting, right? So a couple of days ago, we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about velocity, right? It's all about we still are perceiving DevOps as being all about speed. And so to me the key advice is, in order to create kind of this spiritual colocation in order to foster engagement. We have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams. To understand how they're delivering value into allowing everything they do from a metrics to delivery to flow to those metrics. And only with data, I think, are we going to be able to actually start to to restart to align kind of all these roles across the organizations and drive not just speed, but business outcomes. >> All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me. Today and our guests, I think this was an incredibly valuable, fruitful conversation. And we appreciate all of you taking the time to spiritually colocate with us today. Guys, thank you. >> Thank you Lisa. >> Thank you. >> Thank you. >> For Jeff Hammond, Serge Lucio and Glyn Martin. I'm Lisa Martin. Thank you for watching the Broadcom DevOps virtual forum. (upbeat music)