>>Hello and welcome back to the Cube's coverage of AWS Reinvent 2022. I'm John Furrier, host of the Cube. We got a great conversation with Patrick Kauflin, vice president of Go to Market Strategy and specialization at Splunk. We're talking about the open cybersecurity scheme of framework, also known as the O C sf, a joint strategic collaboration between Splunk and aws. It's got a lot of traction momentum. Patrick, thanks for coming on the cube for reinvent coverage. >>John, great to be here. I'm excited for this. >>You know, I love this open source movement and open source and continues to add value, almost sets the standards. You know, we were talking at the CNCF Linux Foundation this past fall about how standards are coming outta open source. Not so much the the classic standards groups, but you start to see the developers voting with their code groups deciding what to adopt de facto standards and security is a real key part of that where data becomes key for resilience. And this has been the top conversation at reinvent and all around the industry, is how to make data a key part of building into cyber resilience. So I wanna get your thoughts about the problem that you see that's emerging that you guys are solving with this group kind of collaboration around the ocs f >>Yeah, well look, John, I I think, I think you, you've already, you've already hit the high notes there. Data is proliferating across the enterprise. The attack surface area is rapidly expanding. The threat landscape is ever changing. You know, we, we just had a, a lot of scares around open SSL before that we had vulnerabilities and, and Confluence and Atlassian, and you go back to log four J and SolarWinds before that and, and challenges with the supply chain. In this year in particular, we've had a, a huge acceleration in, in concerns and threat vectors around operational technology. In our customer base alone, we saw a huge uptake, you know, and double digit percentage of customers that we're concerned about the traditional vectors like, like ransomware, like business email compromise, phishing, but also from insider threat and others. So you've got this, this highly complex environment where data continues to proliferate and flow through new applications, new infrastructure, new services, driving different types of outcomes in the digitally transformed enterprise of today. >>And, and what happens there is, is our customers, particularly in security, are, are left with having to stitch all of this together. And they're trying to get visibility across multiple different services, infrastructure applications across a number of different point solutions that they've bought to help them protect, defend, detect, and respond better. And it's a massive challenge. And you know, when our, when our customers come to us, they are often looking for ways to drive more consolidation across a variety of different solutions. They're looking to drive better outcomes in terms of speed to detection. How do I detect faster? How do I bind the thing that when bang in the night faster? How do I then fix it quickly? And then how do I layer in some automation so hopefully I don't have to do it again? Now, the challenge there that really OCF Ocsf helps to, to solve is to do that effectively, to detect and to respond at the speed at which attackers are demanding. >>Today we have to have normalization of data across this entire landscape of tools, infrastructure, services. We have to have integration to have visibility, and these tools have to work together. But the biggest barrier to that is often data is stored in different structures and in different formats across different solution providers, across different tools that are, that are, that our customers are using. And that that lack of data, normalization, chokes the integration problem. And so, you know, several years ago, a number of very smart people, and this was, this was a initiative s started by Splunk and AWS came together and said, look, we as an industry have to solve this for our customers. We have to start to shoulder this burden for our customers. We can't, we can't make our customers have to be systems integrators. That's not their job. Our job is to help make this easier for them. And so OCS was born and over the last couple of years we've built out this, this collaboration to not just be AWS and Splunk, but over 50 different organizations, cloud service providers, solution providers in the cybersecurity space have come together and said, let's decide on a single unified schema for how we're gonna represent event data in this industry. And I'm very proud to be here today to say that we've launched it and, and I can't wait to see where we go next. >>Yeah, I mean, this is really compelling. I mean, it's so much packed in that, in that statement, I mean, data normalization, you mentioned chokes, this the, the solution and integration as you call it. But really also it's like data's not just stored in silos. It may not even be available, right? So if you don't have availability of data, that's an important point. Number two, you mentioned supply chain, there's physical supply chain that's coming up big time at reinvent this time as well as in open source, the software supply chain. So you now have the perimeter's been dead for multiple years. We've been talking with that for years, everybody knows that. But now combined with the supply chain problem, both physical and software, there's so much more to go on. And so, you know, the leaders in the industry, they're not sitting on their hands. They know this, but they're just overloaded. So, so how do leaders deal with this right now before we get into the ocs f I wanna just get your thoughts on what's the psychology of the, of the business leader who's facing this landscape? >>Yeah, well, I mean unfortunately too many leaders feel like they have to face these trade offs between, you know, how and where they are really focusing cyber resilience investments in the business. And, and often there is a siloed approach across security, IT developer operations or engineering rather than the ability to kind of drive visibility integration and, and connection of outcomes across those different functions. I mean, the truth is the telemetry that, that you get from an application for application performance monitoring or infrastructure monitoring is often incredibly valuable when there's a security incident and vice versa. Some of the security data that, that you may see in a security operation center can be incredibly valuable in trying to investigate a, a performance degradation in an application and understanding where that may come from. And so what we're seeing is this data layer is collapsing faster than the org charts are or the budget line items are in the enterprise. And so at Splunk here, you know, we believe security resilience is, is fundamentally a data problem. And one of the things that we do often is, is actually help connect the dots for our customers and bring our customers together across the silos they may have internally so that they can start to see a holistic picture of what resilience means for their enterprise and how they can drive faster detection outcomes and more automation coverage. >>You know, we recently had an event called Super Cloud, we're going into the next gen kind of a cloud, how data and security are all kind of part of this NextGen application. It's not just us. And we had a panel that was titled The Innovators Dilemma, kind of talk about you some of the challenges. And one of the panelists said, it's not the innovator's dilemma, it's the integrator's dilemma. And you mentioned that earlier, and I think this a key point right now into integration is so critical, not having the data and putting pieces together now open source is becoming a composability market. And I think having things snap together and work well, it's a platform system conversation, not a tool conversation. So I really wanna get into where the OCS f kind of intersects with this area people are working on. It's not just solution architects or cloud cloud native SREs, especially where DevSecOps is. So this that's right, this intersection is critical. How does Ocsf integrate into that integration of the data making that available to make machine learning and automation smarter and more relevant? >>Right, right. Well look, I mean, I I think that's a fantastic question because, you know, we talk about, we use Bud buzzwords like machine learning and, and AI all the time. And you know, I know they're all over the place here at Reinvent and, and the, there's so much promise and hope out there around these technologies and these innovations. However, machine learning AI is only as effective as the data is clean and normalized. And, and we will not realize the promise of these technologies for outcomes in resilience unless we have better ways to normalize data upstream and better ways to integrate that data to the downstream tools where detection and response is happening. And so Ocsf was really about the industry coming together and saying, this is no longer the job of our customers. We are going to create a unified schema that represents the, an event that we will all bite down on. >>Even some of us are competitors, you know, this is, this is that, that no longer matters because at the point, the point is how do we take this burden off of our customers and how do we make the industry safer together? And so 15 initial members came together along with AWS and Splunk to, to start to create that, that initial schema and standardize it. And if you've ever, you know, if you've ever worked with a bunch of technical grumpy security people, it's kind of hard to drive consensus about around just about anything. But, but I, I'm really happy to see how quickly this, this organization has come together, has open sourced the schema, and, and, and just as you said, like I think this, this unlocks the potential for real innovation that's gonna be required to keep up with the bad guys. But right now is getting stymied and held back by the lack of normalization and the lack of integration. >>I've always said Splunk was a, it eats data for breakfast, lunch, and dinner and turns it into insights. And I think you bring up the silo thing. What's interesting is the cross company sharing, I think this hits point on, so I see this as a valuable opportunity for the industry. What's the traction on that? Because, you know, to succeed it does take a village, it takes a community of security practitioners and, and, and architects and developers to kind of coalesce around this defacto movement has been, has been the uptake been good? How's traction? Can you share your thoughts on how this is translating across companies? >>Yeah, absolutely. I mean, look, I, I think cybersecurity has a, has a long track record of, of, of standards development. There's been some fantastic standards recently. Things like sticks and taxi for threat intelligence. There's been things like the, you know, the Mir attack framework coming outta mi mir and, and, and the adoption, the traction that we've seen with Attack in particular has been amazing to, to watch how that has kind of roared onto the scene in the last couple of years and has become table stakes for how you do security operations and incident response. And, you know, I think with ocs f we're gonna see something similar here, but, you know, we are in literally the first innings of, of this. So right now, you know, we're architecting this into our, into every part of our sort of backend systems here at Polan. I know our our collaborators at AWS and elsewhere are doing it too. >>And so I think it starts with bringing this standard now that the standard exists on a, you know, in schema format and there, there's, you know, confluence and Jira tickets around it, how do we then sort of build this into the code of, of the, the collaborators that have been leading the way on this? And you know, it's not gonna happen overnight, but I think in the coming quarters you'll start to see this schema be the standard across the leaders in this space. Companies like Splunk and AWS and others who are leading the way. And often that's what helps drive adoption of a standard is if you can get the, the big dogs, so to speak, to, to, to embrace it. And, and, you know, there's no bigger one than aws and I think there's no, no more important one than Splunk in the cybersecurity space. And so as we adopt this, we hope others will follow. And, and like I said, we've got over 50 organizations contributing to it today. And so I think we're off to a running >>Start. You know, it's interesting, choking innovation or having things kind of get, get slowed down has really been a problem. We've seen successes recently over the past few years. Like Kubernetes has really unlocked and accelerated the cloud native worlds of runtime with containers to, to kind of have the consensus of the community to say, Hey, if we just do this, it gets better. I think this is really compelling with the o the ocs F because if people can come together around this and get unified as well as all the other official standards, things can go highly accelerated. So I think, I think it looks really good and I think it's great initiative and I really appreciate your insight on that, on, on your relationship with Amazon. Okay. It's not just a partnership, it's a strategic collaboration. Could you share that relationship dynamic, how to start, how's it going, what's strategic about it? Share to the audience kind of the relationship between Splunk and a on this important OCS ocsf initiative. >>Look, I, I mean I think this, this year marks the, the 10th year anniversary that, that Splunk and AWS have been collaborating in a variety of different ways. I, I think our, our companies have a fantastic and, and long standing relationship and we've, we've partnered on a number of really important projects together that bring value obviously to our individual companies, but also to our shared customers. When I think about some of the most important customers at Splunk that I spend a significant amount of time with, I I I know how many of those are, are AWS customers as well, and I know how important AWS is to them. So I think it's, it's a, it's a collaboration that is rooted in, in a respect for each other's technologies and innovation, but also in a recognition that, that our shared customers want to see us work better together over time. And it's not, it's not two companies that have kind of decided in a back room that they should work together. It's actually our customers that are, that are pushing us. And I think we're, we're both very customer centric organizations and I think that has helped us actually be better collaborators and better partners together because we're, we're working back backwards from our customers >>As security becomes a physical and software approach. We've seen the trend where even Steven Schmidt at Amazon Web Services is, is the cso, he is not the CSO anymore. So, and I asked him why, he says, well, security's also physical stuff too. So, so he's that's right. Whole lens is now expanded. You mentioned supply chain, physical, digital, this is an important inflection point. Can you summarize in your mind why open cybersecurity schema for is important? I know the unification, but beyond that, what, why is this so important? Why should people pay attention to this? >>You know, I, if, if you'll let me be just a little abstract in meta for a second. I think what's, what's really meaningful at the highest level about the O C S F initiative, and that goes beyond, I think, the tactical value it will provide to, to organizations and to customers in terms of making them safer over the coming years and, and decades. I think what's more important than that is it's really the, one of the first times that you've seen the industry come together and say, we got a problem. We need to solve. That, you know, doesn't really have anything to do with, with our own economics. Our customers are, are hurt. And yeah, some of us may be competitors, you know, we got different cloud service providers that are participating in this along with aws. We got different cybersecurity solution providers participating in this along with Splunk. >>But, but folks who've come together and say, we can actually solve this problem if, if we're able to kind of put aside our competitive differences in the markets and approach this from the perspective of what's best for information security as a whole. And, and I think that's what I'm most proud of and, and what I hope we can do more of in other places in this industry, because I think that kind of collaboration from real market leaders can actually change markets. It can change the, the, the trend lines in terms of how we are keeping up with the bad guys. And, and I'd like to see a lot more of >>That. And we're seeing a lot more new kind of things emerging in the cloud next kind of this next generation architecture and outcomes are happening. I think it's interesting, you know, we always talk about sustainability, supply chain sustainability about making the earth a better place. But you're hitting on this, this meta point about businesses are under threat of going under. I mean, we want to keep businesses to businesses to be sustainable, not just, you know, the, the environment. So if a business goes outta business business, which they, their threats here are, can be catastrophic for companies. I mean, there is, there is a community responsibility to protect businesses so they can sustain and and stay Yeah. Stay producing. This is a real key point. >>Yeah. Yeah. I mean, look, I think, I think one of the things that, you know, we, we, we complain a lot of in, in cyber security about the lack of, of talent, the talent shortage in cyber security. And every year we kinda, we kind of whack ourselves over the head about how hard it is to bring people into this industry. And it's true. But one of the things that I think we forget, John, is, is how important mission is to so many people in what they do for a living and how they work. And I think one of the things that cybersecurity is strongest in information Security General and has been for decades is this sense of mission and people work in this industry be not because it's, it's, it's always the, the, the most lucrative, but because it, it really drives a sense of safety and security in the enterprises and the fabric of the economy that we use every day to go through our lives. And when I think about the spun customers and AWS customers, I think about the, the different products and tools that power my life and, and we need to secure them. And, and sometimes that means coming to work every day at that company and, and doing your job. And sometimes that means working with others better, faster, and stronger to help drive that level of, of, of maturity and security that this industry >>Needs. It's a human, is a human opportunity, human problem and, and challenge. That's a whole nother segment. The role of the talent and the human machines and with scale. Patrick, thanks so much for sharing the information and the insight on the Open cybersecurity schema frame and what it means and why it's important. Thanks for sharing on the Cube, really appreciate it. >>Thanks for having me, John. >>Okay, this is AWS Reinvent 2022 coverage here on the Cube. I'm John Furry, you're the host. Thanks for watching.

(gentle music) >> Good morning from Las Vegas. It's theCUBE live at AWS Reinvent 2022 with tons of thousands of people today. Really kicks off the event. Big keynote that I think is probably just wrapping up. Lisa Martin here with Dave Vellante. Dave, this is going to be an action packed week on theCUBE no doubt. We talked with so many different companies. Every company's a software company these days but we're also seeing a lot of companies leaving software that can help them operate more efficiently in the background. >> Yeah, well some things haven't changed at Reinvent. A lot of people here, you know, back to 2019 highs and I think we exceeded those two hour keynotes. Peter DeSantis last night talking about new Graviton instances and then Adam Selipsky doing the typical two hour keynote. But what was different he was a lot more poetic than we used to hear from Andy Jassy, right? He was talking about the universe as an analogy for data. >> I loved that. >> Talked about ocean exploration as for the security piece and then exploring into the Antarctic for, you know, better chips, you know? So yeah, I think he did a good job there. I think a lot of people might not love it but I thought it was very well done. >> I thought so too. We're having kicking off a great day of live content for you all day today. We've got Eleanor Dorfman joining us, the sales leader at Retool. Eleanor, welcome to theCUBE. It's great to have you. >> Thank you so much for having me. >> So let's talk a little bit about Retool. I was looking on your LinkedIn page. I love the tagline, build custom internal tools best. >> Eleanor: Yep. >> Talk to us a little bit about the company you recently raised, series C two. Give us the backstory. >> Yeah, so the company was founded in 2017 by two co-founders who are best friends from college. They actually set out to build a FinTech company, a payments company. And as they were building that, they needed to build a ton of custom operations software that goes with that. If you're going to be managing people's money, you need to be able to do refunds. You need to be able to look up accounts, you need to be able to detect fraud, you need to do know your customer operations. And as they were building the sort of operations software that supports the business, they realized that there were patterns to all of it and that the same components were used at and again. And had the insight that that was actually probably a better direction to go in than recreating Venmo, which was I think the original idea. And that actually this is a problem every company has because every company needs operations engineering and operations software to run their business. And so they pivoted and started building Retool which is a platform for building custom operations software or internal tools. >> Dave: Good pivot. >> In hindsight, actually probably in the moment as well, was a good pivot. >> But you know, when you talk about some of those things, refunds, fraud, you know, KYC, you know, you think of operations software, you think of it as just internal, but all those things are customer facing. >> Eleanor: Yep. >> Right so, are we seeing as sort of this new era? Is that a trend that you guys, your founders saw that hey, these internal operations can be pointed at customers to support what, a better customer service, maybe even generate revenue, subscriptions? >> I think it's a direction we're actually heading now but we're just starting to scratch the surface of that. The focus for the last five years has very much been on this operations software and sort of changing the economics of developing it and making it easy and fast to productize workflows that were previously being done in spreadsheets or hacky workarounds and make it easier for companies to prioritize those so they can run their business more efficiently. >> And where are you having your customer conversations these days? Thinking of operations software in the background, but to Dave's point, it ends up being part of the customer experience. So where are you having your customer conversations, target audience, who's that persona? >> Mainly developers. So we're working almost exclusively with developer teams who have backlogs and backlogs of internal tools requests to build that sales teams are building manual forecasts. Support teams are in 19 different tools. Their supply chain teams are using seven different spreadsheets to do demand forecasting or freight forwarding or things like that. But they've never been able to be prioritized to the top of the list because customer facing software, revenue generating software, always takes prioritization. And in this economic environment, which is challenging for many companies right now, it's important to be able to do more with less and maximize the productivity especially of high value employees like engineers and developers. >> So what would you say the biggest business outcomes are? If the developer is really the focus, productivity is the- >> Productivity. It's for both, I would say. Developer productivity and being able to maximize your sort of R and D and maximize the productivity of your engineers and take away some of the very boring parts of the job. But, so I would say developer productivity, but then also the tools and the software that they're building are very powerful for end users. So I would say efficiency and productivity across your business. >> Across the business. >> I mean historically, you know, operations is where we focused IT and code. How much of the code out there is dedicated to sort of operations versus that customer facing? >> So I think it would actually be, it's kind of surprising. We have run a few surveys on this sort of, we call them the state of engineering time, and focusing on what developers are spending their time on. And a third of all code that is being written today is actually for this internal operations software. >> Interesting. And do you guys have news at the show? Are you announcing anything interesting or? >> Yeah, so our focus historically, you sort of gave away with one of your early questions, but our focus has always been on this operations, this building web applications on building UIs on top of databases and APIs and doing that incredibly fast and being able to do it all in one place and integrate with as any data source that you need. We abstract away access authentication deployment and you build applications for your internal teams. But recently, we've launched two new products. We're actually supporting more external use cases and more customer facing use cases as well as automating CRON jobs, ETL jobs alerting with the new retail workflows product. So we're expanding the scope of operations software from web applications to also internal operations like CRON jobs and ETL jobs. >> Explain that. Explain the scourge of CRON jobs to the audience. >> Yeah, so operations software businesses run on operations software. It's interesting, zooming out, it's actually something you said earlier as well. Every company has become a software company. So when you think about software, you tend to think about here. Very cool software that people are selling. And software that you use as a consumer. But Coca-Cola for example, has hundreds of software engineers that are building tools to make the business run for forecasting, for demand gen, for their warehouse distribution and monitoring inventory. And there's two types of that. There's the applications that they build and then the operations that have to run behind that. Maybe a workflow that is detecting how many bottles of Coca-Cola are in every warehouse and sending a notification to the right person when they're out or when they, a refill is very strong, but you know when you need a refill. So it does that, it takes those tasks, those jobs that run in the background and enables you to customize them and build them very rapidly in a code first way. >> So some of the notes that you guys provided say that there's over 500 million software apps that are going to be built in the next few years alone. That's tremendous. How much of that is operation software? >> I mean I think at least a third of that, if not more. To the point where every company is being forced to maximize their resources today and operational efficiency is the way to do that. And so it can become a competitive advantage when you can take the things that humans are doing in spreadsheets with 19 open tabs and automate that. That saves hours a day. That's a significant, significant driver of efficiency and productivity for a business >> It does, and there's direct correlation to the customer experience. The use experience. >> Almost certainly. When you think about building support tooling, I was web chat, chatting on the with Gogo wifi support on my flight over here and they asked for my order number and I sent it and they looked up my account and that's a custom piece of software they were using to look up the account, create a new account for me, and restore my second wifi purchase. And so when you think about it, you're actually, even just as a consumer, interacting with this custom software on the day time. And that's because that's what companies use to have a good customer experience and have an efficient business. >> And what's the relationship with AWS? You guys started, I think you said 2017, so you obviously started in the cloud, but I'm particularly interested in from a seller perspective, what that's like. Working with Amazon, how's that affected your business? >> Yeah, I mean so we're built on AWS, so we're customers and big fans. And obviously like from a selling perspective, we have a ton of integrations with AWS so we're able to integrate directly into all the different AWS products that people are using for databases, for data warehouses, for deployment configurations, for monitoring, for security, for observability, we can basically fit into your existing AWS stack in order to make it as seamless integration with your software so that building in Retool is just as seamless as building it on your own, just much, much faster. >> So in your world, I know you wanted to but, in your world is it more analytics? is it more transactional, sort of? Is it both? >> It's all of the above. And I think what's, over Thanksgiving, I was asked a lot to explain what Retool did with people who were like, we just got our first iPhone. And so I tried to explain with an example because I have yet to stumble on the perfect metaphor. But the example I typically use is DoorDash is a customer of ours. And for about three years, and three years ago, they had a problem. They had no way of turning off delivery in certain zip codes during storms. Which as someone who has had orders canceled during a storm, it's an incredibly frustrating experience. And the way it worked is that they had operation team members manually submitting requests to engineers to say there's a storm in this zip code and an engineer would run a manual task. This didn't scale with Doordash as they were opening in new countries all over the world that have very different weather patterns. And so they looked, they had one, they were sort of confronted with a choice. They could buy a piece of software out of the box. There is not a startup that does this yet. They could build it by hand, which would mean scoping the requirements designing a UI, building authentication, building access controls, putting it into a, putting it into a sprint, assigning an engineer. This would've taken months and months. And then it would take just as long to iterate on it or they could use Retool. So they used Retool, they built this app, it saved, I think they were saying up to two years of engineering time for this one application because of how quickly it was. And since then they've built, I think 50 or 60 more automating away other tasks like that that were one out of spreadsheets or in Jira or in Slack notifications or an email saying, "Hey, could you please do this thing? There's a storm." And so now they use us for dozens and dozens of operations like that. >> A lot of automation and of course a lot of customer delight on the other end of the spectrum as you were talking about. It is frustrating when you don't get that order but it's also the company needs to be able to have the the tools in place to automate to be able to react quickly. >> Eleanor: Exactly. >> Because the consumers are, as we know, quite demanding. I wanted to ask you, I mentioned the tagline in the beginning, build custom internal tools fast. You just gave us a great example of DoorDash. Huge business outcomes they're achieving but how fast are we talking? How fast can the average developer build these internal tools? >> Well, we've been doing a fun thing at our booth where we ask people what a problem is and build a tool for them while we're there. So for something lightweight, you can build it in 10 minutes. For something a little more complex, it can take up to a few weeks depending on what the requirements are. But we all have people who will be on a call with us introducing them to our software for the first time and they'll start telling us about their problems and in the background we'll be building it and then at the end we're like, is this what you meant? And they're like, we'd like to add that to our cart. And obviously, it's a platform so you can't do that. But we've been able to build applications on a call before while people are telling us what they need. >> So fast is fast. >> I would say very fast, yeah. >> Now how do you price? >> Right now, we have a couple different plans. We actually have a motion where you can sign up on our website and get started. So we have a free plan, we've got plans for startups, and then we've got plans all the way up to the enterprise. >> Right. And that's a subscription pricing kind of thing? >> Subscription model, yes. >> So I get a subscription to the platform and then what? Is there also a consumption component? >> Exactly. So there's a consumption component as well. So there's access to the platform and then you can build as many applications as you need. Or build as many workflows. >> When you're having customer conversations with prospects, what do you define as Retool's superpowers? You're the sales leader. What are some of those key superpowers that you think really differentiate Retool? >> I do think, well, the sales team first and foremost, but that's not a fair answer. I would say that people are a bit differentiator though. We have a lot of very talented people who are have a ton of domain expertise and care a ton about the customer outcomes, which I do actually think is a little more rare than it should be. But we're one of the only products out there that's built with a developer first mindset, a varied code first mindset, built to integrate with your software development life cycle but also built with the security and robustness that enterprise companies require. So it's able to take an enterprise grade software with a developer first approach while still having a ton of agility and nimbleness which is what people are really craving as the earth keeps moving around them. So I would say that's something that really sets us apart from the field. >> And then talk about some of the what developers are saying, some of the feedback, some of the responses, and maybe even, I know we're just on day one of the show, but any feedback from the booth so far? >> We've had a few people swing by our booth and show us their Retool apps, which is incredibly cool. That's my absolute favorite thing is encountering a Retool application in the wild which happens a lot more than I would've thought, which I shouldn't say, but is incredibly rewarding. But people love it. It's the reason I joined is I'd never heard someone have a product that customers talked about the way they talk about Retool because Retool enables them to do things. For some folks who use it, it enables them to do something they previously couldn't do. So it gives them super powers in their job and to triple their impact. And then for others, it just makes things so fast. And it's a very delightful experience. It's very much built by developers, for developers. And so it's built with a developer's first mindset. And so I think it's quite fun to build in Retool. Even I can build and Retool, though not well. And then it's extremely impactful and people are able to really impact their business and delight their coworkers which I think can be really meaningful. >> Absolutely. Delighting the coworkers directly relates to delighting the customers. >> Eleanor: Exactly. >> Those customer experience, employee experience, they're like this. >> Eleanor: Exactly. >> They go hand in hand and the employee experience has to be outstanding to be able to delight those customers, to reduce churn, to increase revenue- >> Eleanor: Exactly. >> And for brand reputation. >> And it also, I think there is something as someone who is customer facing, when my coworkers and developers I work with build tools that enable me to do my job better and feel better about my own performance and my ability to impact the customer experience, it's just this incredibly virtuous cycle. >> So Retool.com is where folks can go to learn more and also try that subscription that you said was free for up to five users. >> Yes, exactly. >> All right. I guess my last question, well couple questions for you. What are some of the things that excited you that you heard from Adam Selipsky this morning? Anything from the keynote that stood out in terms of- >> Dave: Did you listen to the keynote? >> I did not. I had customer calls this morning. >> Okay, so they're bringing- >> East coast time, east coast time. >> One of the things that will excite you I think is they're connecting, making it easier to connect their databases. >> Eleanor: That would very much exciting. >> Aurora and Redshift, right? Okay. And they're making it easier to share data. I dunno if it goes across regions, but they're doing better integration. >> Amazing. >> Right? And you guys are integrating with those tools, right? Those data platforms. So that to me was a big thing for you guys. >> It is also and what a big thing Retool does is you can build a UI layer for your application on top of every single data source. And you hear, it's funny, you hear people talk about the 360 degree review of the customer so much. This is another, it's not our primary value proposition, but it is certainly another way to get there is if you have data from their desk tickets from in Redshift, you have data from Stripe, from their payments, you have data from Twilio from their text messages, you have data from DataDog where they're having your observability where you can notice analytics issues. You can actually just use Retool to build an app that sits on top of that so that you can give your support team, your sales team, your account management team, customer service team, all of the data that they need on their customers. And then you can build workflows so that you can do automated customer engagement reports. I did a Slack every week that shows what our top customers are doing with the product and that's built using all of our automation software as well. >> The integration is so important, as you just articulated, because every, you know, we say every company's a software company these days. Every company's a data company. But also, the data democratization that needs to happen to be able for lines of business so that data moves out of certain locked in functions and enables lines of business to use it. To get that visibility that you were just talking about is really going to be a competitive advantage for those that survive and thrive and grow in this market. >> It's able to, I think it's first it's visibility, but then it's action. And I think that's what Retool does very uniquely as well is it can take and unite the data from all the places, takes it out of the black box, puts it in front of the teams, and then enables them to act on it safely and securely. So not only can you see who might be fraudulent, you can flag them as fraud. Not only can you see who's actually in danger, you can click a button and send them an email and set up a meeting. You can set up an approval workflow to bring in an exec for engagement. You can update a password for someone in one place where you can see that they're having issues and not have to go somewhere else to update the password. So I think that's the key is that Retool can unlock the data visibility and then the action that you need to serve your customers. >> That's a great point. It's all about the actions, the insights that those actions can be acted upon. Last question for you. If you had a billboard that you could put any message that you want on Retool, what would it say? What's the big aha? This is why Retool is so great. >> I mean, I think the big thing about Retool is it's changing the economics of software development. It takes something that previously would've been below the line and that wouldn't get prioritized because it wasn't customer facing and makes it possible. And so I would say one of two billboards if I could be a little bit greedy, one would be Retool changed the economics of software development and one would be build operations software at the speed of thought. >> I love that. You're granted two billboards. >> Eleanor: Thank you. >> Those are both outstanding. Eleanor, it's been such a pleasure having you on the program. Thank you for talking to us about Retool. >> Eleanor: Thank you. >> Operations software and the massive impact that automating it can make for developers, businesses alike, all the way to the top line. We appreciate your insights. >> Thank you so much. >> For our guests and Dave Vellante, I'm Lisa Martin. You're watching theCUBE, the leader in live, emerging, and enterprise tech coverage. (gentle music)

(upbeat music) >> Hey everyone, welcome to this CUBE conversation. I'm John Furrier, your host of theCUBE here in Palo Alto, California. We've got Sam Kassoumeh, co-founder and chief operating office at SecurityScorecard here remotely coming in. Thanks for coming on Sam. Security, Sam. Thanks for coming on. >> Thank you, John. Thanks for having me. >> Love the security conversations. I love what you guys are doing. I think this idea of managed services, SaaS. Developers love it. Operation teams love getting into tools easily and having values what you guys got with SecurityScorecard. So let's get into what we were talking before we came on. You guys have a unique solution around ratings, but also it's not your grandfather's pen test want to be security app. Take us through what you guys are doing at SecurityScorecard. >> Yeah. So just like you said, it's not a point in time assessment and it's similar to a traditional credit rating, but also a little bit different. You can really think about it in three steps. In step one, what we're doing is we're doing threat intelligence data collection. We invest really heavily into R&D function. We never stop investing in R&D. We collect all of our own data across the entire IPV force space. All of the different layers. Some of the data we collect is pretty straightforward. We might crawl a website like the example I was giving. We might crawl a website and see that the website says copyright 2005, but we know it's 2022. Now, while that signal isn't enough to go hack and break into the company, it's definitely a signal that someone might not be keeping things up to date. And if a hacker saw that it might encourage them to dig deeper. To more complex signals where we're running one of the largest DNS single infrastructures in the world. We're monitoring command and control malware and its behaviors. We're essentially collecting signals and vulnerabilities from the entire IPV force space, the entire network layer, the entire web app player, leaked credentials. Everything that we think about when we talk about the security onion, we collect data at each one of those layers of the onion. That's step one. And we can do all sorts of interesting insights and information and reports just out of that thread intel. Now, step two is really interesting. What we do is we go identify the attack surface area or what we call the digital footprint of any company in the world. So as a customer, you can simply type in the name of a company and we identify all of the domains, sub domains, subsidiaries, organizations that are identified on the internet that belong to that organization. So every digital asset of every company we go out and we identify that and we update that every 24 hours. And step three is the rating. The rating is probabilistic and it's deterministic. The rating is a benchmark. We're looking at companies compared to their peers of similar size within the same industry and we're looking at how they're performing. And it's probabilistic in the sense that companies that have an F are about seven to eight times more likely to experience a breach. We're an A through F scale, universally understood. Ds and Fs, more likely to experience a breach. A's we see less breaches now. Like I was mentioning before, it doesn't mean that an F is always going to get hacked or an A can never get hacked. If a nation state targets an A, they're going to eventually get in with enough persistence and budget. If the pizza shop on the corner has an F, they may never get hacked because no one cares, but natural correlation, more doors open to the house equals higher likelihood someone unauthorized is going to walk in. So it's really those three steps. The collection, we map it to the surface area of the company and then we produce a rating. Today we're rating about 12 million companies every single day. >> And how many people do you have as customers? >> We have 50,000 organizations using us, both free and paid. We have a freemium tier where just like Yelp or a LinkedIn business profile. Any company in the world has a right to go claim the score. We never extort companies to fix the score. We never charge a company to see the score or fix it. Any company in a world without paying us a cent can go in. They can understand what we're seeing about them, what a hacker could see about their environment. And then we empower them with the tools to fix it and they can fix it and the score will go up. Now companies pay us because they want enterprise capabilities. They want additional modules, insights, which we can talk about. But in total, there's about 50,000 companies that at any given point in time, they're monitoring about a million and a half organizations of the 12 million that we're rating. It sounds like Google. >> If you want to look at it. >> Sounds like Google Search you got going on there. You got a lot of search and then you create relevance, a score, like a ranking. >> That's precisely it. And that's exactly why Google ventures invested in us in our Series B round. And they're on our board. They looked and they said, wow, you guys are building like a Google Search engine over some really impressive threat intelligence. And then you're distilling it into a score which anybody in the world can easily understand. >> Yeah. You obviously have page rank, which changed the organic search business in the late 90s, early 2000s and the rest is history. AdWords. >> Yeah. >> So you got a lot of customer growth there potentially with the opt-in customer view, but you're looking at this from the outside in. You're looking at companies and saying, what's your security posture? Getting a feel for what they got going on and giving them scores. It sounds like it's not like a hacker proof. It's just more of a indicator for management and the team. >> It's an indicator. It's an indicator. Because today, when we go look at our vendors, business partners, third parties were flying blind. We have no idea how they're doing, how they're performing. So the status quo for the last 20 years has been perform a risk assessments, send a questionnaire, ask for a pen test and an audit evidence. We're trying to break that cycle. Nobody enjoys it. They're long tail. It's a trust without verification. We don't really like that. So we think we can evolve beyond this point in time assessment and give a continuous view. Now, today, historically, we've been outside in. Not intrusive, and we'll show you what a hacker can see about an environment, but we have some cool things percolating under the hood that give more of a 360 view outside, inside, and also a regulatory compliance view as well. >> Why is the compliance of the whole third party thing that you're engaging with important? Because I mean, obviously having some sort of way to say, who am I dealing with is important. I mean, we hear all kinds of things in the security landscape, oh, zero trust, and then we hear trust, supply chain, software risk, for example. There's a huge trust factor there. I need to trust this tool or this container. And then you got the zero trust, don't trust anything. And then you've got trust and verify. So you have all these different models and postures, and it just seems hard to keep up with. >> Sam: It's so hard. >> Take us through what that means 'cause pen tests, SOC reports. I mean the clouds help with the SOC report, but if you're doing agile, anything DevOps, you basically would need to do a pen test like every minute. >> It's impossible. The market shifted to the cloud. We watched and it still is. And that created a lot of complexity, not to date myself. But when I was starting off as a security practitioner, the data center used to be in the basement and I would have lunch with the database administrator and we talk about how we were protecting the data. Those days are long gone. We outsource a lot of our key business practices. We might use, for example, ADP for a payroll provider or Dropbox to store our data. But we've shifted and we no longer no who that person is that's protecting our data. They're sitting in another company in another area unknown. And I think about 10, 15 years ago, CISOs had the realization, Hey, wait a second. I'm relying on that third party to function and operate and protect my data, but I don't have any insight, visibility or control of their program. And we were recommended to use questionnaires and audit forms, and those are great. It's good hygiene. It's good practice. Get to know the people that are protecting your data, ask them the questions, get the evidence. The challenge is it's point in time, it's limited. Sometimes the information is inaccurate. Not intentionally, I don't think people intentionally want to go lie, but Hey, if there's a $50 million deal we're trying to close and it's dependent on checking this one box, someone might bend a rule a little bit. >> And I said on theCUBE publicly that I think pen test reports are probably being fudged and dates being replicated because it's just too fast. And again, today's world is about velocity on developers, trust on the code. So you got all kinds of trust issues. So I think verification, the blue check mark on Twitter kind of thing going on, you're going to see a lot more of that and I think this is just the beginning. I think what you guys are doing is scratching the surface. I think this outside in is a good first step, but that's not going to solve the internal problem that still coming and have big surface areas. So you got more surface area expanding. I mean, IOT's coming in, the Edge is coming fast. Never mind hybrid on-premise cloud. What's your organizations do to evaluate the risk and the third party? Hands shaking, verification, scorecards. Is it like a free look here or is it more depth to it? Do you double click on it? Take us through how this evolves. >> John it's become so disparate and so complex, Because in addition to the market moving to the cloud, we're now completely decentralized. People are working from home or working hybrid, which adds more endpoints. Then what we've learned over time is that it's not just a third party problem, because guess what? My third parties behind the scenes are also using third parties. So while I might be relying on them to process my customer's payment information, they're relying on 20 vendors behind the scene that I don't even know about. I might have an A, they might have an A. It's really important that we expand beyond that. So coming out of our innovation hub, we've developed a number of key capabilities that allow us to expand the value for the customer. One, you mentioned, outside in is great, but it's limited. We can see what a hacker sees and that's helpful. It gives us pointers where to maybe go ask double click, get comfort, but there's a whole nother world going on behind the firewall inside of an organization. And there might be a lot of good things going on that CISO security teams need to be rewarded for. So we built an inside module and component that allows teams to start plugging in the tools, the capabilities, keys to their cloud environments. And that can show anybody who's looking at the scorecard. It's less like a credit score and more like a social platform where we can go and look at someone's profile and say, Hey, how are things going on the inside? Do they have two-factor off? Are there cloud instances configured correctly? And it's not a point in time. This is a live connection that's being made. This is any point in time, we can validate that. The other component that we created is called an evidence locker. And an evidence locker, it's like a secure vault in my scorecard and it allows me to upload things that you don't really stand for or check for. Collateral, compliance paperwork, SOC 2 reports. Those things that I always begrudgingly email. I don't want to share with people my trade secrets, my security policies, and have it sit on their exchange server. So instead of having to email the same documents out, 300 times a month, I just upload them to my evidence locker. And what's great is now anybody following my scorecard can proactively see all the great things I'm doing. They see the outside view. They see the inside view. They see the compliance view. And now they have the holy grail view of my environment and can have a more intelligent conversation. >> Access to data and access methods are an interesting innovation area around data lineage. Tracing is becoming a big thing. We're seeing that. I was just talking with the Snowflake co-founder the other day here in theCUBE about data access and they're building a proprietary mesh on top of the clouds to figure out, Hey, I don't want to give just some tool access to data because I don't know what's on the other side of those tools. Now they had a robust ecosystem. So I can see this whole vendor risk supply chain challenge around integration as a huge problem space that you guys are attacking. What's your reaction to that? >> Yeah. Integration is tricky because we want to be really particular about who we allow access into our environment or where we're punching holes in the firewall and piping data out out of the environment. And that can quickly become unwieldy just with the control that we have. Now, if we give access to a third party, we then don't have any control over who they're sharing our information with. When I talk to CISOs today about this challenge, a lot of folks are scratching their head, a lot of folks treat this as a pet project. Like how do I control the larger span beyond just the third parties? How do I know that their software partners, their contractors that they're working with building their tools are doing a good job? And even if I know, meaning, John, you might send me a list of all of your vendors. I don't want to be the bad guy. I don't really have the right to go reach out to my vendors' vendors knocking on their door saying, hi, I'm Sam. I'm working with John and he's your customer. And I need to make sure that you're protecting my data. It's an awkward chain of conversation. So we're building some tools that help the security teams hold the entire ecosystem accountable. We actually have a capability called automatic vendor discovery. We can go detect who are the vendors of a company based on the connections that we see, the inbound and outbound connections. And what often ends up happening John is we're bringing to the attention to our customers, awareness about inbound and outbound connections. They had no idea existed. There were the shadow IT and the ghost vendors that were signed without going through an assessment. We detect those connections and then they can go triage and reduce the risk accordingly. >> I think that risk assessment of vendors is key. I was just reading a story about this, about how a percentage, I forget the number. It was pretty large of applications that aren't even being used that are still on in companies. And that becomes a safe haven for bad actors to hang out and penetrate 'cause they get overlooked 'cause no one's using them, but they're still online. And so there's a whole, I called cleaning up the old dead applications that are still connected. >> That happens all the time. Those applications also have applications that are dead and applications that are alive may also have users that are dead as well. So you have that problem at the application level, at the user level. We also see a permutation of what you describe, which is leftover artifacts due to configuration mistakes. So a company just put up a new data center, a satellite office in Singapore and they hired a team to go install all the hardware. Somebody accidentally left an administrative portal exposed to the public internet and nobody knew the internet works, the lights are on, the office is up and running, but there was something that was supposed to be turned off that was left turned on. So sometimes we bring to company's attention and they say, that's not mine. That doesn't belong to me. And we're like, oh, well, we see some reason why. >> It's his fault. >> Yeah and they're like, oh, that was the contractor set up the thing. They forgot to turn off the administrative portal with the default login credentials. So we shut off those doors. >> Yeah. Sam, this is really something that's not talked about a lot in the industry that we've become so reliant on managed services and other people, CISOs, CIOs, and even all departments that have applications, even marketing departments, they become reliant on agencies and other parties to do stuff for them which inherently just increases the risk here of what they have. So there inherently could be as secure as they could be, but yet exposed completely on the other side. >> That's right. We have so many virtual touch points with our partners, our vendors, our managed service providers, suppliers, other third parties, and all the humans that are involved in that mix. It creates just a massive ripple effect. So everybody in a chain can be doing things right. And if there's one bad link, the whole chain breaks. I know it's like the cliche analogy, but it rings true. >> Supply chain trust again. Trust who you trust. Let's see how those all reconcile. So Sam, I have to ask you, okay, you're a former CISO. You've seen many movies in the industry. Co-founded this company. You're in the front lines. You've got some cool things happening. I can almost imagine the vision is a lot more than just providing a rating and score. I'm sure there's more vision around intelligence, automation. You mentioned vault, wallet capabilities, exchanging keys. We heard at re:Inforce automated reasoning, metadata reasoning. You got all kinds of crypto and quantum. I mean, there's a lot going on that you can tap into. What's your vision where you see SecurityScorecard going? >> When we started the company, the rating was the thing that we sold and it was a language that helped technical and non-technical folks alike level the playing field and talk about risk and use it to drive their strategy. Today, the rating just opens the door to that discussion and there's so much additional value. I think in the next one to two years, we're going to see the rating becomes standardized. It's going to be more frequently asked or even required or leveraged by key decision makers. When we're doing business, it's going to be like, Hey, show me your scorecard. So I'm seeing the rating get baked more and more the lexicon of risk. But beyond the rating, the goal is really to make a world a safer place. Help transform and rise the tide. So all ships can lift. In order to do that, we have to help companies, not only identify the risk, but also rectify the risk. So there's tools we build to really understand the full risk. Like we talked about the inside, the outside, the fourth parties, fifth parties, the real ecosystem. Once we identified where are all the Fs and bad things, will then what? So couple things that we're doing. We've launched a pro serve arm to help companies. Now companies don't have to pay to fix the score. Anybody, like I said, can fix the score completely free of charge, but some companies need help. They ask us and they say, Hey, I'm looking for a trusted advisor. A Sherpa, a guide to get me to a better place or they'll say, Hey, I need some pen testing services. So we've augmented a service arm to help accelerate the remediation efforts. We're also partnered with different industries that use the rating as part of a larger picture. The cyber rating isn't the end all be all. When companies are assessing risk, they may be looking at a financial ratings, ESG ratings, KYC AML, cyber security, and they're trying to form a complete risk profile. So we go and we integrate into those decision points. Insurance companies, all the top insurers, re-insurers, brokers are leveraging SecurityScorecard as an ingredient to help underwrite for cyber liability insurance. It's not the only ingredient, but it helps them underwrite and identify the help and price the risk so they can push out a policy faster. First policy is usually the one that's signed. So time to quote is an important metric. We help to accelerate that. We partner with credit rating agencies like Fitch, who are talking to board members, who are asking, Hey, I need a third party, independent verification of what my CISO is saying. So the CISO is presenting the rating, but so are the proxy advisors and the ratings companies to the board. So we're helping to inform the boards and evolve how they're thinking about cyber risk. We're helping with the insurance space. I think that, like you said, we're only scratching the surface. I can see, today we have about 50,000 companies that are engaging a rating and there's no reason why it's not going to be in the millions in just the next couple years here. >> And you got the capability to bring in more telemetry and see the new things, bring that into the index, bring that into the scorecard and then map that to potential any vulnerabilities. >> Bingo. >> But like you said, the old days, when you were dating yourself, you were in a glass room with a door lock and key and you can see who's two folks in there having lunch, talking database. No one's going to get hurt. Now that's gone, right? So now you don't know who's out there and machines. So you got humans that you don't know and you got machines that are turning on and off services, putting containers out there. Who knows what's in those payloads. So a ton of surface area and complexity to weave through. I mean only is going to get done with automation. >> It's the only way. Part of our vision includes not attempting to make a faster questionnaire, but rid ourselves of the process all altogether and get more into the continuous assessment mindset. Now look, as a former CISO myself, I don't want another tool to log into. We already have 50 tools we log into every day. Folks don't need a 51st and that's not the intent. So what we've done is we've created today, an automation suite, I call it, set it and forget it. Like I'm probably dating myself, but like those old infomercials. And look, and you've got what? 50,000 vendors business partners. Then behind there, there's another a hundred thousand that they're using. How are you going to keep track of all those folks? You're not going to log in every day. You're going to set rules and parameters about the things that you care about and you care depending on the nature of the engagement. If we're exchanging sensitive data on the network layer, you might care about exposed database. If we're doing it on the app layer, you're going to look at application security vulnerabilities. So what our customers do is they go create rules that say, Hey, if any of these companies in my tier one critical vendor watch list, if they have any of these parameters, if the score drops, if they drop below a B, if they have these issues, pick these actions and the actions could be, send them a questionnaire. We can send the questionnaire for you. You don't have to send pen and paper, forget about it. You're going to open your email and drag the Excel spreadsheet. Those days are over. We're done with that. We automate that. You don't want to send a questionnaire, send a report. We have integrations, notify Slack, create a Jira ticket, pipe it to ServiceNow. Whatever system of record, system of intelligence, workflow tools companies are using, we write in and allow them to expedite the whole. We're trying to close the window. We want to close the window of the attack. And in order to do that, we have to bring the attention to the people as quickly as possible. That's not going to happen if someone logs in every day. So we've got the platform and then that automation capability on top of it. >> I love the vision. I love the utility of a scorecard, a verification mark, something that could be presented, credential, an image, social proof. To security and an ongoing way to monitor it, observe it, update it, add value. I think this is only going to be the beginning of what I would see as much more of a new way to think about credentialing companies. >> I think we're going to reach a point, John, where and some of our customers are already doing this. They're publishing their scorecard in the public domain, not with the technical details, but an abstracted view. And thought leaders, what they're doing is they're saying, Hey, before you send me anything, look at my scorecard securityscorecard.com/securityrating, and then the name of their company, and it's there. It's in the public domain. If somebody Googles scorecard for certain companies, it's going to show up in the Google Search results. They can mitigate probably 30, 40% of inbound requests by just pointing to that thing. So we want to give more of those tools, turn security from a reactive to a proactive motion. >> Great stuff, Sam. I love it. I'm going to make sure when you hit our site, our company, we've got camouflage sites so we can make sure you get the right ones. I'm sure we got some copyright dates. >> We can navigate the decoys. We can navigate the decoys sites. >> Sam, thanks for coming on. And looking forward to speaking more in depth on showcase that we have upcoming Amazon Startup Showcase where you guys are going to be presenting. But I really appreciate this conversation. Thanks for sharing what you guys are working on. We really appreciate. Thanks for coming on. >> Thank you so much, John. Thank you for having me. >> Okay. This is theCUBE conversation here in Palo Alto, California. Coming in from New York city is the co-founder, chief operating officer of securityscorecard.com. I'm John Furrier. Thanks for watching. (gentle music)

(upbeat music) >> Hello, everyone. Welcome to the theCUBE's presentation of the AWS Startup Showcase. The theme, this showcase is MarTech, the emerging cloud scale customer experiences. Season two of episode three, the ongoing series covering the startups, the hot startups, talking about analytics, data, all things MarTech. I'm your host, John Furrier, here joined by Christian Wiklund, founder and CEO of unitQ here, talk about harnessing the power of user feedback to empower marketing. Thanks for joining us today. >> Thank you so much, John. Happy to be here. >> In these new shifts in the market, when you got cloud scale, open source software is completely changing the software business. We know that. There's no longer a software category. It's cloud, integration, data. That's the new normal. That's the new category, right? So as companies are building their products, and want to do a good job, it used to be, you send out surveys, you try to get the product market fit. And if you were smart, you got it right the third, fourth, 10th time. If you were lucky, like some companies, you get it right the first time. But the holy grail is to get it right the first time. And now, this new data acquisition opportunities that you guys in the middle of that can tap customers or prospects or end users to get data before things are shipped, or built, or to iterate on products. This is the customer feedback loop or data, voice of the customer journey. It's a gold mine. And it's you guys, it's your secret weapon. Take us through what this is about now. I mean, it's not just surveys. What's different? >> So yeah, if we go back to why are we building unitQ? Which is we want to build a quality company. Which is basically, how do we enable other companies to build higher quality experiences by tapping into all of the existing data assets? And the one we are in particularly excited about is user feedback. So me and my co-founder, Nik, and we're doing now the second company together. We spent 14 years. So we're like an old married couple. We accept each other, and we don't fight anymore, which is great. We did a consumer company called Skout, which was sold five years ago. And Skout was kind of early in the whole mobile first. I guess, we were actually mobile first company. And when we launched this one, we immediately had the entire world as our marketplace, right? Like any modern company. We launch a product, we have support for many languages. It's multiple platforms. We have Android, iOS, web, big screens, small screens, and that brings some complexities as it relates to staying on top of the quality of the experience because how do I test everything? >> John: Yeah. >> Pre-production. How do I make sure that our Polish Android users are having a good day? And we found at Skout, personally, like I could discover million dollar bugs by just drinking coffee and reading feedback. And we're like, "Well, there's got to be a better way to actually harness the end user feedback. That they are leaving in so many different places." So, you know what, what unitQ does is that we basically aggregate all different sources of user feedback, which can be app store reviews, Reddit posts, Tweets, comments on your Facebook ads. It can be better Business Bureau Reports. We don't like to get to many of those, of course. But really, anything on the public domain that mentions or refers to your product, we want to ingest that data in this machine, and then all the private sources. So you probably have a support system deployed, a Zendesk, or an Intercom. You might have a chatbot like an Ada, or and so forth. And your end user is going to leave a lot of feedback there as well. So we take all of these channels, plug it into the machine, and then we're able to take this qualitative data. Which and I actually think like, when an end user leaves a piece of feedback, it's an act of love. They took time out of the day, and they're going to tell you, "Hey, this is not working for me," or, "Hey, this is working for me," and they're giving you feedback. But how do we package these very messy, multi-channel, multiple languages, all over the place data? How can we distill it into something that's quantifiable? Because I want to be able to monitor these different signals. So I want to turn user feedback into time series. 'Cause with time series, I can now treat this the same way as Datadog treats machine logs. I want to be able to see anomalies, and I want to know when something breaks. So what we do here is that we break down your data in something called quality monitors, which is basically machine learning models that can aggregate the same type of feedback data in this very fine grained and discrete buckets. And we deploy up to a thousand of these quality monitors per product. And so we can get down to the root cause. Let's say, passive reset link is not working. And it's in that root cause, the granularity that we see that companies take action on the data. And I think historically, there has been like the workflow between marketing and support, and engineering and product has been a bit broken. They've been siloed from a data perspective. They've been siloed from a workflow perspective, where support will get a bunch of tickets around some issue in production. And they're trained to copy and paste some examples, and throw it over the wall, file a Jira ticket, and then they don't know what happens. So what we see with the platform we built is that these teams are able to rally around the single source of troop or like, yes, passive recent link seems to have broken. This is not a user error. It's not a fix later, or I can't reproduce. We're looking at the data, and yes, something broke. We need to fix it. >> I mean, the data silos a huge issue. Different channels, omnichannel. Now, there's more and more channels that people are talking in. So that's huge. I want to get to that. But also, you said that it's a labor of love to leave a comment or a feedback. But also, I remember from my early days, breaking into the business at IBM and Hewlett-Packard, where I worked. People who complain are the most loyal customers, if you service them. So it's complaints. >> Christian: Yeah. >> It's leaving feedback. And then, there's also reading between the lines with app errors or potentially what's going on under the covers that people may not be complaining about, but they're leaving maybe gesture data or some sort of digital trail. >> Yeah. >> So this is the confluence of the multitude of data sources. And then you got the siloed locations. >> Siloed locations. >> It's complicated problem. >> It's very complicated. And when you think about, so I started, I came to Bay Area in 2005. My dream was to be a quant analyst on Wall Street, and I ended up in QA at VMware. So I started at VMware in Palo Alto, and didn't have a driver's license. I had to bike around, which was super exciting. And we were shipping box software, right? This was literally a box with a DVD that's been burned, and if that DVD had bugs in it, guess what it'll be very costly to then have to ship out, and everything. So I love the VMware example because the test cycles were long and brutal. It was like a six month deal to get through all these different cases, and they couldn't be any bugs. But then as the industry moved into the cloud, CI/CD, ship at will. And if you look at the modern company, you'll have at least 20 plus integrations into your product. Analytics, add that's the case, authentication, that's the case, and so forth. And these integrations, they morph, and they break. And you have connectivity issues. Is your product working as well on Caltrain, when you're driving up and down, versus wifi? You have language specific bugs that happen. Android is also quite a fragmented market. The binary may not perform as well on that device, or is that device. So how do we make sure that we test everything before we ship? The answer is, we can't. There's no company today that can test everything before the ship. In particular, in consumer. And the epiphany we had at our last company, Skout, was that, "Hey, wait a minute. The end user, they're testing every configuration." They're sitting on the latest device, the oldest device. They're sitting on Japanese language, on Swedish language. >> John: Yeah. >> They are in different code paths because our product executed differently, depending on if you were a paid user, or a freemium user, or if you were certain demographical data. There's so many ways that you would have to test. And PagerDuty actually had a study they came out with recently, where they said 51% of all end user impacting issues are discovered first by the end user, when they serve with a bunch of customers. And again, like the cool part is, they will tell you what's not working. So now, how do we tap into that? >> Yeah. >> So what I'd like to say is, "Hey, your end user is like your ultimate test group, and unitQ is the layer that converts them into your extended test team." Now, the signals they're producing, it's making it through to the different teams in the organization. >> I think that's the script that you guys are flipping. If I could just interject. Because to me, when I hear you talking, I hear, "Okay, you're letting the customers be an input into the product development process." And there's many different pipelines of that development. And that could be whether you're iterating, or geography, releases, all kinds of different pipelines to get to the market. But in the old days, it was like just customer satisfaction. Complain in a call center. >> Christian: Yeah. >> Or I'm complaining, how do I get support? Nothing made itself into the product improvement, except for slow moving, waterfall-based processes. And then, maybe six months later, a small tweak could be improved. >> Yes. >> Here, you're taking direct input from collective intelligence. Okay. >> Is that have input and on timing is very important here, right? So how do you know if the product is working as it should in all these different flavors and configurations right now? How do you know if it's working well? And how do you know if you're improving or not improving over time? And I think the industry, what can we look at, as far as when it relates to quality? So I can look at star ratings, right? So what's the star rating in the app store? Well, star ratings, that's an average over time. So that's something that you may have a lot of issues in production today, and you're going to get dinged on star ratings over the next few months. And then, it brings down the score. NPS is another one, where we're not going to run NPS surveys every day. We're going to run it once a quarter, maybe once a month, if we're really, really aggressive. That's also a snapshot in time. And we need to have the finger on the pulse of product quality today. I need to know if this release is good or not good. I need to know if anything broke. And I think that real time aspect, what we see as stuff sort of bubbles up the stack, and not into production, we see up to a 50% reduction in time to fix these end user impacting issues. And I think, we also need to appreciate when someone takes time out of the day to write an app review, or email support, or write that Reddit post, it's pretty serious. It's not going to be like, "Oh, I don't like the shade of blue on this button." It's going to be something like, "I got double billed," or "Hey, someone took over my account," or, "I can't reset my password anymore. The CAPTCHA, I'm solving it, but I can't get through to the next phase." And we see a lot of these trajectory impacting bugs and quality issues in these work, these flows in the product that you're not testing every day. So if you work at Snapchat, your employees probably going to use Snapchat every day. Are they going to sign up every day? No. Are they going to do passive reset every day? No. And these things are very hard to instrument, lower in the stack. >> Yeah, I think this is, and again, back to these big problems. It's smoke before fire, and you're essentially seeing it early with your process. Can you give an example of how this new focus or new mindset of user feedback data can help customers increase their experience? Can you give some examples, 'cause folks watching and be like, "Okay, I love this value. Sell me on this idea, I'm sold. Okay, I want to tap into my prospects, and my customers, my end users to help me improve my product." 'Cause again, we can measure everything now with data. >> Yeah. We can measure everything. we can even measure quality these days. So when we started this company, I went out to talk to a bunch of friends, who are entrepreneurs, and VCs, and board members, and I asked them this very simple question. So in your board meetings, or on all hands, how do you talk about quality of the product? Do you have a metric? And everyone said, no. Okay. So are you data driven company? Yes, we're very data driven. >> John: Yeah. Go data driven. >> But you're not really sure if quality, how do you compare against competition? Are you doing as good as them, worse, better? Are you improving over time, and how do you measure it? And they're like, "Well, it's kind of like a blind spot of the company." And then you ask, "Well, do you think quality of experience is important?" And they say, "Yeah." "Well, why?" "Well, top of fund and growth. Higher quality products going to spread faster organically, we're going to make better store ratings. We're going to have the storefronts going to look better." And of course, more importantly, they said the different conversion cycles in the product box itself. That if you have bugs and friction, or an interface that's hard to use, then the inputs, the signups, it's not going to convert as well. So you're going to get dinged on retention, engagement, conversion to paid, and so forth. And that's what we've seen with the companies we work with. It is that poor quality acts as a filter function for the entire business, if you're a product led company. So if you think about product led company, where the product is really the centerpiece. And if it performs really, really well, then it allows you to hire more engineers, you can spend more on marketing. Everything is fed by this product at them in the middle, and then quality can make that thing perform worse or better. And we developed a metric actually called the unitQ Score. So if you go to our website, unitq.com, we have indexed the 5,000 largest apps in the world. And we're able to then, on a daily basis, update the score. Because the score is not something you do once a month or once a quarter. It's something that changes continuously. So now, you can get a score between zero and 100. If you get the score 100, that means that our AI doesn't find any quality issues reported in that data set. And if your score is 90, that means that 10% will be a quality issue. So now you can do a lot of fun stuff. You can start benchmarking against competition. So you can see, "Well, I'm Spotify. How do I rank against Deezer, or SoundCloud, or others in my space?" And what we've seen is that as the score goes up, we see this real big impact on KPI, such as conversion, organic growth, retention, ultimately, revenue, right? And so that was very satisfying for us, when we launched it. quality actually still really, really matters. >> Yeah. >> And I think we all agree at test, but how do we make a science out of it? And that's so what we've done. And when we were very lucky early on to get some incredible brands that we work with. So Pinterest is a big customer of ours. We have Spotify. We just signed new bank, Chime. So like we even signed BetterHelp recently, and the world's largest Bible app. So when you look at the types of businesses that we work with, it's truly a universal, very broad field, where if you have a digital exhaust or feedback, I can guarantee you, there are insights in there that are being neglected. >> John: So Chris, I got to. >> So these manual workflows. Yeah, please go ahead. >> I got to ask you, because this is a really great example of this new shift, right? The new shift of leveraging data, flipping the script. Everything's flipping the script here, right? >> Yeah. >> So you're talking about, what the value proposition is? "Hey, board example's a good one. How do you measure quality? There's no KPI for that." So it's almost category creating in its own way. In that, this net new things, it's okay to be new, it's just new. So the question is, if I'm a customer, I buy it. I can see my product teams engaging with this. I can see how it can changes my marketing, and customer experience teams. How do I operationalize this? Okay. So what do I do? So do I reorganize my marketing team? So take me through the impact to the customer that you're seeing. What are they resonating towards? Obviously, getting that data is key, and that's holy gray, we all know that. But what do I got to do to change my environment? What's my operationalization piece of it? >> Yeah, and that's one of the coolest parts I think, and that is, let's start with your user base. We're not going to ask your users to ask your users to do something differently. They're already producing this data every day. They are tweeting about it. They're putting in app produce. They're emailing support. They're engaging with your support chatbot. They're already doing it. And every day that you're not leveraging that data, the data that was produced today is less valuable tomorrow. And in 30 days, I would argue, it's probably useless. >> John: Unless it's same guy commenting. >> Yeah. (Christian and John laughing) The first, we need to make everyone understand. Well, yeah, the data is there, and we don't need to do anything differently with the end user. And then, what we do is we ask the customer to tell us, "Where should we listen in the public domain? So do you want the Reddit post, the Trustpilot? What channels should we listen to?" And then, our machine basically starts ingesting that data. So we have integration with all these different sites. And then, to get access to private data, it'll be, if you're on Zendesk, you have to issue a Zendesk token, right? So you don't need any engineering hours, except your IT person will have to grant us access to the data source. And then, when we go live. We basically build up this taxonomy with the customers. So we don't we don't want to try and impose our view of the world, of how do you describe the product with these buckets, these quality monitors? So we work with the company to then build out this taxonomy. So it's almost like a bespoke solution that we can bootstrap with previous work we've done, where you don't have these very, very fine buckets of where stuff could go wrong. And then what we do is there are different ways to hook this into the workflow. So one is just to use our products. It's a SaaS product as anything else. So you log in, and you can then get this overview of how is quality trending in different markets, on different platforms, different languages, and what is impacting them? What is driving this unitQ Score that's not good enough? And all of these different signals, we can then hook into Jira for instance. We have a Jira integration. We have a PagerDuty integration. We can wake up engineers if certain things break. We also tag tickets in your support system, which is actually quite cool. Where, let's say, you have 200 people, who wrote into support, saying, "I got double billed on Android." It turns out, there are some bugs that double billed them. Well, now we can tag all of these users in Zendesk, and then the support team can then reach out to that segment of users and say, "Hey, we heard that you had this bug with double billing. We're so sorry. We're working on it." And then when we push fix, we can then email the same group again, and maybe give them a little gift card or something, for the thank you. So you can have, even big companies can have that small company experience. So, so it's groups that use us, like at Pinterest, we have 800 accounts. So it's really through marketing has vested interest because they want to know what is impacting the end user. Because brand and product, the lines are basically gone, right? >> John: Yeah. >> So if the product is not working, then my spend into this machine is going to be less efficient. The reputation of our company is going to be worse. And the challenge for marketers before unitQ was, how do I engage with engineering and product? I'm dealing with anecdotal data, and my own experience of like, "Hey, I've never seen these type of complaints before. I think something is going on." >> John: Yeah. >> And then engineering will be like, "Ah, you know, well, I have 5,000 bugs in Jira. Why does this one matter? When did it start? Is this a growing issue?" >> John: You have to replicate the problem, right? >> Replicate it then. >> And then it goes on and on and on. >> And a lot of times, reproducing bugs, it's really hard because it works on my device. Because you don't sit on that device that it happened on. >> Yup. >> So now, when marketing can come with indisputable data, and say, "Hey, something broke here." And we see the same with support. Product engineering, of course, for them, we talk about, "Hey, listen, you you've invested a lot in observability of your stack, haven't you?" "Yeah, yeah, yeah." "So you have a Datadog in the bottom?" "Absolutely." "And you have an APP D on the client?" "Absolutely." "Well, what about the last mile? How the product manifests itself? Shouldn't you monitor that as well using machines?" They're like, "Yeah, that'd be really cool." (John laughs) And we see this. There's no way to instrument everything, lowering the stack to capture these bugs that leak out. So it resonates really well there. And even for the engineers who's going to fix it. >> Yeah. >> I call it like empathy data. >> Yup. >> Where I get assigned a bug to fix. Well, now, I can read all the feedback. I can actually see, and I can see the feedback coming in. >> Yeah. >> Oh, there's users out there, suffering from this bug. And then when I fix it and I deploy the fix, and I see the trend go down to zero, and then I can celebrate it. So that whole feedback loop is (indistinct). >> And that's real time. It's usually missed too. This is the power of user feedback. You guys got a great product, unitQ. Great to have you on. Founder and CEO, Christian Wiklund. Thanks for coming on and sharing, and showcase. >> Thank you, John. For the last 30 seconds, the minute we have left, put a plug in for the company. What are you guys looking for? Give a quick pitch for the company, real quick, for the folks out there. Looking for more people, funding status, number of employees. Give a quick plug. >> Yes. So we raised our A Round from Google, and then we raised our B from Excel that we closed late last year. So we're not raising money. We are hiring across go-to-markets, engineering. And we love to work with people, who are passionate about quality and data. We're always, of course, looking for customers, who are interested in upping their game. And hey, listen, competing with features is really hard because you can copy features very quickly. Competing with content. Content is commodity. You're going to get the same movies more or less on all these different providers. And competing on price, we're not willing to do. You're going to pay 10 bucks a month for music. So how do you compete today? And if your competitor has a better fine tuned piano than your competitor will have better efficiencies, and they're going to retain customers and users better. And you don't want to lose on quality because it is actually a deterministic and fixable problem. So yeah, come talk to us if you want to up the game there. >> Great stuff. The iteration lean startup model, some say took craft out of building the product. But this is now bringing the craftsmanship into the product cycle, when you can get that data from customers and users. >> Yeah. >> Who are going to be happy that you fixed it, that you're listening. >> Yeah. >> And that the product got better. So it's a flywheel of loyalty, quality, brand, all off you can figure it out. It's the holy grail. >> I think it is. It's a gold mine. And every day you're not leveraging this assets, your use of feedback that's there, is a missed opportunity. >> Christian, thanks so much for coming on. Congratulations to you and your startup. You guys back together. The band is back together, up into the right, doing well. >> Yeah. We we'll check in with you later. Thanks for coming on this showcase. Appreciate it. >> Thank you, John. Appreciate it very much. >> Okay. AWS Startup Showcase. This is season two, episode three, the ongoing series. This one's about MarTech, cloud experiences are scaling. I'm John Furrier, your host. Thanks for watching. (upbeat music)

>> Narrator: TheCUBE presents Kubecon and Cloudnativecon Europe, 2022. Brought to you by Red Hat, the cloud native computing foundation and its ecosystem partners. >> Welcome to Valencia, Spain in Kubecon, Cloudnativecon Europe, 2022. I'm Keith Townsend and we are in a beautiful locale. The city itself is not that big, 100,000, I mean, sorry, about 800,000 people. And we got out, got to see a little bit of the sites. It is an amazing city. I'm from the US, it's hard to put in context how a city of 800,000 people can be so beautiful. I'm here with Anish Dhar and Ganesh Datta, Co-founder and CTO of Cortex. Anish you're CEO of Cortex. We were having a conversation. One of the things that I asked my client is what is good. And you're claiming to answer the question about what is quality when it comes to measuring microservices? What is quality? >> Yeah, I think it really depends on the company and I think that's really the philosophy we have. When we built Cortex, is that we understood that different companies have different definitions of quality, but they need to be able to be represented in really objective ways. I think what ends up happening in most engineering organizations is that quality lives in people's heads. The engineers who write the services they're often the ones who understand all the intricacies with the service. What are the downstream dependencies, who's on call for this service? Where does the documentation live? All of these things I think impact the quality of the service. And as these engineers leave the company or they switch teams, they often take that tribal knowledge with them. And so I think quality really comes down to being able to objectively codify your best practices in some way and have that distributed to all engineers in the company. >> And to add to that, I think very concrete examples for an organization that's already modern like their idea of quality might be uptime incidents. For somebody that's like going through a modernization strategy, they're trying to get to the 21st century, they're trying to get to Kubernetes. For them, quality means where are we in that journey? Are you on our latest platforms? Are you running CI, are you doing continuous delivery? Like quality can mean a lot of things and so our perspective is how do we give you the tools to say as an organization, here's what quality means to us. >> So at first, my mind was going through when you said quality, Anish, you started out the conversation about having this kind of non-codified set of measurements, historical knowledge, et cetera. I was thinking observability, measuring how much time does it take to have a transaction. But Ganesh you're introducing this new thing. I'm working with this project where we're migrating a monolith application to a set of microservices. And you're telling me Cortex helps me measure the quality of what I'm doing in my project? >> Ganesh: Absolutely. >> How is that? >> Yeah, it's a great question. So I think when you think about observability, you think about uptime and latency and transactions and throughput and all this stuff. And I think that's very high level and I think that's one perspective of what quality is, but as you're going through this journey, you might say like the fact that we're tracking that stuff, the fact that you're using APM, you're using distributed tracing, that is one element of service quality. Maybe service quality means you're doing CICD, you're running vulnerability scans. You're using Docker. Like what that means to us can be very different. So observability is just one aspect of are you doing things the right way? Good to us means you're using SLOs. You are tracking those metrics. You're reporting that somewhere. And so that's like one component for our organization of what quality can mean. >> I'm kind of taken back by this because I've not seen someone kind of give the idea. And I think later on, this is the perfect segment to introduce theCUBE clock in which I'm going to give you a minute to kind of like give me the elevator pitch, but we're going to have the deep conversation right now. When you go in and you... What's the first process you do when you engage in a customer? Does a customer go and get this off of repository, install it, the open source version, and then what? I mean, what's the experience? >> Yeah, absolutely. So we have both a smart and on-prem version of Cortex. It's really straightforward. Basically we have a service discovery onboarding flow where customers can connect to different sets of source for their services. It could be Kubernetes, ECS, Git Repos, APM tools, and then we'll actually automatically map all of that service data with all of the integration data in the company. So we'll take that service and map it to its on call rotation to the JIRA tickets that have the service tag associated with it, to the data algo SLOs. And what that ends ends up producing is this service catalog that has all the information you need to understand your service. Almost like a single pane of glass to work with the service. And then once you have all of that data inside Cortex, then you can start writing scorecards, which grade the quality of those services across those different verticals Ganesh was talking about. Like whether it's a monolith, a microservice transition, whether it's production readiness or security standards, you can really start tracking that. And then engineers start understanding where the areas of risk with my service across reliability or security or operation maturity. I think it gives us in insane visibility into what's actually being built and the quality of that compared to your standards. >> So, okay, I have a standards for SLO that is usually something that is, it might not even be measured. So how do you help me understand that I'm lacking a measurable system for tracking SLO and what's the next step for helping me get that system? >> Yeah, I think our perspective is very much how do we help you create a culture where developers understand what's expected of them? So if SLOs are part of what we consider observability or reliability, then Cortex's perspective is, hey, we want to help your organization adopt SLOs. And so that service cataloging concept, the service catalog says, hey, here's my API integration. Then a scorecard, the organization goes in and says, we want every service owner to define their SLOs, we want you to define your thresholds. We want you to be tracking them, are you passing your SLOs? And so we're not being prescriptive about here's what we think your SLOs should be, ours is more around, hey, we're going to help you like if you care about SLOs, we're going to tell the service owners saying, hey, you need to have at least two SLOs for your service and you got to be tracking them. And the service catalog that data flows from a service catalog into those scorecards. And so we're helping them adopt that mindset of, hey, SLOs are important. It is a component of like a holistic service reliability excellence metric that we care about. >> So what happens when I already have systems for like SLO, how do I integrate that system with Cortex? >> That's one of the coolest things. So the service catalog can be pretty smart about it. So let's say you've sucked in your services from your GitHub. And so now your services are in Cortex. What we can do is we can actually discover from your APM tools, you can say like, hey, for this service, we have guessed that this is the corresponding APM in Datadog. And so from Datadog, here are your SLOs, here are your monitors. And so we can start mapping all the different parts of your world into the Cortex. And that's the power of the service catalog. The service catalog says, given a service, here's everything about that service. Here's the vulnerability scans. Here's the APM, the monitors, the SLOs, the JIRA ticket is like all that stuff comes into a single place. And then our scorecards product can go back out and say, hey, Datadog, tell me about this SLOs for the service. And so we're going to get that information live and then score your services against that. And so we're like integrating with all of your third party tools and integrations to create that single pan of glass. >> Yeah, and to add to that, I think one of the most interesting use cases with scorecards is, okay, which teams have actually adopted SLOs in the first place? I think a lot of companies struggle with how do we make sure engineers defined SLOs are passing them actually care about them. And scorecards can be used to one, which teams are actually meeting these guidelines? And then two, let's get those teams adopted on SLOs. Let's track that, you can do all of that in Cortex, which is I think a really interesting use case that we've seen. >> So let's talk about kind of my use case in the end to end process for integrating Cortex into migrations. So I have this monolithic application, I want to break it into microservices and then I want to ensure that I'm delivering if not, you know what, let's leave it a little bit more open ended. How do I know that I'm better at the end of I was in a monolith before, how do I measure that now that I'm in microservices and on cloud native, that I'm better? >> That's a good question. I think it comes down to, and we talk about this all the time for our customers that are going through that process. You can't define better if you don't define a baseline, like what does good mean to us? And so you need to start by saying, why are we moving to microservices? Is it because we want teams to move faster? Is it because we care about reliability up time? Like what is the core metric that we're tracking? And so you start by defining that as an organization. And that is kind of like a hand wavy thing. Why are we doing microservices? Once you have that, then you define this scorecard. And that's like our golden path. Once we're done doing this microservice migration, can we say like, yes, we have been successful and those metrics that we care about are being tracked. And so where Cortex fits in is from the very first step of creating a service, you can use Cortex to define templates. Like one click, you go in, it spins up a microservice for you that follows all your best practices. And so from there, ideally you're meeting 80% of your standards already. And then you can use scorecards to track historical progress. So you can say, are we meeting our golden path standards? Like if it's uptime, you can track uptime metrics and scorecards. If it's around velocity, you can track velocity metrics. Is it just around modernization? Are you doing CICD and vulnerability scans, like moving faster as a team? You can track that. And so you can start seeing like trends at a per team level, at a per department level, at a per product level saying, hey, we are seeing consistent progress in the metrics that we care about. And this microservice journey is helping us with that. So I think that's the kind of phased progress that we see with Cortex. >> So I'm going to give you kind of a hand wavy thing. We're told that cloud native helps me to do things faster with less defects so that I can do new opportunities. Let's stretch into kind of this non-tech, this new opportunities perspective. I want to be able to move my microservices. I want to be able to move my architecture to microservices, so I reduce call wait time on my customer service calls. So I can easily see how I can measure are we iterating faster? Are we putting out more updates quicker? That's pretty easy to measure. The number of defects, easy to measure. I can imagine a scorecard, but what about this wait time? I don't necessarily manage the call center system, but I get the data. How do I measure that the microservice migration was successful from a business process perspective? >> Yeah, that's a good question. I think it comes down to two things. One, the flexibility of scorecard means you can pipe in that data to Cortex. And what we recommend customers is track the outcome metrics and track the input metrics as well. And so what is the input metric to call wait time? Like maybe it's the fact that if something goes wrong, we have the run books to quickly roll back to an older version that we know is running. That way MTTR is faster. Or when something happens, we know the owner for that service and we can go back to them and say like, hey, we're going to ping you as an incident commander. Those are kind of the input metrics to, if we do these things, then we know our call wait time is going to drop because we're able to respond faster to incidents. And so you want to track those input metrics. And then you want to track the output metrics as well. And so if you have those metrics coming in from your Prometheus or your Datadogs or whatever, you can pipe that into Cortex and say, hey, we're going to look at both of these things holistically. So we want to see is there a correlation between those input metrics like are we doing things the right way, versus are we seeing the value that we want to come out of that? And so I think that's the value of Cortex is not so much around, hey, we're going to be prescriptive about it. It's here's this framework that will let you track all of that and say, are we doing things the right way and is it giving us the value that we want? And being able to report that update to engineer leadership and say, hey, maybe these services are not doing like we're not improving call wait time. Okay, why is that? Are these services behind on the actual input metrics that we care about? And so being able to see that I think is super valuable. >> Yeah, absolutely, I think just to touch on the reporting, I think that's one of the most value add things Cortex can provide. If you think about it, the service is atomic unit of your software. It represents everything that's being built and that bubbles up into teams, products, business units, and Cortex lets you represent that. So now I can, as a CTO, come in and say, hey, these product lines are they actually meeting our standards? Where are the areas of risk? Where should I be investing more resources? I think Cortex is almost like the best way to get the actual health of your engineering organization. >> All right Anish and Ganesh. We're going to go into the speed round here. >> Ganesh: It's time for the Q clock? >> Time for the Q clock. Start the Q clock. (upbeat music) Let's go on. >> Ganesh: Let's do it. >> Anish: Let's do it. >> Let's go on. You're you're 10 seconds in. >> Oh, we can start talking. Okay, well I would say, Anish was just touching on this. For a CTO, their question is how do I know if engineering quality is good? And they don't care about the microservice level. They care about as a business, is my engineering team actually producing. >> Keith: Follow the green, not the dream. (Ganesh laughs) >> And so the question is, well, how do we codify service quality? We don't want this to be a hand wavy thing that says like, oh, my team is good, my team is bad. We want to come in and define here's what service quality means. And we want that to be a number. You want that to be something that can- >> A goal without a timeline is just a dream. >> And CTO comes in and they say, here's what we care about. Here's how we're tracking it. Here are the teams that are doing well. We're going to reward the winners. We're going to move towards a world where every single team is doing service quality. And that's where Cortex can provide. We can give you that visibility that you never have before. >> For that five seconds. >> And hey, your SRE can't be the one handling all this. So let Cortex- >> Shoot the bad guy. >> Shot that, we're done. From Valencia Spain, I'm Keith Townsend. And you're watching theCube. The leader in high tech coverage. (soft music) (soft music) >> Narrator: TheCube presents Kubecon and Cloudnativecon Europe, 2022 brought to you by Red Hat, the cloud native computing foundation and its ecosystem partners. >> Welcome to Valencia, Spain in Kubecon, Cloudnativecon Europe, 2022. I'm Keith Townsend. And we are in a beautiful locale. The city itself is not that big 100,000, I mean, sorry, about 800,000 people. And we got out, got to see a little bit of the sites. It is an amazing city. I'm from the US, it's hard to put in context how a city of 800,000 people can be so beautiful. I'm here with Anish Dhar and Ganesh Datta, Co-founder and CTO of Cortex. Anish you're CEO of Cortex. We were having a conversation. One of the things that I asked my client is what is good. And you're claiming to answer the question about what is quality when it comes to measuring microservices? What is quality? >> Yeah, I think it really depends on the company. And I think that's really the philosophy we have when we build Cortex is that we understood that different companies have different definitions of quality, but they need to be able to be represented in really objective ways. I think what ends up happening in most engineering organizations is that quality lives in people's heads. Engineers who write the services, they're often the ones who understand all the intricacies with the service. What are the downstream I dependencies, who's on call for this service, where does the documentation live? All of these things, I think impact the quality of the service. And as these engineers leave the company or they switch teams, they often take that tribal knowledge with them. And so I think quality really comes down to being able to objectively like codify your best practices in some way, and have that distributed to all engineers in the company. >> And to add to that, I think like very concrete examples for an organization that's already modern their idea of quality might be uptime incidents. For somebody that's like going through a modernization strategy, they're trying to get to the 21st century. They're trying to get to Kubernetes. For them quality means like, where are we in that journey? Are you on our latest platforms? Are you running CI? Are you doing continuous delivery? Like quality can mean a lot of things. And so our perspective is how do we give you the tools to say as an organization here's what quality means to us. >> So at first my mind was going through when you said quality and as you started out the conversation about having this kind of non codified set of measurements, historical knowledge, et cetera. I was thinking observability measuring how much time does it take to have a transaction? But Ganesh you're introducing this new thing. I'm working with this project where we're migrating a monolith application to a set of microservices. And you're telling me Cortex helps me measure the quality of what I'm doing in my project? >> Ganesh: Absolutely. >> How is that? >> Yeah, it's a great question. So I think when you think about observability, you think about uptime and latency and transactions and throughput and all this stuff and I think that's very high level. And I think that's one perspective of what quality is. But as you're going through this journey, you might say like the fact that we're tracking that stuff, the fact that you're using APM, you're using distributed tracing, that is one element of service quality. Maybe service quality means you're doing CICD, you're running vulnerability scans. You're using Docker. Like what that means to us can be very different. So observability is just one aspect of, are you doing things the right way? Good to us means you're using SLOs. You are tracking those metrics. You're reporting that somewhere. And so that's like one component for our organization of what quality can mean. >> Wow, I'm kind of taken me back by this because I've not seen someone kind of give the idea. And I think later on, this is the perfect segment to introduce theCube clock in which I'm going to give you a minute to kind of like give me the elevator pitch, but we're going to have the deep conversation right now. When you go in and you... what's the first process you do when you engage in a customer? Does a customer go and get this off of repository, install it, the open source version and then what, I mean, what's the experience? >> Yeah, absolutely. So we have both a smart and on-prem version of Cortex. It's really straightforward. Basically we have a service discovery onboarding flow where customers can connect to different set of source for their services. It could be Kubernetes, ECS, Git Repos, APM tools, and then we'll actually automatically map all of that service data with all of the integration data in the company. So we'll take that service and map it to its on call rotation to the JIRA tickets that have the service tag associated with it, to the data algo SLOs. And what that ends up producing is this service catalog that has all the information you need to understand your service. Almost like a single pane of glass to work with the service. And then once you have all of that data inside Cortex, then you can start writing scorecards, which grade the quality of those services across those different verticals Ganesh was talking about. like whether it's a monolith, a microservice transition, whether it's production readiness or security standards, you can really start tracking that. And then engineers start understanding where are the areas of risk with my service across reliability or security or operation maturity. I think it gives us insane visibility into what's actually being built and the quality of that compared to your standards. >> So, okay, I have a standard for SLO. That is usually something that is, it might not even be measured. So how do you help me understand that I'm lacking a measurable system for tracking SLO and what's the next step for helping me get that system? >> Yeah, I think our perspective is very much how do we help you create a culture where developers understand what's expected of them? So if SLOs are part of what we consider observability and reliability, then Cortex's perspective is, hey, we want to help your organization adopt SLOs. And so that service cataloging concept, the service catalog says, hey, here's my APM integration. Then a scorecard, the organization goes in and says, we want every service owner to define their SLOs. We want to define your thresholds. We want you to be tracking them. Are you passing your SLOs? And so we're not being prescriptive about here's what we think your SLOs should be. Ours is more around, hey, we're going to help you like if you care about SLOs, we're going to tell the service owners saying, hey, you need to have at least two SLOs for your service and you've got to be tracking them. And the service catalog that data flows from the service catalog into those scorecards. And so we're helping them adopt that mindset of, hey, SLOs are important. It is a component of like a holistic service reliability excellence metric that we care about. >> So what happens when I already have systems for like SLO, how do I integrate that system with Cortex? >> That's one of the coolest things. So the service catalog can be pretty smart about it. So let's say you've sucked in your services from your GitHub. And so now your services are in Cortex. What we can do is we can actually discover from your APM tools, we can say like, hey, for this service we have guessed that this is the corresponding APM in Datadog. And so from Datadog, here are your SLOs, here are your monitors. And so we can start mapping all the different parts of your world into the Cortex. And that's the power of the service catalog. The service catalog says, given a service, here's everything about that service. Here's the vulnerability scans, here's the APM, the monitor, the SLOs, the JIRA ticket, like all that stuff comes into a single place. And then our scorecard product can go back out and say, hey, Datadog, tell me about this SLOs for the service. And so we're going to get that information live and then score your services against that. And so we're like integrating with all of your third party tools and integrations to create that single pan of glass. >> Yeah and to add to that, I think one of the most interesting use cases with scorecards is, okay, which teams have actually adopted SLOs in the first place? I think a lot of companies struggle with how do we make sure engineers defined SLOs are passing them actually care about them? And scorecards can be used to one, which teams are actually meeting these guidelines? And then two let's get those teams adopted on SLOs. Let's track that. You can do all of that in Cortex, which is, I think a really interesting use case that we've seen. >> So let's talk about kind of my use case in the end to end process for integrating Cortex into migrations. So I have this monolithic application, I want to break it into microservices and then I want to ensure that I'm delivering you know what, let's leave it a little bit more open ended. How do I know that I'm better at the end of I was in a monolith before, how do I measure that now that I'm in microservices and on cloud native, that I'm better? >> That's a good question. I think it comes down to, and we talk about this all the time for our customers that are going through that process. You can't define better if you don't define a baseline, like what does good mean to us? And so you need to start by saying, why are we moving to microservices? Is it because we want teams to move faster? Is it because we care about reliability up time? Like what is the core metric that we're tracking? And so you start by defining that as an organization. And that is kind of like a hand wavy thing. Why are we doing microservices? Once you have that, then you define the scorecard and that's like our golden path. Once we're done doing this microservice migration, can we say like, yes, we have been successful. And like those metrics that we care about are being tracked. And so where Cortex fits in is from the very first step of creating a service. You can use Cortex to define templates. Like one click, you go in, it spins up a microservice for you that follows all your best practices. And so from there, ideally you're meeting 80% of your standards already. And then you can use scorecards to track historical progress. So you can say, are we meeting our golden path standards? Like if it's uptime, you can track uptime metrics and scorecards. If it's around velocity, you can track velocity metrics. Is it just around modernization? Are you doing CICD and vulnerability scans, like moving faster as a team? You can track that. And so you can start seeing like trends at a per team level, at a per department level, at a per product level. Saying, hey, we are seeing consistent progress in the metrics that we care about. And this microservice journey is helping us with that. So I think that's the kind of phased progress that we see with Cortex. >> So I'm going to give you kind of a hand wavy thing. We're told that cloud native helps me to do things faster with less defects so that I can do new opportunities. Let's stretch into kind of this non-tech, this new opportunities perspective. I want to be able to move my microservices. I want to be able to move my architecture to microservices so I reduce call wait time on my customer service calls. So, I could easily see how I can measure are we iterating faster? Are we putting out more updates quicker? That's pretty easy to measure. The number of defects, easy to measure. I can imagine a scorecard. But what about this wait time? I don't necessarily manage the call center system, but I get the data. How do I measure that the microservice migration was successful from a business process perspective? >> Yeah, that's a good question. I think it comes down to two things. One, the flexibility of scorecard means you can pipe in that data to Cortex. And what we recommend customers is track the outcome metrics and track the input metrics as well. And so what is the input metric to call wait time? Like maybe it's the fact that if something goes wrong, we have the run book to quickly roll back to an older version that we know is running that way MTTR is faster. Or when something happens, we know the owner for that service and we can go back to them and say like, hey, we're going to ping you as an incident commander. Those are kind the input metrics to, if we do these things, then we know our call wait time is going to drop because we're able to respond faster to incidents. And so you want to track those input metrics and then you want to track the output metrics as well. And so if you have those metrics coming in from your Prometheus or your Datadogs or whatever, you can pipe that into Cortex and say, hey, we're going to look at both of these things holistically. So we want to see is there a correlation between those input metrics? Are we doing things the right way versus are we seeing the value that we want to come out of that? And so I think that's the value of Cortex is not so much around, hey, we're going to be prescriptive about it. It's here's this framework that will let you track all of that and say, are we doing things the right way and is it giving us the value that we want? And being able to report that update to engineer leadership and say, hey, maybe these services are not doing like we're not improving call wait time. Okay, why is that? Are these services behind on like the actual input metrics that we care about? And so being able to see that I think is super valuable. >> Yeah, absolutely. I think just to touch on the reporting, I think that's one of the most value add things Cortex can provide. If you think about it, the service is atomic unit of your software. It represents everything that's being built and that bubbles up into teams, products, business units, and Cortex lets you represent that. So now I can, as a CTO, come in and say, hey, these product lines are they actually meeting our standards? Where are the areas of risk? Where should I be investing more resources? I think Cortex is almost like the best way to get the actual health of your engineering organization. >> All right, Anish and Ganesh. We're going to go into the speed round here. >> Ganesh: It's time for the Q clock >> Time for the Q clock. Start the Q clock. (upbeat music) >> Let's go on. >> Ganesh: Let's do it. >> Anish: Let's do it. >> Let's go on, you're 10 seconds in. >> Oh, we can start talking. Okay, well I would say, Anish was just touching on this, for a CTO, their question is how do I know if engineering quality is good? And they don't care about the microservice level. They care about as a business, is my enduring team actually producing- >> Keith: Follow the green, not the dream. (Ganesh laughs) >> And so the question is, well, how do we codify service quality? We don't want this to be a hand wavy thing that says like, oh, my team is good, my team is bad. We want to come in and define here's what service quality means. And we want that to be a number. You want that to be something that you can- >> A goal without a timeline is just a dream. >> And a CTO comes in and they say, here's what we care about, here's how we're tracking it. Here are the teams that are doing well. We're going to reward the winners. We're going to move towards a world where every single team is doing service quality. And that's what Cortex can provide. We can give you that visibility that you never had before. >> For that five seconds. >> And hey, your SRE can't be the one handling all this. So let Cortex- >> Shoot the bad guy. >> Shot that, we're done. From Valencia Spain, I'm Keith Townsend. And you're watching theCube, the leader in high tech coverage. (soft music)

(bright music) >> Hi, everyone. Welcome to the CUBE's presentation of the AWS Startup Showcase around open cloud innovations. It's the season two episode one of the ongoing series covering exciting startups from the AWS ecosystem and talking about open source and innovation. I'm John Furrier, your host. Today, we're joined by two great guests. Dan Garfield, chief open source officer and co-founder of Codefresh IO, and Raziel Tabib, CEO and co-founder. Two co-founders in the middle of all the innovation. Gentlemen thanks for coming on. >> Thank you. >> So you guys have a great platform and as cloud native goes mainstream in the enterprise and for developers, the big topic is unification, end-to-end, horizontally scalable, leveraging data. All these things around agile that I call agile cloud next level. This is kind of what we're seeing. The CNCF is growing. You've seen KubeCon every year is more about these kinds of things. Words like orchestration, Kubernetes, container, security. All of those complexities are now at the center of making things easier for developers. This is a key value proposition and you guys at Codefresh are offering really the first enterprise delivery solution powered by Argo, which is an open source project. Again, open source driving really big changes. So let's get into it. And first of all, congratulations, and thanks for working on this project. What's so special about- >> Thank you for that. >> Argo the project, and why have you guys decided to build a platform on it, and where is this coming together? Take us through why this is so important. >> I think Argo has been a very fast growing open source project for multiple reasons. A, it has been built for the new way of building and deploying an application. It's cloud native. You mentioned Kubernetes becoming kind of the de facto way of running application. It's the de facto way to run automation and pipeline. But also Argo has been built from the ground up to the latest practices of how we deploy software. We deploy software now differently. We deploy it using a GitOps practice. We're deploying it using canary blue-green progressive deployment. And Argo has been built around these practices, around these technologies, and has been very much widely adopted by the community. In the past, the KubeCon you've mentioned, Argo was all over the place. And we were very glad to be working with the community to talk about what the next steps with Argo. >> Yeah, it's a really good point. I would like to just follow up on that because you see this being talked about. It always comes up, where is open source really outside of a pure contributors matter? And when you have corporations contributing, you seeing this has been the trend. You saw it with Lyft, with Envoy, companies doing more and more open source. This is part of a big collaboration. And again, this comes back down to this whole why it's relevant and why it's so special with Argo. Continue to talk about relationship because it's not just you guys, it's now community. >> Yeah, I can speak to that. The Argo project is something that we maintain in partnership with several other companies and really our relationship with it is that this is something that we're actively contributing to. This is something that we're helping build the roadmap on and planning the events around and all those kinds of things. And we're doing that because we really believe in this technology and we've built our platform on it. So when you deploy Codefresh, you're deploying technology that's built directly on Argo and is designed specifically to solve that problem that you spoke to at the top of the hour. We all want to deliver software faster. We all want to have fewer regressions. We want to have fewer breaking changes. We want software to be super reliable. We want to be comfortable with what we're doing. That's really why we picked Argo because that technology that we have it is to Raziel's point delivered in this new way. It's delivered using GitOps. And that's a whole revolution and change in the way that people build and deploy software. And bringing cohesion into that experience is so critical to building the confidence that lets you actually deploy often and frequently and more. >> Dan, if you don't mind just expanding on that one point about the problem you solve, because to me, this has been kind of that evolution. It's almost like, yeah, there's been problems, plural, and opportunities that you saw with those in growing markets like this with DevOps and DevSecOps and now cloud native. What is the catalyst behind all of this? What was the epiphany behind it? How did it get so much momentum? What was it really doing under the covers? >> Well, it's a very simple and easy to use set of tools. And that's one of the big things is that if you look at the ideas of GitOps and there's actually a foundation around this that were part of called open GitOps to GitOps working group under the CNCF. And those principles of, I want to, yes, do my software defined as code. I want to do my infrastructure defined as code and I need something monitoring by production run times and making sure that the declared desired state is always matching the actual state. Those principles have actually been around for a number of years. And with Kubernetes, we really unlocked an API that allowed us to start doing GitOps and this is why we bring in Argo and you see the rise of Argo CD and other workflows and what we've been doing is really because that technology has been unlocked now. So the ability to define how your software is supposed to run and now your entire software delivery stack should run, all defined and then monitored and then kept in check using the GitOps operator. That critical unlock is what's really driving the massive adoption. And like Raziel said, Argo is the fastest growing and most popular open source project for delivering software. And it's not even close. >> Yeah, this is really great point. And I want to get into that 'cause I want to know why, what you guys do on your platform versus the open source and get that relationship settled? Before we get there, though, I want to get your reaction to some of the commentary in the industry 'cause GitOps trend has been exploding into new directions. I mean, it used to be a term about 10 years ago called big data. And at the beginning where data was all big data. Now it was DevOps revolution around data as well. But now you're hearing people talk about big code. Like, I mean, the code bases are becoming so huge. So as a developer, you're leveraging large open source code. This idea of the software delivery with existing code and new code just adds to more code. There's more code being developed every day. >> There is more code delivered every day. And I think that organization realize today, almost in every industry that they have to pace up how fast and how frequent they update their software delivery. We're living in a world in which every aspect of our life has been disrupted by software and organization realize that they have to keep up and figure out how to deploy software more frequent and more lively. And I think, you mentioned that really Kubernetes, the cloud native became the de facto way of running application. I think most of organization has made that decision to move into cloud native. The second question is after, is okay, now we have all applications running, how fast and how more frequent we can deploy applications to the cloud native? And that's the stage in which we're super excited about Argo and our up platform because that's basically streamline the building application for these cloud native, deploying applications for the cloud native, and so on. >> Yeah, and I think that highlights the business value. You getting a lot of the conversations with businesses that say they want the modern application on the cloud scale. And at the end of the day, it comes down to speed and security. So how fast can I get the app out? How well does it work? Does it run performance? And does it have security? And I don't want a slow. >> Exactly. Exactly. It kind of oversimplifies it, but that's kind of the net net. So when you look at Argo open source, what's that's done and kind of where you guys are taking it. Can you talk about the differences between your enterprise version and the open source version and the interplay there, the relationship, the business model health customers can play on both sides or understand the difference? >> Sure. >> Go ahead. >> Go ahead, Raziel. Okay, so I think Argo, as you mentioned, is probably the most advanced technology today to both run pipelines. They're like events to trigger pipelines and Argo work for the one that pipelines, the Argo CD for GitOps and Rollout, for Canary blue-green strategies. And the adoption is really exploding. Just as an Advocate that we had in December, we have worked with the community and organized ArgoCon events in which we had initially kind of thought about 500 attendees. And so we have more than 4,000 registrants and majority of them are coming from enterprise. Now as we have talked to the community during this conference and figure out, okay, so what are the things that you're still missing? And that will help you take the benefit that you get from Argo to the next level. The few things that came up. One is Argo is a great technology. However, Argo now is fragmented into four projects. There is an advance. There is workflow. There is Argo CD. And there is Argo Rollout. And there is a need to bring them all together into a solid platform, solid one run time that can be easily installed, monitor all of these in a single UI, in a single control plane. That's one aspect. The second is the scalability. Really being able to manage it centrally across multiple clusters, not in one cluster. And what we bring in with the new one, we're so excited about this platform, is we're bringing that big. The first to get all of these four projects in one runtime, and one control plane, but also allow the community to run it across multiple cluster from one place getting into the solution, not just as a technology. >> If I may add to that, the value of bringing these projects together, it provides so many insights. So when you're trying to figure out, there's some breaking change that has been made, but you don't necessarily know where it is because you have a lot of microservices that are out there. You have a lot of teams working on it. By bringing all of these things together, we're able to look at all of the commits, all of the deployments, all of the Jira issues. All of these components combined together, so you really get a single view where you can see everything that's going on. And this is another element where when you're trying to deploy software at scale, you're trying to deliver it faster. People are getting a little bit overwhelmed because there are so many updates and so many different services and so many teams working that they're starting to miss that visibility. So this is what we want to bring into the ecosystem is we really want them that visibility to be super clear. And by bringing all of the Argo components, the Argo tools together, we're able to do that in a single dashboard. >> Yeah, so if I get this right, let me just double click on that because it sounds like, yeah, Argo's great. It's been organically growing, a lot of different components to it, but when you start getting into pushing code in an organization, you have, I call the old-school version control kind of vibe going on where it's like you don't know what's out there and how that affects the system as it's a distributed system, which cloud is. There are consequences when stuff breaks. So we all know that. Is that kind of where you guys are getting at? The challenge is actually the opportunity at the same time where it's all goodness, but then when you start looking at scale and the system impact, is that kind of where the open source and you guys pick up, is that right? >> This is one aspect. I think the second one is that again, when you look at each individual component of Argo, each provide a lot of value by itself. But when you sum it, the value of the sum is greater than the value of the individual. So when you're taking, really the events and workflow, Argo CD and Argo Rollout, and you bring them all together into single runtime. The value of its time is really automation all the way from code to cloud. It's not breaking into, there is like an automation for CI, there's an automation for CD, there's information for progressive delivery. It's actually automated all the way from the Git commit through the GitOps through the deployment strategy, and so on. And being able to monitor it and scale it in the enterprise scale. So, of course, it's helping enterprise and make Argo to some level more crucial for enterprise, if I may say, but second is really bringing all of these components together and get the outcome be greater than the individual parts. >> Yeah, that's a good point. Yeah, make it make a commercial grade, if you will, for enterprise who wants to have support and consistency and whatnot. What other problems are you solving? Dan, can you chime in on the whole, how you guys resolve some of these challenges for the enterprise? Because, again, some stability is key as well, but also the business benefit has got to be there for the development teams. >> Yeah. So there's several. One aspect is that the way that most people operate today is they essentially do a bunch of commands and engage with systems. And then hopefully at the end, they write those things to Git. And this is a little bit backwards if you think about it because there's a situation where you can end up with things in production that were never checked in, or maybe somebody is operating and they're making a change. If we look at most of the downtime that's occurred over the last two years, it's because people have flubbed a key when they were typing in a command or something like that. The way that this system works is that we provide an interface, both the CLI and the GUI, where those operations interactions actually end with a Git commit. So rather than doing an operation and then hopefully committing to Git, most of the operations are actually done first in Git, or if there is something that can't be done first in Git, it's maybe bootstrapped and then committed to Git as part of a single command. So this means you have end-to-end traceability. It also means your auditability is way better. And then the second, the other component that we're adding is that security and scale layer. So we are securing these things, we're building in single sign-on, and all those robust security things you would expect to have across all these instances. So many organizations, when they're building their software delivery tools, they have to deploy instances in many locations. And so this is how you end up with companies that have 5,000 instances that are all out of date and insecure. Well with Codefresh, if you need to deploy a component onto this end cluster or something like that, you may have thousands of them. All of those are monitored and taken care of in a centralized way, so I can do all of my updates at once. I can make sure they're all up to date. I'm not running with a bunch of known CVEs or something like that and it's clear. The components are also designed in an architectural way. So that only the information that is needed is ever passed out. So I can have a cluster that is remotely managed, that checks out code, that the control plane never has access to. So this hybrid model has been really popular with our customers. We have customers in healthcare, we have customers in defense and in financial services, all these regulated industries. The flow of information is really critical. So this hybrid model allows you to deploy something that has the ease of a SaaS solution, but has the security of an on-prem solution while being centrally managed and easy to take care of. >> Yeah, it's a platform. It's what it is. It's not a tool. It's not a tool anymore. It's a platform. >> Exactly. >> I think the foundational aspect of this is critical. And you mentioned automation before. If you're going to go end-to-end automation, you have some stuff in the system that whether it hasn't been checked in yet. I mean, we know what this leads to. Disaster or a lot of troubleshooting and disruption. That's what it seems to solve. Am I getting that right? Is that right? >> Yeah. >> Go ahead. >> Yeah, it helps automate the whole process. But as you say, it's really like identify what needs not to be going all the way to production and really kind of avoid vulnerabilities or any flaws in the software. So it automates everything, but in a way that the automation can identify issues and avoid them from coming into the production. >> Well, great stuff here. I've got to ask you guys now that you've got that settled. It's really, I see the value there, how you guys are letting it grow organically and with Argo and then building that platform for businesses and developers. It's really cool. And I see the foundational value there. It just only gets better. How you guys contributing back to open source and helping the wider GitOps and Argo communities? Because this is, again, the rising tide that's bringing all the boats into the harbor, so to speak. So this is a good trend and people will acknowledge that. So how's this going to work as you guys work back into the open source community? >> So we work closely with both myself and the other maintainers worked closely with the community on the roadmap and making sure that we're addressing issues. I think if you look in the last quarter, we probably have upwards of 40 or 50 different issues that we've solved in terms of fixing a bug or adding features or things like that. So making sure that these tools, which are really the undergirding components of our platform, they have to be really robust. They have to be really strong. And so we're contributing those things back. And then when it comes to the scalability side, these are things that we can build into the platform. So the value should be really clear. I can deploy this, I can manage it myself, I can build tools on top of it. And if I want to start doing it at scale, maybe I want support. That's when I really am going to go to Codefresh and start saying, let's get the enterprise little platform. >> Awesome. GitOps, a lot of people like some naysayers may say, Hey, it's the latest fad. Is it here to stay? We were talking about big code earlier. GitOps, obviously seeing open source. Just every year, just get better and better and growth. I mean, I remember when I was breaking into the business, you have to sell under the table. Now it's all free and open and getting better every year. Just the growth of code. Is GitOps a fad? How do you talk to people who say that? I mean, besides slapping around saying wake up. I mean, how do you guys address that when people say it's just the latest fad? >> So if I may comment here and Dan feel free to chime in, I think that the GitOps is a continuation of a trend that everything is a source code. As a developer, many years ago myself and still writing code, always both code and code was the source of tool that's where we write the code. But now code actually is also describing how our application is running in production. And we've already seen kind of where it's get next. We also hear about infrastructure as a code. So now actually we storing the code the way the infrastructure should be. And I think that the benefit of storing all this configuration in a source code, which has been built to track changes, to be enabled to roll back, that is just going to be here to stay. And I think that's the new way of doing things. >> All right, gentlemen, great. Closing statements. Please share an update on the company. What it's all about? What event you got coming? I know you got a big launch. Can you take us through? Take us home. >> Join on February 1st, we're going to be launching the Codefresh software delivery platform. Raziel and I will be hosting the event. We've got a number of customers, a number of members of the community who are going to be joining us to show off that platform. So you're going to be able to see it in action, see how the features work, and understand the value of it. And you'll see how it works with GitOps. You'll see how it helps you deliver software at scale. That's February 1st. You can get information at codefresh.io. >> Raziel, Dan, thanks for coming on. >> Thank you. >> Pretty good showcase. Thanks for sharing. Congratulations. Great venture. Loved the approach. Love the growth in cloud native and you guys sure on the cutting edge. Fresh code, people love fresh code, codefresh.io. Thanks for coming on. >> Thank you. Thank you. >> Okay, this is the AWS Startup Showcase Open Cloud Innovations. Cloud scale, software, data. That's the future of modern applications being developed, changing the game to the next level. This is the CUBE's coverage season two episode one of the ongoing AWS Startup series here in theCUBE.

(upbeat music) >> Hello, welcome to this CUBE conversation. I'm John Furrier your host of theCUBE here in our Palo Alto studio. We got a great conversation with the CEO of Lacework, David Hatfield. Who's in on theCUBE remote. David great to see you guys, a security platform at Lacework, you're at the helm as CEO. Welcome to theCUBE conversation. >> Thank you, John. Great to see you congrats to you and the team and all the success. I think what you guys are doing is really important so happy to be part of it. >> Great to have you in the community and you guys are doing great work. I know about Lacework I've done some due diligence on you guys. I love your business model, but for the folks who don't know what you guys do, take a minute to explain who is Lacework? What do you guys do? What's your positioning? And what's your focus? >> Yeah, well, we're a modern data security platform for the cloud. And so I think data science meets cloud security ultimately. The company has been around since 2015. We received one of the largest financing rounds that we're aware of I think in history in security business, $525 million in January. Led by Sutter Hill Ventures which many people may know about they founded PureStorage with the notion that we're going to go fundamentally change and revamp the ownership model for a high speed data storage using flash versus using spinning disc drives. I spent eight years with that company. Love with what we built there. Then Mike Speiser considered an investment in a company called Snowflake computing. I think you're aware of what Snowflake does which is bringing data warehousing into the cloud. And the third big investment that Sutter Hill made is really to help disrupt security, and that's in Lacework. So north of a billion dollar valuation a 300% year over year growth and have a ton of momentum. So at the core of what we do, it's really trying to merge, when we look at we look at security as a data problem, security and compliance the data problem. And when you apply that to the cloud, it's a massive data problem. you literally have trillions of data points across shared infrastructure that we need to be able to ingest and capture and then you need to be able to process efficiently and provide context back to the end-user. And so we approached it very differently than how legacy approaches have been in place, you know largely rules-based engines that are written to be able to try and stop the bad guys. And they miss a lot of things. And so our data-driven approach that we patented is called a polygraph. It's a, it's a security architecture and there are three primary benefits. It does a lot of things, but the three things that we think are most profound first is it eliminates the need for, you know dozens of point solutions. I was shocked when I, you know kind of learned about security. I was at Symantec back in the day. And just to see how fragmented this market is, it's one of the biggest markets in tech. $124 billion in annual spend growing at, to $300 billion in the next three years. And it's massively fragmented. And the average number of point solutions that customers have to deal with is dozens. Like literally 75 is the average number. And so we wanted to take a platform approach to solve this problem where the larger the attack surface that you put in the more data that you put into our machine learning algorithms the smarter it gets and the higher, the efficacy. So eliminating point solutions is his value proposition one. Point two is that we have to be 10 X better than everybody else in the business. Otherwise the merchant companies don't get a breakout and become long and during companies. And so there's a number of different dimensions. The first dimension that I think is probably the most important is efficacy, you know in anomaly detection or in, you know threat detection where you're trying to identify what risks we have in the business. It's, it's generally a very noisy activity. And so rules-based approaches on average will produce a hundred alerts to our one or two. Those, the signal to noise ratio, is, is, you know is a massive a 100x, but call it 10x a reduction. And so we're actually delivering the needle versus the haystack for security administrators and dev developers to actually solve the problem. So it's 10x, higher efficacy it's 10x faster to be able to resolve the problems. And obviously the ROI is, is a no-brainer because you're eliminating all these points which is in having to manage it. And the third, and probably the thing that I'm most excited about what we're doing and what our customers are already realizing is that we're transforming security and compliance teams from kind of compliance into business enablers. when you automate all these processes and you build it into, you know the CICD platforms for the developers you actually enable the developers to write code to differentiate their business, you know to create new customer experiences to get competitive advantage and drive revenue for their businesses. And, and you know that's not what security has done up to this point. We oftentimes, they're the ones we're the ones having to say, no, you know we're slow down or it's too risky, etc. But when you automate that and you increase the efficacy you can enable the developers to do their thing. And it allows the CSOs and allows the security professionals to up level their responsibility into selling and driving revenue. And that is increasingly going to become more and more important for supply chains and partners of these cloud native businesses of how secure am I working with you, etc. And so we think that that transformation of the role of security is going to be as, as meaningful as the technology that we're providing the business. So we're super excited about it. >> I could tell you have so much going on this investment team Sutter Hill, you mentioned big time players huge success track record. Just saw them written up in the wall street journal as one of the best venture capital firms and returns. It's just that the bets are all coming home, but their bet strategy is simple. Disrupting the market that's growing and changing PureStorage, you mentioned company you've worked for, you know people were saying, oh, they'll never get escape velocity. They disrupted an existing, boring storage market changed the game there, security, right for change. A lot of tools, a lot of people have buying tools off the shelf, you know and everyone fighting for the platform. That seems to be the conversation. So I have to ask you, you guys want to be the player that that platform you are, that platform what's different in this platform where everyone's trying to be a security platform, what's makes you different. >> Yeah. So I mean, I think the platform wars are, are clearly, upon us, you know I think what's different about our approach is that we were built on the cloud, for the cloud so we're a cloud native business that, you know runs our business on AWS and everything that we do. We don't have hardware, we don't own data centers. we don't have any of the legacy elements that are there. we use software run on the cloud to enable this. So that's point number one point number two is we did the hard work of mapping the data elements that are out there and adjusting them in and then have this polygraph, you know behavioral anomaly detection, that is it can be applied to today. It's being applied to vulnerability and discovery management and containers and Kubernetes. But over time we believe it extends very naturally to a larger part of the attack server. So we don't have to rewrite the data engine to develop solutions across broader attack services. We already have that, you know so I think our time to develop and innovate will be profound. And I think the third thing that we're seeing companies do and largely the legacy bigger companies is that they're just acquiring their way there. And, it's very, very difficult to acquire 8 to 10 to 20, 30 companies, 30 different CTOs 30 different code bases and try and integrate them to provide a delightful customer experience. And, the parallels, you know in the storage business are, are are pretty similar actually, Dell bought EMC, EMC bought a hundred companies. And, we went after a platform approach to be able to go attack them with a unified file system in a in a unified customer experience that was native for the media that we're working with. We're doing the same playbook here, you know which is you have to have the hard work of the foundation elements in place to be cloud native to deliver great outcomes, great efficacy and and a really great customer experience. So when we get head to head with any of these points coming out and trying to solve something for containers or Kubernetes, or just vulnerability discovery and management, etc, or we're competing with the legacy companies that have, a hodgepodge of acquisitions that they're trying to pull together we went North of 95% of the time. our POC win rates are phenomenal better than anything I've ever seen. We had a pretty good one to appear too. And the, the product and the experience and the efficacy kind of stand on their own once we're in those fights. So part of why we enjoy working with AWS and are really focused on building the partnership together is that it creates awareness of what could be and what possibilities all we want is a shot. And, our approach is such that you can be up and running in minutes, you know and every single one of our customers does a POC. So we'll stand behind our technology as our real differentiator compared to anybody else that's out there. >> Great. You guys had great traction going on with the company certainly saw the investment news that you mentioned earlier at the top. Why did you come on as CEO? And when did you come on and join the team? And what was the reason? What, what, what attracted you to join as the CEO of Lacework? >> Well, I've been involved in the company for since the beginning actually I invested in the early rounds participated on the board and I've always bought into this. The thesis that security is fundamentally a data problem. And if we can get the data problem and the data processing right, you know you can fundamentally change the industry but you need to have a major inflection. And that inflection is people moving to the cloud. And we all have seen it during the pandemic. things are accelerating. AWS just did their earnings yesterday. I think they increased their top-line guidance from 46 billion to 56 billion this year. I mean, it's a machine that is continuing to move forward. They have 30% market share. Azure's investing at 20% GCP still investing people are moving their businesses online aggressively. And as they shift to the cloud the rules-based approach just doesn't work. It doesn't scale. And so a new approach needs to be done. And so by being cloud native and best of breed and solving the thorny problem of this data processing problem first, you know it gives us an opportunity to use that to then extend and build a business, you know at an enduring level over the next 10 to 20 years. And that's Sutter's model, that's their playbook. They don't invest in 400 companies and kind of spray and pray, which is what most venture funds do. And I love them. They're great. And we appreciate the investment in tech, but Sutter's focus is find a really big market find a catalyst for change. In our case, it's moving to the cloud and then build a modern approach. that is 10x better in every dimension. And that attracted to me. I mean, it's, it's a, it's one of the biggest markets in tech and it's one of the most important things that we can do is a digital business is to ensure that we're secure and we're safe and the threats are becoming much more skilled much more deliberate, much better funded. And so the importance for us to ensure that company's security is really tight is, is increasingly critical. So the combination of those factors, and then as I dove back into it and talked to a bunch of customers and talk to partners and seeing the outcomes and enthusiasm that they had and the, the team is phenomenal. And so talking to them, and I just kind of got energized by the opportunity to go build a really important company that really delivers great outcomes. So I'm having a ball great to be back into it. >> Yeah. It's great to have leadership that has experienced that you have and go to the next level because this is classic next level. When you talk about Amazon's earnings and cloud scale and hybrid and edge right around the corner at scale as well. So you start to see that transformation really hit the tipping point, which is changing the landscape on the developer side, which I think is super valuable. I think you hit that. You mentioned core problem. You guys look at that through the lens of data problem. How does this trend of everything going hybrid and soon to be, you know edge core to edge impact your businesses of tailwind? How do you see you capturing that next level of scale from a business perspective for lease work? >> Well, I think that the trend, you know from core to edge, you know, hybrid and, you know ultimately cloud a hundred percent, there we've started with the cloud native businesses. Like, we've been focusing on those companies that are already there, you know and so now we're we just had finished a phenomenal record-breaking Q1 and multiple seven figure deals, you know with very complex global environments where they do have a hybrid environment and they are leveraging the edge. And we're perfect for that. I mean, as you think about what we deliver in its most simplistic context, you know we're effectively delivering a security solution from the container to control plane, right. You know we want to be able to have a granular understanding of operated trillions of data points coming in and those can be collected in the core. They can be collected on-prem. They can be collected in the cloud. Ultimately they need to be collected and then contextualized so, you know and this is where our behavioral polygraph technology transitions data into information that's useful via the polygraph. And so we think that, the complexity that's added with environments that are hybrid environments that are leveraging the edge environments that are leveraging the cloud native all need a control plane to run across that to deliver efficacy, you know, for our customers. And, we work with, you know AWS has their own security tools. Azure has some security tools UCPs security tools, but ultimately, our, our challenge and opportunity is to be best of breed to deliver incremental value on top of that and that horizontal value across it. so customers have choice but they know that their security posture is, is, is secure. And so we, we see it as a tailwind for our businesses as we go forward. >> I always said the companies that have the horizontal scalability with cloud and then have that vertical AI kind of vibe where you can get in the context of the data is there to win it all. And I think that you guys have a great solution potentially there. I want to get more information if you don't mind double clicking on that with me, this is kind of a different take on cloud security because you've got the scalability, which gives you the observation space. And then you got to get the context to get the right patterns or whatever magic you guys have in the, in the secret sauce. But you doing that on top of massive exponential velocity. >> Yeah. >> Where's that secret sauce? Is it in the compute? Is it in the software? What's different about what you guys have in security to give us a- >> It's all in the, it's all in the software. Ultimately, it's the intelligence of how you capture it how you ingest it, how you, you process it but then ultimately how you, how you contextualize it and then how you apply it to different problems. and so the attack surface area and security is a very broad, that's why there's so many point solutions that are out there. And so the breadth of solutions, you know we just want to continue to add solutions and capabilities on top of this polygraph security architecture that allows for the same kind of simple experience, the same kind of 10x value proposition, but, but, but wider. And so we can eliminate more and more of those of those point solutions. So, our, our thinking on it is that, you know we can participate once we have a customer the land and expand motion of what we have. We want to make it really really frictionless for customers to try our technology. And so that's why we do POC. That's why it only takes a couple of minutes and you can do it for just Kubernetes or just containers or just vulnerability discovery and managed like wherever your specific pain point is. We want to help identify what that is, you know give you a chance to try it. And then once we prove ourselves it's very easy to extend that across the board. So we get natural growth in velocity from people moving to cloud and just, you know more usage of, of compute and storage and sort of etc, but breadth of actually the security or posture or a tax service that they have as well. So, you know so I think we have an opportunity to benefit from, from both the depth and the breadth, you know but the value that we're delivering is ultimately the software that we're running on top of the infrastructure. And you mentioned observability, there's a number of companies that are leveraging the data and insights collected in different ways to converge security and observability over time. And, we see that, you know that ultimately there's a very very big security company that needs to be built. That really is best of breed, but the data and the insights that we're providing to our primary customer, which is really DevOps. I mean, it's really the development communities and the builders or who we're changing security for and enabling, in addition to the security teams, you know we think that we're going to continue to drive software that adds value on that data set and it can be applied to multiple problems in the future. So today security is a massive market. We're going to focus there, but it does. It does extend pretty naturally to other markets >> It's a hot market security. Everyone needs to have the latest and greatest and also has to be effective. I got to ask you specifically around startup transition to a rapidly growing company to now you're going to the next level where you're starting to having to get into some serious, big complex enterprise go to market sales motions. So what's in it for the customer. What's the, what's the pain point? What's the customer orientation. What do you marketing into as a solution? Is it the developer? Is it the CSO? Is it the CXO, what's in it for the enterprise? Why Lacework, why are they engaging? You guys get record numbers. What's the, what's in it for them. What's the, if I'm the customer what's in it for me? >> Ultimately efficacy, which is your security posture is it goes up significantly, simplicity, which is makes it easier for you to do your other jobs, you know and I'll have to look for those needles in a haystack and ROI, you know which is it's just compelling, and much, much more efficient than what, what you're doing today. So that that's a pretty universal value proposition and applies to cloud native businesses that are high growth that applies to government agencies. It applies to a large complex enterprises. We have a wonderful kind of go to market motion right now. I think Andy Byron and the team who've been here have really done a wonderful job of really making the customer buying experience and the journey really efficient, you know and help them quantify the impact and the risks and then deliver value. And I think, that that applies in sort of the commercial mid-market and cloud native space. And like I mentioned, we had, a number of deals in the quarter that were seven figure deals, you know in very complex organizations with massive demands. And, you know it ultimately selling is a team sport and, you know and still having the process and the rigor, that's there fine tuning that to make sure you have the people and the partnerships, you know, that deliver solutions in the way that customers want to buy them and then ultimately deliver a value proposition that is just unquestionably better. And I think we have all of those elements, you know we'll be entering the, the large enterprise very aggressively in the quarters to come. I that's where I've come from, you know running a multi-tool, you know, kind of go to market engines where you've got mid-market commercial enterprise large enterprise government across all geographies is, is really fun to expand. And, we're we're hiring as fast as we can maintain quality, you know? And so we're out of that startup phase now and entering into real scale. And, I think that, you know in the AWS marketplace I think we're the number one startup vendor. If I, if I got my facts, right. for, for private offers, we're one of the top security players and top 50 ISBs in the marketplace overall. And so in order for us to get the motion we need to make sure that we're delivering our value in the context of how companies want to buy it. And people want to use AWS credits, you know to apply to their solutions. And so it's really important for us to make that frictionless buying experience occur. And so we're excited about it. I think we've got a really nice start and it's the fun part of building companies, which is how do you attune things to make sure you're making it really really easy for the market to absorb your technology. And then once you're there, delight the hell out of them and just make sure that, that there's that they're excited in our, our net retention rates are the best I've seen in the marketplace. Our net promoter scores, you know, are in the high fifties low sixties, which, which is fantastic in this space. I think it's best in class by order of magnitude some players, big SIM players that are out there, you know have a customer in net promoter score of four. You know that means 96% of the people or 96 boats that says they wouldn't recommend the solution to their, to their peers. So, at pure, we've got this at scale. So from 70 to, in the, in the low eighties I think we have the opportunity to do the same thing here. So, combination of tailoring the motion that we have making it really easy for the buyer to buy what they want with whom they want from whom they want, you know and then just spreading a value proposition. That is a no brainer is, is I think the secret recipe >> If anything, it's interesting, you know you're so much experience in the enterprise and tech with cloud native you're basically laying out the success formula, which is if you have a value proposition you should be able to get it in quickly. You don't need the top down. win everything you can have a value proposition that can be enabled for usage and then grow rapidly when it's successful and that's cloud, that's the cloud business model. So it's not so much about organic versus this. It's really what the preferred motion is. >> It's speed, and I think developers in particular it's why the cloud happened, right? I.T wasn't delivering services in, in the speed and the efficacy that, that, that the developers wanted. And so in order to appeal to the developer community you need to deliver something that's frictionless and easy and fits into JIRA and fits into their workflow processes and speaks their language. And so we built our platform and our solutions for builders because that's where the money is. That's where the pain point is and that's and they want to build secure code. They just don't want to be told no. And so, we want to automate that process and make code secure and do that, you know in the build phase and then do it in the runtime. And then across the CICD pipeline we want to continuously be adding value across that. And, and the developers, candidly when pure bought the solution, many years ago and I introduced him to the company, it was it was the general manager of our software business unit that bought it not the security team. And I think that's a trend that is continuing that we're going to focus on. >> A lot of people realize that security and compliance and automation kind of all go together where you don't want to disrupt developers to kind of engineer something just to do an integration, for instance. So there's a real business model impact that you're hitting on here. That's not just a technical solution. It's really how the business is operating. And I think that to me is super interesting use case. What's your reaction to that? Do you see this as a, as a- >> No it's, that's that's that third part that I was talking about, you know which is that's most exciting is that, you know people are calling shift left, right. so moving, you know security into the development pipeline as it's happening and in integrating security architects as value added into the development organizations themselves and leveraging automated machine learning tools like ours to be able to simplify and automate the process versus slowing it down. So we think that shift left is, is super exciting and, and will continue. And we actually think we're the leaders in that space. We want to continue to be the leaders in that. >> Congratulations, great insight. Awesome to have you on and to hear from your experience and also the great venture that your scaling up and to the next level. Lacework, David thanks for coming on, but I'll give you the last minute to close us out. Give us a quick plug for the company vitals, what you're working on now, what you're looking for, you're obviously hiring give a quick plug for Lacework. What you, what are you working on? >> So, number one, we love our partnership with AWS. And so we're going to continue to invest, invest there. Two the businesses growing North of 300% year over year. That means that we've got record breaking growth and lots of hiring. So we're hiring across all functions. And three give us an opportunity. I, I think that, you know, you can fundamentally we want to be the bar of what you define all other security companies and all the technology companies. So it's a high bar. We want to make it frictionless, frictionless to try give us a shot, give us some feedback. And I'm grateful and privileged to be part of this, this wonderful team. So look forward to spending more time with you, John, in the future. >> Man, looking forward to a lot lots of talk about David Hatfield CEO of Lacework great company scaling up again. Another success story in cloud, cloud native as Po, COVID comes to a close, if you will for this phase and people get back to real life. The scale of cloud is going to be leading it and a new technology is going to be powering it. This is theCube conversation. I'm John Furrier. Thanks for watching. (soft music playing) (music fades)

>>From around the globe. It's the cube with digital coverage of AWS reinvent 2020, sponsored by Intel and AWS. >>Welcome everyone to the cube live and our coverage of AWS reinvent 2020. It's good to have you. I'm your host Rebecca Knight today, we are joined by Mira Vaidhyanathan. She is the product leader for Amazon honey code at AWS. Thank you so much for coming on the cube Mira, my question to be here. So tell our viewers a little bit about Amazon honey code. This was a product that was announced in June of this year. What was the impetus for it? What were you hearing from customers that made you realize there was a need for this? >>Yeah, so Amazon honey code is a fully managed service that allows customers to build powerful mobile and web applications without the need for any programming. So customer has to, they have a growing need, uh, to manage data over time, manage workflows that involve multiple people that facilitate complex business processes. And today we're doing this through spreadsheets and mailing, these spreadsheets via email. And what ends up happening is you have a whole lot of spreadsheets with different data, and it usually falls to one person to consolidate all the information and create a source of truth, um, organizations that have the resources to build custom applications do so, but quite often, these applications just don't get built. And, um, employees in these businesses are managing with these, uh, band-aids set of tools that I just discussed. And, um, so what we wanted to do was to build a, uh, no code, uh, app building platform that customers can use their existing skills to build the apps that they need for their day-to-day lives. So no programming required. You bring the skills, you have to just, uh, you know, those of using spreadsheets to be able to build, uh, apps to manage, um, all of your productivity and collaboration needs. So we tried to do with honeycomb. >>What has the reception been since you launched back in Sharon, what are, what are you hearing from developers about how it's changed the way they're doing their business? >>Customers are very excited that AWS now has a solution in this space. And from the very first day, from the day of launch, we've just seen a lot of interests from organizations of all sizes, both domestic and international and customers have been building apps to solve various problems. In fact, the very first app that a customer has shared with us was, uh, a COVID tracking app for HL care center in New Hampshire, where, uh, you know, parents had been standing in line for tens of minutes waiting to drop off their children and filling out a form at the entry point. And, um, this, uh, customer built an app over a weekend, uh, and was able to it reducing the drop-off time two minutes. Um, we've also seen a great deal of activity in our community forum, where customers are exchanging ideas and learning from each other. And what they really like about honeycomb is how easy it is to spin up an application without needing to think about databases or servers or deployments. And they also like that by building just one app it's immediately available in both web and in mobile. And, um, of course the best of all is the fact that all of the data is up-to-date and they're able to make informed decisions based on the data in these apps. Um, customers have also been very forthcoming about, uh, feature ideas and requests, and that is continually feeding into our roadmap. >>So I want to talk about some other use cases. You mentioned the childcare center in New Hampshire, which sounds as though you helped save these parents a lot of time and alleviate some of their stress. What other kinds of use cases are you here? >>Sure. Uh, the types of apps that we've heard about include, uh, like leave and vacation requests and organizations, um, a team has built a hotel management, a booking system, contracts management for an unemployment center, sales opportunity tracking, um, status reporting across distributed teams, which is a reality that we're all living today. Um, more specifically we, uh, uh, we know of a customer who has a 6,000 person team, um, and they built an app to manage service costs requests. So this is a systems integrator and they're using this app across 10 partner teams, uh, across the world. Um, we've also heard about a coffee trader who has built an app to manage, uh, their coffee orders across both domestic and foreign markets. And previously they were doing this via email and, uh, through spreadsheets. So those are the different cases that we've heard about >>What kinds of internal interest are you having within AWS for honey code? I'm told that there is a great deal of interest within the organization itself. >>Absolutely. Yeah. There's been a lot of interest that at Amazon, uh, there isn't a day that goes by that I don't hear from a new team that has a use case that they need to build an app today on honeycomb. Um, and these are usually, you know, use cases that customers have been solving with spreadsheets or our internal ticketing tool, uh, because they haven't had the resources to build their own custom app. Um, there, our HR team, um, uh, one of the HR teams at Amazon, in fact, it's built an app that is consolidating across four different tools, so they can get an accurate picture of what is going on with, uh, any particular team, you know, head count, how many roles are still to be filled, et cetera. Um, another example is a marketing team that is managing all of their marketing campaigns, uh, through a Honeycutt app, so they can see how campaigns have we already executed this month. >>How many still remain, what are the results from these campaigns, all of this, uh, in one place. And, um, in fact, in the honeycomb team itself, uh, we, uh, uh, use honey good for, uh, managing all of our internal processes from our product roadmap to, uh, program management, to managing and tracking our goals. And because we're also distributed these days, um, we seem to be spending up on an app on auto practically daily basis. In fact, today the team is running a hackathon and all of the ideas for the hackathon were, um, gathered on a honeycomb app. And then later today we'll be doing demos and voting, uh, on the best time, uh, hackathon project. So it's, it's given rise to a lot of new ideas and, uh, a lot of new ways in which, uh, we're, we're able to work together collaboratively, >>Well, an app a day. I love it. Um, so it does sound like the, kind of the collaboration you're describing and the ways in, within the transparency, particularly during this, these COVID times when people, as you say, we're working, dispersed teams are remote. Um, there's a lot of isolation. It does seem like it's, it's really a revelation. What, you're, what you're doing here. >>Yeah. It's been really, uh, it's been a learning experience for us as well, you know, working remotely and trying to figure out how do we keep each other up-to-date on what we're doing. How do we make sure that, uh, you know, we, we find ways to replace those hallway conversations, those water cooler conversations as we like to call them. Uh, and, and so we find ourselves, uh, interacting via these apps a lot more, trying to keep everyone abreast of what we're doing by updating project status. And so on, in addition, of course, to, you know, uh, uh, meetings, um, online on video, uh, it has certainly helped us all stay on the same page. In fact, um, honeycomb the product launch itself was managed via a honeycomb app. And normally that's something that, you know, most of the teams either build a custom app for, or manage, uh, via a spreadsheet and probably hundreds of post-it notes. >>So the, the product is relatively new, but you had some announcements last week at AWS reinvent. Tell us a little bit about those. >>Yeah. And the last few weeks we've had a slew of new announcements and they fall into three major areas, really, um, integrations, uh, identity and app building features. Um, for the first we announced, um, integrations with Amazon app flow and Zapier to integrate with external data sources to push and pull data into and out of honeycomb. Um, we also announced the ability to set up and log in with multiple identity providers, including Okta and Google, to make it easier for our customers to, uh, manage, manage user accounts, um, as well as the first single sign on and last but not least, uh, we've announced several features to make it easier for app builders, as well as the end users of these apps. Um, not only to make the apps more functional, but also more delightful to use. And these include, uh, features like border styles, uh, conditional styling, as well as easier ways to sort and filter your data in your app screens. >>You used the word delightful, which is, which is absolutely an adjective that so many of us associate with Amazon. Tell us a little bit about how you are working to make these, uh, the, the user interface more delightful, as you say. >>Yeah. We're continually adding new features to make it easier. So, you know, every business user doesn't have to think like a UX designer. So we're, we're trying to, um, think about the ways, uh, you, you, you look at all the productivity apps today, you, you want certain sets of data to pop up in your app. Uh, for example, you know, if the status of a project is red, not only do you want it to notify the appropriate parties, but you also want that information to pop up, um, in an app. So it's very easy just using a very simple expression. You can set up the rules, the conditional rules to say, Hey, if the status is red, then, you know, make sure or status is delayed, then pop it up and, you know, bright red. So it catches my eye. The next time I look and look at an app. So we're trying to find ways to, uh, you know, thinking about all of the business use cases, trying to find ways to help customers make the information, um, pop better in their apps. So they're, uh, so they, you know, deliver more value, um, in businesses >>Up here in Amazon app flow. What are the business use cases in terms of those and what are they, what are available now? >>Yeah, so, uh, both, uh, integrations with Zapier and app flow enable customers to build even richer applications because now they're, uh, you know, previously they were building applications just based on the data that the sitting in, in honeycomb and with these integrations. Now they can bring in data from other sources, programmatically. So these include integrations to apps like Salesforce or Slack, JIRA, Amazon S3, et cetera. And, um, this makes it possible for business users to use Zapier or app flow, uh, to, to build, um, powerful integrations. So I'll give you an example. Um, let's say a sales team can use, uh, a honeycomb to build an app to process their sales inquiries. And, uh, instead of dealing with emails and spreadsheets, what they can do is use Zapier to automatically pull in requests that come into their website. Um, and this can be pulled straight into a honeycomb app, which can then generate a notification to the sales manager to approve a quote. Um, and then the quote can be generated and emailed to the customer. All of this is made possible through, um, and integration with, uh, with Zapier. And you can integrate with Amazon app flow, uh, to pull in data from Salesforce. So it makes it possible for customers to, um, use more up-to-date information and their apps making it, uh, driving better decisions and more informed decisions. And what >>Type of new templates are now available. >>Yeah. Uh, back in June, when we launched, we had about 10 templates for common use cases in businesses. And since then we've been regularly adding more to that repository. Uh, our most recent additions to the template library include, uh, the ability to run meetings virtually, which we're doing eight hours a day, these days, instant polls, a collaborative brainstorming template, um, as well as applicant tracking. And we'll continue to add more in the coming months. >>There's just no question that this is such a high growth area. Uh, Gartner estimated last year, that low-code no-code approaches will represent more than 65% of application development inside companies by 2024 foresters also projecting a $21 billion by 2024. What I want you to look into your crystal ball here and just tell us a little bit about what you're expecting in the next six to 12 months and what, and what's what you're hoping for the future for Amazon honey honeycomb. >>Uh, we remained focused on, uh, you know, helping business users solve problems that were previously out of their reach because they either lacked the resources or the skills or support from it. Um, honey code apps have the scale and security that customers expect from an AWS service. And over time, uh, we expect to add more features that make it progressively easier for business users to develop without needing how to learn, how to code. And we will also expect to add features that are required by it, departments for adoption in enterprises, >>Mirror. What have you hearing from customers about what they, what they're wanting to hear from you just w I want you to close us out here and, and give us what you're hearing, and then what your best advice is for managers who are thinking about, uh, trying to adopt some of these low-code no-code approaches and are, and are interested in what they're hearing in terms of what you're saying about the collaboration and transparency that these, that these tools provide. >>Yeah. Um, the, these, these tools make it possible for, uh, anyone in any business, you know, HR marketing program managers, product managers, really, um, anyone to, uh, build applications that are very specific and tailored to your business needs. And these, because these applications don't require the typical process of, uh, you know, selecting a database, selecting the storage layer, selecting all of these things. Um, and they're deployed almost immediately, like as soon as an app is built, it's available to the end users to use it makes it possible for the applications to evolve with your needs. Um, in fact, this is, this is what I see everyday with the apps that we build for ourselves is, you know, it works this week next week. We're like, Hmm, what if we tried something slightly differently? Uh, because our, our, um, you know, we, we we've become more efficient, our techniques evolve over time. >>Uh, and th th th the situation changes as we're seeing every day, uh, uh, in COVID times. So it makes it possible to just, uh, to, to have the applications grow with you as your business grows and evolves. And, um, so that's, what's really exciting for me with a honeycomb is, um, uh, things that were, uh, you know, usually, uh, out of the reach of, of business users now, uh, you know, they're able to build these applications and they use the exact same skills that, um, they might have used with spreadsheets. So, uh, that's, what's really exciting and we're going to continue to listen to our customers. And, uh, we know that business users want to be more productive and want easier to use tools. And that's what we'll be working on >>Mira bitey enough. And thank you so much for coming on the Cuba. It was a real pleasure talking to you. Likewise, thank you so much. And thank you for tuning into the cubes coverage of AWS reinvent 2020 I'm Rebecca Knight stay tuned.

>>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi, Lisa Martin here covering the Broadcom dev ops virtual forum. I'm very pleased to be joined today by a cube alumni, Jeffrey Hammond, the vice president and principal analyst serving CIO is at Forester. Jeffrey. Nice to talk with you today. >>Good morning. It's good to be here. Yeah. >>So a virtual forum, great opportunity to engage with our audiences so much has changed in the last it's an understatement, right? Or it's an overstated thing, but it's an obvious, so much has changed when we think of dev ops. One of the things that we think of is speed, you know, enabling organizations to be able to better serve customers or adapt to changing markets like we're in now, speaking of the need to adapt, talk to us about what you're seeing with respect to dev ops and agile in the age of COVID, what are things looking like? >>Yeah, I think that, um, for most organizations, we're in a, uh, a period of adjustment, uh, when we initially started, it was essentially a sprint, you know, you run as hard as you can for as fast as you can for as long as you can and you just kind of power through it. And, and that's actually what, um, the folks that get hub saw in may when they ran an analysis of how developers, uh, commit times and a level of work that they were committing and how they were working, uh, in the first couple of months of COVID was, was progressing. They found that developers, at least in the Pacific time zone were actually increasing their work volume, maybe because they didn't have two hour commutes or maybe because they were stuck away in their homes, but for whatever reason, they were doing more work. >>And it's almost like, you know, if you've ever run a marathon the first mile or two in the marathon, you feel great and you just want to run and you want to power through it and you want to go hard. And if you do that by the time you get to mile 18 or 19, you're going to be gassed. It's sucking for wind. Uh, and, and that's, I think where we're starting to hit. So as we start to, um, gear our development chops out for the reality that most of us won't be returning into an office until 2021 at the earliest and many organizations will, will be fundamentally changing, uh, their remote workforce, uh, policies. We have to make sure that the agile processes that we use and the dev ops processes and tools that we use to support these teams are essentially aligned to help developers run that marathon instead of just kind of power through. >>So, um, let me give you a couple of specifics for many organizations, they have been in an environment where they will, um, tolerate Rover remote work and what I would call remote work around the edges like developers can be remote, but product managers and, um, you know, essentially scrum masters and all the administrators that are running the, uh, uh, the SCM repositories and, and the dev ops pipelines are all in the office. And it's essentially centralized work. That's not, we are anymore. We're moving from remote workers at the edge to remote workers at the center of what we do. And so one of the implications of that is that, um, we have to think about all the activities that you need to do from a dev ops perspective or from an agile perspective, they have to be remote people. One of the things I found with some of the organizations I talked to early on was there were things that administrators had to do that required them to go into the office to reboot the SCM server as an example, or to make sure that the final approvals for production, uh, were made. >>And so the code could be moved into the production environment. And so it actually was a little bit difficult because they had to get specific approval from the HR organizations to actually be allowed to go into the office in some States. And so one of the, the results of that is that while we've traditionally said, you know, tools are important, but they're not as important as culture as structure as organization as process. I think we have to rethink that a little bit because to the extent that tools enable us to be more digitally organized and to hiring, you know, achieve higher levels of digitization in our processes and be able to support the idea of remote workers in the center. They're now on an equal footing with so many of the other levers, uh, that, that, um, uh, that organizations have at their disposal. Um, I'll give you another example for years. >>We've said that the key to success with agile at the team level is cross-functional co located teams that are working together physically co located. It's the easiest way to show agile success. We can't do that anymore. We can't be physically located at least for the foreseeable future. So, you know, how do you take the low hanging fruits of an agile transformation and apply it in, in, in, in the time of COVID? Well, I think what you have to do is that you have to look at what physical co-location has enabled in the past and understand that it's not so much the fact that we're together looking at each other across the table. It's the fact that we're able to get into a shared mindspace, uh, from, um, uh, from a measurement perspective, we can have shared purpose. We can engage in high bandwidth communications. It's the spiritual aspect of that physical co-location that is actually important. So one of the biggest things that organizations need to start to ask themselves is how do we achieve spiritual colocation with our agile teams? Because we don't have the, the ease of physical co-location available to us anymore? >>Well, the spiritual co-location is such an interesting kind of provocative phrase there, but something that probably was a challenge here, we are seven, eight months in for many organizations, as you say, going from, you know, physical workspaces, co-location being able to collaborate face to face to a, a light switch flip overnight. And this undefined period of time where all we were living with with was uncertainty, how does spiritual, what do you, when you talk about spiritual co-location in terms of collaboration and processes and technology help us unpack that, and how are you seeing organizations adopted? >>Yeah, it's, it's, um, it's a great question. And, and I think it goes to the very root of how organizations are trying to transform themselves to be more agile and to embrace dev ops. Um, if you go all the way back to the, to the original, uh, agile manifesto, you know, there were four principles that were espoused individuals and interactions over processes and tools. That's still important. Individuals and interactions are at the core of software development, processes and tools that support those individual and interact. Uh, those individuals in those interactions are more important than ever working software over comprehensive documentation. Working software is still more important, but when you are trying to onboard employees and they can't come into the office and they can't do the two day training session and kind of understand how things work and they can't just holler over the cube, uh, to ask a question, you may need to invest a little bit more in documentation to help that onboarding process be successful in a remote context, uh, customer collaboration over contract negotiation. >>Absolutely still important, but employee collaboration is equally as important if you want to be spiritually, spiritually co-located. And if you want to have a shared purpose and then, um, responding to change over following a plan. I think one of the things that's happened in a lot of organizations is we have focused so much of our dev ops effort around velocity getting faster. We need to run as fast as we can like that sprinter. Okay. You know, trying to just power through it as quickly as possible. But as we shift to, to the, to the marathon way of thinking, um, velocity is still important, but agility becomes even more important. So when you have to create an application in three weeks to do track and trace for your employees, agility is more important. Um, and then just flat out velocity. Um, and so changing some of the ways that we think about dev ops practices, um, is, is important to make sure that that agility is there for one thing, you have to defer decisions as far down the chain to the team level as possible. >>So those teams have to be empowered to make decisions because you can't have a program level meeting of six or seven teams and one large hall and say, here's the lay of the land. Here's what we're going to do here are our processes. And here are our guardrails. Those teams have to make decisions much more quickly that developers are actually developing code in smaller chunks of flow. They have to be able to take two hours here or 50 minutes there and do something useful. And so the tools that support us have to become tolerant of the reality of, of, of, of how we're working. So if they work in a way that it allows the team together to take as much autonomy as they can handle, um, to, uh, allow them to communicate in a way that, that, that delivers shared purpose and allows them to adapt and master new technologies, then they're in the zone in their spiritual, they'll get spiritually connected. I hope that makes sense. >>It does. I think we all could use some of that, but, you know, you talked about in the beginning and I've, I've talked to numerous companies during the pandemic on the cube about the productivity, or rather the number of hours of work has gone way up for many roles, you know, and, and, and times that they normally late at night on the weekends. So, but it's a cultural, it's a mind shift to your point about dev ops focused on velocity, sprints, sprints, sprints, and now we have to, so that cultural shift is not an easy one for developers. And even at this folks to flip so quickly, what have you seen in terms of the velocity at which businesses are able to get more of that balance between the velocity, the sprint and the agility? >>I think, I think at the core, this really comes down to management sensitivity. Um, when everybody was in the office, you could kind of see the mental health of development teams by, by watching how they work. You know, you call it management by walking around, right. We can't do that. Managers have to, um, to, to be more aware of what their teams are doing, because they're not going to see that, that developer doing a check-in at 9:00 PM on a Friday, uh, because that's what they had to do, uh, to meet the objectives. And, um, and, and they're going to have to, to, um, to find new ways to measure engagement and also potential burnout. Um, friend of mine once had, uh, had a great metric that he called the parking lot metric. It was helpful as the parking lot at nine. And how full was it at five? >>And that gives you an indication of how engaged your developers are. Um, what's the digital equivalent equivalent to the parking lot metric in the time of COVID it's commit stats, it's commit rates. It's, um, you know, the, uh, the turn rate, uh, that we have in our code. So we have this information, we may not be collecting it, but then the next question becomes, how do we use that information? Do we use that information to say, well, this team isn't delivering as at the same level of productivity as another team, do we weaponize that data or do we use that data to identify impedances in the process? Um, why isn't a team working effectively? Is it because they have higher levels of family obligations and they've got kids that, that are at home? Um, is it because they're working with, um, you know, hardware technology, and guess what, they, it's not easy to get the hardware technology into their home office because it's in the lab at the, uh, at the corporate office, uh, or they're trying to communicate, uh, you know, halfway around the world. >>And, uh, they're communicating with a, with an office lab that is also shut down and, and, and the bandwidth just doesn't enable the, the level of high bandwidth communications. So from a dev ops perspective, managers have to get much more sensitive to the, the exhaust that the dev ops tools are throwing off, but also how they're going to use that in a constructive way to, to prevent burnout. And then they also need to, if they're not already managing or monitoring or measuring the level of developer engagement, they have, they really need to start whether that's surveys around developer satisfaction, um, whether it's, you know, more regular social events, uh, where developers can kind of just get together and drink a beer and talk about what's going on in the project, uh, and monitoring who checks in and who doesn't, uh, they have to, to, um, work harder, I think, than they ever have before. >>Well, and you mentioned burnout, and that's something that I think we've all faced in this time at varying levels and it changes. And it's a real, there's a tension in the air, regardless of where you are. There's a challenge, as you mentioned, people having, you know, coworker, their kids as coworkers and fighting for bandwidth, because everyone is forced in this situation. I'd love to get your perspective on some businesses that are, that have done this well, this adaptation, what can you share in terms of some real-world examples that might inspire the audience? >>Yeah. Uh, I'll start with, uh, stack overflow. Uh, they recently published a piece in the journal of the ACM around some of the things that they had discovered. Um, you know, first of all, just a cultural philosophy. If one person is remote, everybody is remote. And you just think that way from an executive level, um, social spaces. One of the things that they talk about doing is leaving a video conference room open at a team level all day long, and the team members, you know, we'll go on mute, you know, so that they don't have to, that they don't necessarily have to be there with somebody else listening to them. But if they have a question, they can just pop off mute really quickly and ask the question. And if anybody else knows the answer, it's kind of like being in that virtual pod. Uh, if you, uh, if you will, um, even here at Forrester, one of the things that we've done is we've invested in social ceremonies. >>We've actually moved our to our team meetings on, on my analyst team from, from once every two weeks to weekly. And we have built more time in for social Ajay socialization, just so we can see, uh, how, how, how we're doing. Um, I think Microsoft has really made some good, uh, information available in how they've managed things like the onboarding process. I think I'm Amanda silver over there mentioned that a couple of weeks ago when, uh, uh, a presentation they did that, uh, uh, Microsoft onboarded over 150,000 people since the start of COVID, if you don't have good remote onboarding processes, that's going to be a disaster. Now they're not all developers, but if you think about it, um, everything from how you do the interviewing process, uh, to how you get people, their badges, to how they get their equipment. Um, security is a, is another issue that they called out typically, uh, it security, um, the security of, of developers machines ends at, at, at the corporate desktop. >>But, you know, since we're increasingly using our own machines, our own hardware, um, security organizations kind of have to extend their security policies to cover, uh, employee devices, and that's caused them to scramble a little bit. Uh, so, so the examples are out there. It's not a lot of, like, we have to do everything completely differently, but it's a lot of subtle changes that, that have to be made. Um, I'll give you another example. Um, one of the things that, that we are seeing is that, um, more and more organizations to deal with the challenges around agility, with respect to delivering software, embracing low-code tools. In fact, uh, we see about 50% of firms are using low-code tools right now. We predict it's going to be 75% by the end of next year. So figuring out how your dev ops processes support an organization that might be using Mendix or OutSystems, or, you know, the power platform building the front end of an application, like a track and trace application really, really quickly, but then hooking it up to your backend infrastructure. Does that happen completely outside the dev ops investments that you're making and the agile processes that you're making, or do you adapt your organization? Um, our hybrid teams now teams that not just have professional developers, but also have business users that are doing some development with a low-code tool. Those are the kinds of things that we have to be, um, willing to, um, to entertain in order to shift the focus a little bit more toward the agility side, I think >>Lot of obstacles, but also a lot of opportunities for businesses to really learn, pay attention here, pivot and grow, and hopefully some good opportunities for the developers and the business folks to just get better at what they're doing and learning to embrace spiritual co-location Jeffrey, thank you so much for joining us on the program today. Very insightful conversation. >>My pleasure. It's it's, it's an important thing. Just remember if you're going to run that marathon, break it into 26, 10 minute runs, take a walk break in between each and you'll find that you'll get there. >>Digestible components, wise advice. Jeffery Hammond. Thank you so much for joining for Jeffrey I'm Lisa Martin, you're watching Broadcom's dev ops virtual forum >>From around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom, >>Continuing our conversations here at Broadcom's dev ops virtual forum. Lisa Martin here, please. To welcome back to the program, Serge Lucio, the general manager of the enterprise software division at Broadcom. Hey, Serge. Welcome. Thank you. Good to be here. So I know you were just, uh, participating with the biz ops manifesto that just happened recently. I just had the chance to talk with Jeffrey Hammond and he unlocked this really interesting concept, but I wanted to get your thoughts on spiritual co-location as really a necessity for biz ops to succeed in this unusual time in which we're living. What are your thoughts on spiritual colocation in terms of cultural change versus adoption of technologies? >>Yeah, it's a, it's, it's quite interesting, right? When we, when we think about the major impediments for, uh, for dev ops implementation, it's all about culture, right? And swore over the last 20 years, we've been talking about silos. We'd be talking about the paradox for these teams to when it went to align in many ways, it's not so much about these teams aligning, but about being in the same car in the same books, right? It's really about fusing those teams around kind of the common purpose, a common objective. So to me, the, this, this is really about kind of changing this culture where people start to look at a kind of OKR is instead of the key objective, um, that, that drives the entire team. Now, what it means in practice is really that's, uh, we need to change a lot of behaviors, right? It's not about the Yarki, it's not about roles. It's about, you know, who can do what and when, and, uh, you know, driving a bias towards action. It also means that we need, I mean, especially in this school times, it becomes very difficult, right? To drive kind of a kind of collaboration between these teams. And so I think there there's a significant role that especially tools can play in terms of providing this complex feedback from teams to, uh, to be in that preface spiritual qualification. >>Well, and it talked about culture being, it's something that, you know, we're so used to talking about dev ops with respect to velocity, all about speed here. But of course this time everything changed so quickly, but going from the physical spaces to everybody being remote really does take it. It's very different than you can't replicate it digitally, but there are collaboration tools that can kind of really be essential to help that cultural shift. Right? >>Yeah. So 2020, we, we touch to talk about collaboration in a very mundane way. Like, of course we can use zoom. We can all get into, into the same room. But the point when I think when Jeff says spiritual, co-location, it's really about, we all share the same objective. Do we, do we have a niece who, for instance, our pipeline, right? When you talk about dev ops, probably we all started thinking about this continuous delivery pipeline that basically drives the automation, the orchestration across the team, but just thinking about a pipeline, right, at the end of the day, it's all about what is the meantime to beat back to these teams. If I'm a developer and a commit code, I don't, does it take where, you know, that code to be processed through pipeline pushy? Can I get feedback if I am a finance person who is funding a product or a project, what is my meantime to beat back? >>And so a lot of, kind of a, when we think about the pipeline, I think what's been really inspiring to me in the last year or so is that there is much more of an adoption of the Dora metrics. There is way more of a focus around value stream management. And to me, this is really when we talk about collaboration, it's really a balance. How do you provide the feedback to the different stakeholders across the life cycle in a very timely matter? And that's what we would need to get to in terms of kind of this, this notion of collaboration. It's not so much about people being in the same physical space. It's about, you know, when I checked in code, you know, to do I guess the system to automatically identify what I'm going to break. If I'm about to release some allegation, how can the system help me reduce my change pillar rates? Because it's, it's able to predict that some issue was introduced in the outpatient or work product. Um, so I think there's, there's a great role of technology and AI candidate Lynch to, to actually provide that new level of collaboration. >>So we'll get to AI in a second, but I'm curious, what are some of the, of the metrics you think that really matter right now is organizations are still in some form of transformation to this new almost 100% remote workforce. >>So I'll just say first, I'm not a big fan of metrics. Um, and the reason being that, you know, you can look at a change killer rate, right, or a lead time or cycle time. And those are, those are interesting metrics, right? The trend on metric is absolutely critical, but what's more important is you get to the root cause what is taught to you lean to that metric to degrade or improve or time. And so I'm much more interested and we, you know, fruit for Broadcom. Are we more interested in understanding what are the patterns that contribute to this? So I'll give you a very mundane example. You know, we know that cycle time is heavily influenced by, um, organizational boundaries. So, you know, we talk a lot about silos, but, uh, we we've worked with many of our customers doing value stream mapping. And oftentimes what you see is that really the boundaries of your organization creates a lot of idle time, right? So to me, it's less about the metrics. I think the door metrics are a pretty, you know, valid set metrics, but what's way more important is to understand what are the antiperspirants, what are the things that we can detect through the data that actually are affecting those metrics. And, uh, I mean, over the last 10, 20 years, we've learned a lot about kind of what are, what are the antiperspirants within our large enterprise customers. And there are plenty of them. >>What are some of the things that you're seeing now with respect to patterns that have developed over the last seven to eight months? >>So I think the two areas which clearly are evolving very quickly are on kind of the front end of the life cycle, where DevOps is more and more embracing value stream management value stream mapping. Um, and I think what's interesting is that in many ways the product is becoming the new silo. Uh, the notion of a product is very difficult by itself to actually define people are starting to recognize that a value stream is not its own little kind of Island. That in reality, when I define a product, this product, oftentimes as dependencies on our products and that in fact, you're looking at kind of a network of value streams, if you will. So, so even on that, and there is clearly kind of a new sets, if you will, of anti-patterns where products are being defined as a set of OTRs, they have interdependencies and you have have a new set of silos on the operands, uh, the Abra key movement to Israel and the SRE space where, um, I think there is a cultural clash while the dev ops side is very much embracing this notion of OTRs and value stream mapping and Belgium management. >>On the other end, you have the it operations teams. We still think business services, right? For them, they think about configure items, think about infrastructure. And so, you know, it's not uncommon to see, you know, teams where, you know, the operations team is still thinking about hundreds of thousands, tens of thousands of business services. And so the, the, there is there's this boundary where, um, I think, well, SRE is being put in place. And there's lots of thinking about what kind of metrics can be fined. I think, you know, going back to culture, I think there's a lot of cultural evolution that's still required for true operations team. >>And that's a hard thing. Cultural transformation in any industry pandemic or not is a challenging thing. You talked about, uh, AI and automation of minutes ago. How do you think those technologies can be leveraged by DevOps leaders to influence their successes and their ability to collaborate, maybe see eye to eye with the SRS? >>Yeah. Um, so th you're kind of too. So even for myself, as a leader of a, you know, 1500 people organization, there's a number of things I don't see right. On a daily basis. And, um, I think the, the, the, the technologies that we have at our disposal today from the AI are able to mind a lot of data and expose a lot of, uh, issues that's as leaders we may not be aware of. And some of the, some of these are pretty kind of easy to understand, right? We all think we're agile. And yet when you, when you start to understand, for instance, uh, what is the, what is the working progress right to during the sprint? Um, when you start to analyze the data you can detect, for instance, that maybe the teams are over committed, that there is too much work in progress. >>You can start to identify kind of, interdepencies either from a technology, from a people point of view, which were hidden, uh, you can start to understand maybe the change filler rates he's he is dragging. So I believe that there is a, there's a fundamental role to be played by the tools to, to expose again, these anti parents, to, to make these things visible to the teams, to be able to even compare teams. Right. One of the things that's, that's, uh, that's amazing is now we have access to tons of data, not just from a given customer, but across a large number of customers. And so we start to compare all of these teams kind of operate, and what's working, what's not working >>Thoughts on AI and automation as, as a facilitator of spiritual co-location. >>Yeah, absolutely. Absolutely. It's um, you know, th there's, uh, the problem we all face is the unknown, right? The, the law city, but volume variety of the data, uh, everyday we don't really necessarily completely appreciate what is the impact of our actions, right? And so, um, AI can really act as a safety net that enables us to, to understand what is the impact of our actions. Um, and so, yeah, in many ways, the ability to be informed in a timely matter to be able to interact with people on the basis of data, um, and collaborate on the data. And the actual matter, I think is, is a, is a very powerful enabler, uh, on, in that respect. I mean, I, I've seen, um, I've seen countless of times that, uh, for instance, at the SRE boundary, um, to basically show that we'll turn the quality attributes, so an incoming release, right. And exposing that to, uh, an operations person and a sorry person, and enabling that collaboration dialogue through data is a very, very powerful tool. >>Do you have any recommendations for how teams can use, you know, the SRE folks, the dev ops says can use AI and automation in the right ways to be successful rather than some ways that aren't going to be nonproductive. >>Yeah. So to me, the th there, there's a part of the question really is when, when we talk about data, there are there different ways you can use data, right? Um, so you can, you can do a lot of an analytics, predictive analytics. So I think there is a, there's a tendency, uh, to look at, let's say a, um, a specific KPI, like a, an availability KPI, or change filler rate, and to basically do a regression analysis and projecting all these things, going to happen in the future. To me, that that's, that's a, that's a bad approach. The reason why I fundamentally think it's a better approach is because we are systems. The way we develop software is, is a, is a non-leader kind of system, right? Software development is not linear nature. And so I think there's a D this is probably the worst approach is to actually focus on metrics on the other end. >>Um, if you, if you start to actually understand at a more granular level, what har, uh, which are the things which are contributing to this, right? So if you start to understand, for instance, that whenever maybe, you know, you affect a specific part of the application that translates into production issues. So we, we have, I've actually, uh, a customer who, uh, identified that, uh, over 50% of their unplanned outages were related to specific components in your architecture. And whenever these components were changed, this resulted in these plant outages. So if you start to be able to basically establish causality, right, cause an effect between kind of data across the last cycle. I think, I think this is the right way to, uh, to, to use AI. And so pharma to be, I think it's way more God could have a classification problem. What are the classes of problems that do exist and affect things as opposed to analytics, predictive, which I don't think is as powerful. >>So I mentioned in the beginning of our conversation, that just came off the biz ops manifesto. You're one of the authors of that. I want to get your thoughts on dev ops and biz ops overlapping, complimenting each other, what, from a, the biz ops perspective, what does it mean to the future of dev ops? >>Yeah, so, so it's interesting, right? If you think about DevOps, um, there's no felony document, right? Can we, we can refer to the Phoenix project. I mean, there are a set of documents which have been written, but in many ways, there's no clear definition of what dev ops is. Uh, if you go to the dev ops Institute today, you'll see that they are specific, um, trainings for instance, on value management on SRE. And so in many ways, the problem we have as an industry is that, um, there are set practices between agile dev ops, SRE Valley should management. I told, right. And we all basically talk about the same things, right. We all talk about essentially, um, accelerating in the meantime fee to feedback, but yet we don't have the common framework to talk about that. The other key thing is that we add to wait, uh, for, uh, for jeans, Jean Kim's Lascaux, um, to, uh, to really start to get into the business aspect, right? >>And for value stream mapping to start to emerge for us to start as an industry, right. It, to start to think about what is our connection with the business aspect, what's our purpose, right? And ultimately it's all about driving these business outcomes. And so to me, these ops is really about kind of, uh, putting a lens on this critical element that it's not business and it, that we in fact need to fuse business 19 that I need needs to transform itself to recognize that it's, it's this value generator, right. It's not a cost center. And so the relationship to me, it's more than BizOps provides kind of this Oliver or kind of framework, if you will. That set the context for what is the reason, uh, for it to exist. What's part of the core values and principles that it needs to embrace to, again, change from a cost center to a value center. And then we need to start to use this as a way to start to unify some of the, again, the core practices, whether it's agile, DevOps value, stream mapping SRE. Um, so, so I think over time, my hope is that we start to optimize a lot of our practices, language, um, and, uh, and cultural elements. >>Last question surgeon, the last few seconds we have here talking about this, the relation between biz ops and dev ops, um, what do you think as DevOps evolves? And as you talked to circle some of your insights, what should our audience keep their eyes on in the next six to 12 months? >>So to me, the key, the key, um, challenge for, for the industry is really around. So we were seeing a very rapid shift towards kind of, uh, product to product, right. Which we don't want to do is to recreate kind of these new silos, these hard silos. Um, so that, that's one of the big changes, uh, that I think we need to be, uh, to be really careful about, um, because it is ultimately, it is about culture. It's not about, uh, it's not about, um, kind of how we segment the work, right. And, uh, any true culture that we can overcome kind of silos. So back to, I guess, with Jeffrey's concept of, um, kind of the spiritual co-location, I think it's, it's really about that too. It's really about kind of, uh, uh, focusing on the business outcomes on kind of aligning on driving engagement across the teams, but, but not for create a, kind of a new set of silos, which instead of being vertical are going to be these horizontal products >>Crazy by surge that looking at culture as kind of a way of really, uh, uh, addressing and helping to, uh, re re reduce, replace challenges. We thank you so much for sharing your insights and your time at today's DevOps virtual forum. >>Thank you. Thanks for your time. >>I'll be right back >>From around the globe it's the cube with digital coverage of devops virtual forum brought to you by Broadcom. >>Welcome to Broadcom's DevOps virtual forum, I'm Lisa Martin, and I'm joined by another Martin, very socially distanced from me all the way coming from Birmingham, England is Glynn Martin, the head of QA transformation at BT. Glynn, it's great to have you on the program. Thank you, Lisa. I'm looking forward to it. As we said before, we went live to Martins for the person one in one segment. So this is going to be an interesting segment guys, what we're going to do is Glynn's going to give us a really kind of deep inside out view of devops from an evolution perspective. So Glynn, let's start. Transformation is at the heart of what you do. It's obviously been a very transformative year. How have the events of this year affected the >> transformation that you are still responsible for driving? Yeah. Thank you, Lisa. I mean, yeah, it has been a difficult year. >>Um, and although working for BT, which is a global telecommunications company, um, I'm relatively resilient, I suppose, as a, an industry, um, through COVID obviously still has been affected and has got its challenges. And if anything, it's actually caused us to accelerate our transformation journey. Um, you know, we had to do some great things during this time around, um, you know, in the UK for our emergency and, um, health workers give them unlimited data and for vulnerable people to support them. And that's spent that we've had to deliver changes quickly. Um, but what we want to be able to do is deliver those kinds of changes quickly, but sustainably for everything that we do, not just because there's an emergency. Um, so we were already on the kind of journey to agile, but ever more important now that we are, we are able to do those, that kind of work, do it more quickly. >>Um, and that it works because the, the implications of it not working is, can be terrible in terms of you know, we've been supporting testing centers,  new hospitals to treat COVID patients. So we need to get it right. And then therefore the coverage of what we do, the quality of what we do and how quickly we do it really has taken on a new scale and what was already a very competitive market within the telco industry within the UK. Um, you know, what I would say is that, you know, we are under pressure to deliver more value, but we have small cost challenges. We have to obviously, um, deal with the fact that, you know, COVID 19 has hit most industries kind of revenues and profits. So we've got this kind of paradox between having less costs, but having to deliver more value quicker and  to higher quality. So yeah, certainly the finances is, um, on our minds and that's why we need flexible models, cost models that allow us to kind of do growth, but we get that growth by showing that we're delivering value. Um, especially in these times when there are financial challenges on companies. So one of the things that I want to ask you about, I'm again, looking at DevOps from the inside >>Out and the evolution that you've seen, you talked about the speed of things really accelerating in this last nine months or so. When we think dev ops, we think speed. But one of the things I'd love to get your perspective on is we've talked about in a number of the segments that we've done for this event is cultural change. What are some of the things that you've seen there as, as needing to get, as you said, get things right, but done so quickly to support essential businesses, essential workers. How have you seen that cultural shift? >>Yeah, I think, you know, before test teams for themselves at this part of the software delivery cycle, um, and actually now really our customers are expecting that quality and to deliver for our customers what they want, quality has to be ingrained throughout the life cycle. Obviously, you know, there's lots of buzzwords like shift left. Um, how do we do shift left testing? Um, but for me, that's really instilling quality and given capabilities shared capabilities throughout the life cycle that drive automation, drive improvements. I always say that, you know, you're only as good as your lowest common denominator. And one thing that we were finding on our dev ops journey was that we  would be trying to do certain things quick, we had automated build, automated tests. But if we were taking a weeks to create test scripts, or we were taking weeks to manually craft data, and even then when we had taken so long to do it, that the coverage was quite poor and that led to lots of defects later on in the life cycle, or even in our production environment, we just couldn't afford to do that. >>And actually, focusing on continuous testing over the last nine to 12 months has really given us the ability to deliver quickly across the whole life cycle. And therefore actually go from doing a kind of semi agile kind of thing, where we did the user stories, we did a few of the kind of agile ceremonies, but we weren't really deploying any quicker into production because our stakeholders were scared that we didn't have the same control that we had when we had more waterfall releases. And, you know, when we didn't think of ourselves. So we've done a lot of work on every aspect, um, especially from a testing point of view, every aspect of every activity, rather than just looking at automated tests, you know, whether it is actually creating the test in the first place, whether it's doing security testing earlier in the lot and performance testing in the life cycle, et cetera. So, yeah,  it's been a real key thing that for CT, for us to drive DevOps, >>Talk to me a little bit about your team. What are some of the shifts in terms of expectations that you're experiencing and how your team interacts with the internal folks from pipeline through life cycle? >>Yeah, we've done a lot of work on this. Um, you know, there's a thing that I think people will probably call it a customer experience gap, and it reminds me of a Gilbert cartoon, where we start with the requirements here and you're almost like a Chinese whisper effects and what we deliver is completely different. So we think the testing team or the delivery teams, um, know in our teeth has done a great job. This is what it said in the acceptance criteria, but then our customers are saying, well, actually that's not working this isn't working and there's this kind of gap. Um, we had a great launch this year of agile requirements, it's one of the Broadcom tools. And that was the first time in, ever since I remember actually working within BT, I had customers saying to me, wow, you know, we want more of this. >>We want more projects to have extra requirements design on it because it allowed us to actually work with the business collaboratively. I mean, we talk about collaboration, but how do we actually, you know, do that and have something that both the business and technical people can understand. And we've actually been working with the business , using agile requirements designer to really look at what the requirements are, tease out requirements we hadn't even thought of and making sure that we've got high levels of test coverage. And what we actually deliver at the end of it, not only have we been able to generate tests more quickly, but we've got much higher test coverage and also can more smartly, using the kind of AI within the tool and then some of the other kinds of pipeline tools, actually deliver to choose the right tasks, and actually doing a risk based testing approach. So that's been a great launch this year, but just the start of many kinds of things that we're doing >>Well, what I hear in that, Glynn is a lot of positives that have come out of a very challenging situation. Talk to me about it. And I liked that perspective. This is a very challenging time for everybody in the world, but it sounds like from a collaboration perspective you're right, we talk about that a lot critical with devops. But those challenges there, you guys were able to overcome those pretty quickly. What other challenges did you face and figure out quickly enough to be able to pivot so fast? >>I mean, you talked about culture. You know, BT is like most companies  So it's very siloed. You know we're still trying to work to become closer as a company. So I think there's a lot of challenges around how would you integrate with other tools? How would you integrate with the various different technologies. And BT, we have 58 different IT stacks. That's not systems, that's stacks, all of those stacks can have hundreds of systems. And we're trying to, we've got a drive at the moment, a simplified program where we're trying to you know, reduce that number to 14 stacks. And even then there'll be complexity behind the scenes that we will be challenged more and more as we go forward. How do we actually highlight that to our users? And as an it organization, how do we make ourselves leaner, so that even when we've still got some of that legacy, and we'll never fully get rid of it and that's the kind of trade off that we have to make, how do we actually deal with that and hide that from our users and drive those programs, so we can, as I say, accelerate change,  reduce that kind of waste and that kind of legacy costs out of our business. You know, the other thing as well, I'm sure telecoms is probably no different to insurance or finance. When you take the number of products that we do, and then you combine them, the permutations are tens and hundreds of thousands of products. So we, as a business are trying to simplify, we are trying to do that in an agile way. >>And haven't tried to do agile in the proper way and really actually work at pace, really deliver value. So I think what we're looking more and more at the moment is actually  more value focused. Before we used to deliver changes sometimes into production. Someone had a great idea, or it was a great idea nine months ago or 12 months ago, but actually then we ended up deploying it and then we'd look at the users, the usage of that product or that application or whatever it is, and it's not being used for six months. So we haven't got, you know, the cost of the last 12 months. We certainly haven't gotten room for that kind of waste and, you know, for not really understanding the value of changes that we are doing. So I think that's the most important thing of the moment, it's really taking that waste out. You know, there's lots of focus on things like flow management, what bits of our process are actually taking too long. And we've started on that journey, but we've got a hell of a long way to go. But that involves looking at every aspect of the software delivery cycle. >> Going from, what 58 IT stacks down to 14 or whatever it's going to be, simplifying sounds magical to everybody. It's a big challenge. What are some of the core technology capabilities that you see really as kind of essential for enabling that with this new way that you're working? >>Yeah. I mean, I think we were started on a continuous testing journey, and I think that's just the start. I mean as I say, looking at every aspect of, you know, from a QA point of view is every aspect of what we do. And it's also looking at, you know, we've started to branch into more like AI, uh, AI ops and, you know, really the full life cycle. Um, and you know, that's just a stepping stone to, you know, I think autonomics is the way forward, right. You know, all of this kind of stuff that happens, um, you know, monitoring, uh, you know, watching the systems what's happening in production, how do we feed that back? How'd you get to a point where actually we think about change and then suddenly it's in production safely, or if it's not going to safety, it's automatically backing out. So, you know, it's a very, very long journey, but if we want to, you know, in a world where the pace is in ever-increasing and the demands for the team, and, you know, with the pressures on, at the moment where we're being asked to do things, uh, you know, more efficiently and as lean as possible, we need to be thinking about every part of the process and how we put the kind of stepping stones in place to lead us to a more automated kind of, um, you know, um, the future. >>Do you feel that that planned outcomes are starting to align with what's delivered, given this massive shift that you're experiencing? >>I think it's starting to, and I think, you know, as I say, as we look at more of a value based approach, um, and, um, you know, as I say, print, this was a kind of flow management. I think that that will become ever, uh, ever more important. So, um, I think it starting to people certainly realize that, you know, teams need to work together, you know, the kind of the cousin between business and it, especially as we go to more kind of SAS based solutions, low code solutions, you know, there's not such a gap anymore, actually, some of our business partners that expense to be much more tech savvy. Um, so I think, you know, this is what we have to kind of appreciate what is its role, how do we give the capabilities, um, become more of a centers of excellence rather than actually doing mounds amounts of work. And for me, and from a testing point of view, you know, mounds and mounds of testing, actually, how do we automate that? How do we actually generate that instead of, um, create it? I think that's the kind of challenge going forward. >>What are some, as we look forward, what are some of the things that you would like to see implemented or deployed in the next, say six to 12 months as we hopefully round a corner with this pandemic? >>Yeah, I think, um, you know, certainly for, for where we are as a company from a QA perspective, we are, um, you let's start in bits that we do well, you know, we've started creating, um, continuous delivery and DevOps pipelines. Um, there's still manual aspects of that. So, you know, certainly for me, I I've challenged my team with saying how do we do an automated journey? So if I put a requirement in JIRA or rally or wherever it is and why then click a button and, you know, with either zero touch for one such, then put that into production and have confidence that, that has been done safely and that it works and what happens if it doesn't work. So, you know, that's, that's the next, um, the next few months, that's what our concentration, um, is, is about. But it's also about decision-making, you know, how do you actually understand those value judgments? >>And I think there's lots of the things dev ops, AI ops, kind of that always ask aspects of business operations. I think it's about having the information in one place to make those kinds of decisions. How does it all try and tie it together? As I say, even still with kind of dev ops, we've still got elements within my company where we've got lots of different organizations doing some, doing similar kinds of things, but they're all kind of working in silos. So I think having AI ops as it comes more and more to the fore as we go to cloud, and that's what we need to, you know, we're still very early on in our cloud journey, you know, so we need to make sure the technologies work with cloud as well as you can have, um, legacy systems, but it's about bringing that all together and having a full, visible pipeline, um, that everybody can see and make decisions. >>You said the word confidence, which jumped out at me right away, because absolutely you've got to have be able to have confidence in what your team is delivering and how it's impacting the business and those customers. Last question then for you is how would you advise your peers in a similar situation to leverage technology automation, for example, dev ops, to be able to gain the confidence that they're making the right decisions for their business? >>I think the, the, the, the, the approach that we've taken actually is not started with technology. Um, we've actually taken a human centered design, uh, as a core principle of what we do, um, within the it part of BT. So by using human centered design, that means we talk to our customers, we understand their pain points, we map out their current processes. Um, and then when we mapped out what this process does, it also understand their aspirations as well, you know? Um, and where do they want to be in six months? You know, do they want it to be, um, more agile and, you know, or do they want to, you know, is, is this a part of their business that they want to do one better? We actually then looked at why that's not running well, and then see what, what solutions are out there. >>We've been lucky that, you know, with our partnership, with Broadcom within the payer line, lots of the tools and the PLA have directly answered some of the business's problems. But I think by having those conversations and actually engaging with the business, um, you know, especially if the business hold the purse strings, which in, in, uh, you know, in some companies include not as they do there is that kind of, you know, almost by understanding their, their pain points and then starting, this is how we can solve your problem. Um, is we've, we've tended to be much more successful than trying to impose something and say, well, here's the technology that they don't quite understand. It doesn't really understand how it kind of resonates with their problems. So I think that's the heart of it. It's really about, you know, getting, looking at the data, looking at the processes, looking at where the kind of waste is. >>And then actually then looking at the right solutions. Then, as I say, continuous testing is massive for us. We've also got a good relationship with Apple towards looking at visual AI. And actually there's a common theme through that. And I mean, AI is becoming more and more prevalent. And I know, you know, sometimes what is AI and people have kind of this semantics of, is it true AI or not, but it's certainly, you know, AI machine learning is becoming more and more prevalent in the way that we work. And it's allowing us to be much more effective, be quicker in what we do and be more accurate. And, you know, whether it's finding defects running the right tests or, um, you know, being able to anticipate problems before they're happening in a production environment. >>Well, thank you so much for giving us this sort of insight outlook at dev ops sharing the successes that you're having, taking those challenges, converting them to opportunities and forgiving folks who might be in your shoes, or maybe slightly behind advice enter. They appreciate it. We appreciate your time. >>Well, it's been an absolute pleasure, really. Thank you for inviting me. I have a extremely enjoyed it. So thank you ever so much. >>Excellent. Me too. I've learned a lot for Glenn Martin. I'm Lisa Martin. You're watching the cube >>Driving revenue today means getting better, more valuable software features into the hands of your customers. If you don't do it quickly, your competitors as well, but going faster without quality creates risks that can damage your brand destroy customer loyalty and cost millions to fix dev ops from Broadcom is a complete solution for balancing speed and risk, allowing you to accelerate the flow of value while minimizing the risk and severity of critical issues with Broadcom quality becomes integrated across the entire DevOps pipeline from planning to production, actionable insights, including our unique readiness score, provide a three 60 degree view of software quality giving you visibility into potential issues before they become disasters. Dev ops leaders can manage these risks with tools like Canary deployments tested on a small subset of users, or immediately roll back to limit the impact of defects for subsequent cycles. Dev ops from Broadcom makes innovation improvement easier with integrated planning and continuous testing tools that accelerate the flow of value product requirements are used to automatically generate tests to ensure complete quality coverage and tests are easily updated. >>As requirements change developers can perform unit testing without ever leaving their preferred environment, improving efficiency and productivity for the ultimate in shift left testing the platform also integrates virtual services and test data on demand. Eliminating two common roadblocks to fast and complete continuous testing. When software is ready for the CIC CD pipeline, only DevOps from Broadcom uses AI to prioritize the most critical and relevant tests dramatically improving feedback speed with no decrease in quality. This release is ready to go wherever you are in your DevOps journey. Broadcom helps maximize innovation velocity while managing risk. So you can deploy ideas into production faster and release with more confidence from around the globe. It's the queue with digital coverage of dev ops virtual forum brought to you by Broadcom. >>Hi guys. Welcome back. So we have discussed the current state and the near future state of dev ops and how it's going to evolve from three unique perspectives. In this last segment, we're going to open up the floor and see if we can come to a shared understanding of where dev ops needs to go in order to be successful next year. So our guests today are, you've seen them all before Jeffrey Hammond is here. The VP and principal analyst serving CIO is at Forester. We've also Serge Lucio, the GM of Broadcom's enterprise software division and Glenn Martin, the head of QA transformation at BT guys. Welcome back. Great to have you all three together >>To be here. >>All right. So we're very, we're all very socially distanced as we've talked about before. Great to have this conversation. So let's, let's start with one of the topics that we kicked off the forum with Jeff. We're going to start with you spiritual co-location that's a really interesting topic that we've we've uncovered, but how much of the challenge is truly cultural and what can we solve through technology? Jeff, we'll start with you then search then Glen Jeff, take it away. >>Yeah, I think fundamentally you can have all the technology in the world and if you don't make the right investments in the cultural practices in your development organization, you still won't be effective. Um, almost 10 years ago, I wrote a piece, um, where I did a bunch of research around what made high-performance teams, software delivery teams, high performance. And one of the things that came out as part of that was that these teams have a high level of autonomy. And that's one of the things that you see coming out of the agile manifesto. Let's take that to today where developers are on their own in their own offices. If you've got teams where the team itself had a high level of autonomy, um, and they know how to work, they can make decisions. They can move forward. They're not waiting for management to tell them what to do. >>And so what we have seen is that organizations that embraced autonomy, uh, and got their teams in the right place and their teams had the information that they needed to make the right decisions have actually been able to operate pretty well, even as they've been remote. And it's turned out to be things like, well, how do we actually push the software that we've created into production that would become the challenge is not, are we writing the right software? And that's why I think the term spiritual co-location is so important because even though we may be physically distant, we're on the same plane, we're connected from a, from, from a, a shared purpose. Um, you know, surgeon, I worked together a long, long time ago. So it's been what almost 15, 16 years since we were at the same place. And yet I would say there's probably still a certain level of spiritual co-location between us, uh, because of the shared purposes that we've had in the past and what we've seen in the industry. And that's a really powerful tool, uh, to build on. So what do tools play as part of that, to the extent that tools make information available, to build shared purpose on to the extent that they enable communication so that we can build that spiritual co-location to the extent that they reinforce the culture that we want to put in place, they can be incredibly valuable, especially when, when we don't have the luxury of physical locate physical co-location. Okay. That makes sense. >>It does. I shouldn't have introduced us. This last segment is we're all spiritually co-located or it's a surge, clearly you're still spiritually co located with jump. Talk to me about what your thoughts are about spiritual of co-location the cultural impact and how technology can move it forward. >>Yeah. So I think, well, I'm going to sound very similar to Jeff in that respect. I think, you know, it starts with kind of a shared purpose and the other understanding, Oh, individuals teams, uh, contributed to kind of a business outcome, what is our shared goal or shared vision? What's what is it we're trying to achieve collectively and keeping it kind of aligned to that? Um, and so, so it's really starts with that now, now the big challenge, always these over the last 20 years, especially in large organization, there's been specialization of roles and functions. And so we, we all that started to basically measure which we do, uh, on a daily basis using metrics, which oftentimes are completely disconnected from kind of a business outcome or purpose. We, we kind of reverted back to, okay, what is my database all the time? What is my cycle time? >>Right. And, and I think, you know, which we can do or where we really should be focused as an industry is to start to basically provide a lens or these different stakeholders to look at what they're doing in the context of kind of these business outcomes. So, um, you know, probably one of my, um, favorites experience was to actually weakness at one of a large financial institution. Um, you know, Tuesday Golder's unquote development and operations staring at the same data, right. Which was related to, you know, in calming changes, um, test execution results, you know, Coverity coverage, um, official liabilities and all the all ran. It could have a direction level links. And that's when you start to put these things in context and represent that to you in a way that these different stakeholders can, can look at from their different lens. And, uh, and it can start to basically communicate and, and understand have they joined our company to, uh, to, to that kind of common view or objective. >>And Glen, we talked a lot about transformation with you last time. What are your thoughts on spiritual colocation and the cultural part, the technology impact? >>Yeah, I mean, I agree with Jeffrey that, you know, um, the people and culture, the most important thing, actually, that's why it's really important when you're transforming to have partners who have the same vision as you, um, who, who you can work with, have the same end goal in mind. And w I've certainly found that with our, um, you know, continuing relationship with Broadcom, what it also does though, is although, you know, tools can accelerate what you're doing and can join consistency. You know, we've seen within simplify, which is BTS flagship transformation program, where we're trying to, as it can, it says simplify the number of systems stacks that we have, the number of products that we have actually at the moment, we've got different value streams within that program who have got organizational silos. We were trying to rewrite, rewrite the wheel, um, who are still doing things manually. >>So in order to try and bring that consistency, we need the right tools that actually are at an enterprise grade, which can be flexible to work with in BT, which is such a complex and very dev, uh, different environments, depending on what area of BT you're in, whether it's a consumer, whether it's a mobile area, whether it's large global or government organizations, you know, we found that we need tools that can, um, drive that consistency, but also flex to Greenfield brownfield kind of technologies as well. So it's really important that as I say, for a number of different aspects, that you have the right partner, um, to drive the right culture, I've got the same vision, but also who have the tool sets to help you accelerate. They can't do that on their own, but they can help accelerate what it is you're trying to do in it. >>And a really good example of that is we're trying to shift left, which is probably a, quite a bit of a buzz phrase in their kind of testing world at the moment. But, you know, I could talk about things like continuous delivery direct to when a ball comes tools and it has many different features to it, but very simply on its own, it allows us to give the visibility of what the teams are doing. And once we have that visibility, then we can talk to the teams, um, around, you know, could they be doing better component testing? Could they be using some virtualized services here or there? And that's not even the main purpose of continuous delivery director, but it's just a reason that tools themselves can just give greater visibility of have much more intuitive and insightful conversations with other teams and reduce those organizational silos. >>Thanks, Ben. So we'd kind of sum it up, autonomy collaboration tools that facilitate that. So let's talk now about metrics from your perspectives. What are the metrics that matter? Jeff, >>I'm going to go right back to what Glenn said about data that provides visibility that enables us to, to make decisions, um, with shared purpose. And so business value has to be one of the first things that we look at. Um, how do we assess whether we have built something that is valuable, you know, that could be sales revenue, it could be net promoter score. Uh, if you're not selling what you've built, it could even be what the level of reuse is within your organization or other teams picking up the services, uh, that you've created. Um, one of the things that I've begun to see organizations do is to align value streams with customer journeys and then to align teams with those value streams. So that's one of the ways that you get to a shared purpose, cause we're all trying to deliver around that customer journey, the value with it. >>And we're all measured on that. Um, there are flow metrics which are really important. How long does it take us to get a new feature out from the time that we conceive it to the time that we can run our first experiments with it? There are quality metrics, um, you know, some of the classics or maybe things like defect, density, or meantime to response. Um, one of my favorites came from a, um, a company called ultimate software where they looked at the ratio of defects found in production to defects found in pre production and their developers were in fact measured on that ratio. It told them that guess what quality is your job to not just the test, uh, departments, a group, the fourth level that I think is really important, uh, in, in the current, uh, situation that we're in is the level of engagement in your development organization. >>We used to joke that we measured this with the parking lot metric helpful was the parking lot at nine. And how full was it at five o'clock. I can't do that anymore since we're not physically co-located, but what you can do is you can look at how folks are delivering. You can look at your metrics in your SCM environment. You can look at, uh, the relative rates of churn. Uh, you can look at things like, well, are our developers delivering, uh, during longer periods earlier in the morning, later in the evening, are they delivering, uh, you know, on the weekends as well? Are those signs that we might be heading toward a burnout because folks are still running at sprint levels instead of marathon levels. Uh, so all of those in combination, uh, business value, uh, flow engagement in quality, I think form the backbone of any sort of, of metrics, uh, a program. >>The second thing that I think you need to look at is what are we going to do with the data and the philosophy behind the data is critical. Um, unfortunately I see organizations where they weaponize the data and that's completely the wrong way to look at it. What you need to do is you need to say, you need to say, how is this data helping us to identify the blockers? The things that aren't allowing us to provide the right context for people to do the right thing. And then what do we do to remove those blockers, uh, to make sure that we're giving these autonomous teams the context that they need to do their job, uh, in a way that creates the most value for the customers. >>Great advice stuff, Glenn, over to your metrics that matter to you that really make a big impact. And, and, and also how do you measure quality kind of following onto the advice that Jeff provided? >>That's some great advice. Actually, he talks about value. He talks about flow. Both of those things are very much on my mind at the moment. Um, but there was this, I listened to a speaker, uh, called me Kirsten a couple of months ago. It taught very much around how important flow management is and removing, you know, and using that to remove waste, to understand in terms of, you know, making software changes, um, what is it that's causing us to do it longer than we need to. So where are those areas where it takes long? So I think that's a very important thing for us. It's even more basic than that at the moment, we're on a journey from moving from kind of a waterfall to agile. Um, and the problem with moving from waterfall to agile is with waterfall, the, the business had a kind of comfort that, you know, everything was tested together and therefore it's safer. >>Um, and with agile, there's that kind of, you know, how do we make sure that, you know, if we're doing things quick and we're getting stuff out the door that we give that confidence, um, that that's ready to go, or if there's a risk that we're able to truly articulate what that risk is. So there's a bit about release confidence, um, and some of the metrics around that and how, how healthy those releases are, and actually saying, you know, we spend a lot of money, um, um, an investment setting up our teams, training our teams, are we actually seeing them deliver more quickly and are we actually seeing them deliver more value quickly? So yeah, those are the two main things for me at the moment, but I think it's also about, you know, generally bringing it all together, the dev ops, you know, we've got the kind of value ops AI ops, how do we actually bring that together to so we can make quick decisions and making sure that we are, um, delivering the biggest bang for our buck, absolutely biggest bang for the buck, surge, your thoughts. >>Yeah. So I think we all agree, right? It starts with business metrics, flow metrics. Um, these are kind of the most important metrics. And ultimately, I mean, one of the things that's very common across a highly functional teams is engagements, right? When, when you see a team that's highly functioning, that's agile, that practices DevOps every day, they are highly engaged. Um, that that's, that's definitely true. Now the, you know, back to, I think, uh, Jeff's point on weaponization of metrics. One of the key challenges we see is that, um, organizations traditionally have been kind of, uh, you know, setting up benchmarks, right? So what is a good cycle time? What is a good lead time? What is a good meantime to repair? The, the problem is that this is very contextual, right? It varies. It's going to vary quite a bit, depending on the nature of application and system. >>And so one of the things that we really need to evolve, um, as an industry is to understand that it's not so much about those flow metrics is about our, these four metrics ultimately contribute to the business metric to the business outcome. So that's one thing. The second aspect, I think that's oftentimes misunderstood is that, you know, when you have a bad cycle time or, or, or what you perceive as being a buy cycle time or better quality, the problem is oftentimes like all, do you go and explore why, right. What is the root cause of this? And I think one of the key challenges is that we tend to focus a lot of time on metrics and not on the eye type patterns, which are pretty common across the industry. Um, you know, if you look at, for instance, things like lead time, for instance, it's very common that, uh, organizational boundaries are going to be a key contributor to badly time. >>And so I think that there is, you know, the only the metrics there is, I think a lot of work that we need to do in terms of classifying, descend type patterns, um, you know, back to you, Jeff, I think you're one of the cool offers of waterscrumfall as a, as, as a key pattern, the industry or anti-spatter. Um, but waterscrumfall right is a key one, right? And you will detect that through kind of a defect arrival rates. That's where that looks like an S-curve. And so I think it's beyond kind of the, the metrics is what do you do with those metrics? >>Right? I'll tell you a search. One of the things that is really interesting to me in that space is I think those of us had been in industry for a long time. We know the anti-patterns cause we've seen them in our career maybe in multiple times. And one of the things that I think you could see tooling do is perhaps provide some notification of anti-patterns based on the telemetry that comes in. I think it would be a really interesting place to apply, uh, machine learning and reinforcement learning techniques. Um, so hopefully something that we'd see in the future with dev ops tools, because, you know, as a manager that, that, you know, may be only a 10 year veteran or 15 year veteran, you may be seeing these anti-patterns for the first time. And it would sure be nice to know what to do, uh, when they start to pop up, >>That would right. Insight, always helpful. All right, guys, I would like to get your final thoughts on this. The one thing that you believe our audience really needs to be on the lookout for and to put on our agendas for the next 12 months, Jeff will go back to you. Okay. >>I would say look for the opportunities that this disruption presents. And there are a couple that I see, first of all, uh, as we shift to remote central working, uh, we're unlocking new pools of talent, uh, we're, it's possible to implement, uh, more geographic diversity. So, so look to that as part of your strategy. Number two, look for new types of tools. We've seen a lot of interest in usage of low-code tools to very quickly develop applications. That's potentially part of a mainstream strategy as we go into 2021. Finally, make sure that you embrace this idea that you are supporting creative workers that agile and dev ops are the peanut butter and chocolate to support creative, uh, workers with algorithmic capabilities, >>Peanut butter and chocolate Glen, where do we go from there? What are, what's the one silver bullet that you think folks to be on the lookout for now? I, I certainly agree that, um, low, low code is, uh, next year. We'll see much more low code we'd already started going, moving towards a more of a SAS based world, but low code also. Um, I think as well for me, um, we've still got one foot in the kind of cow camp. Um, you know, we'll be fully trying to explore what that means going into the next year and exploiting the capabilities of cloud. But I think the last, um, the last thing for me is how do you really instill quality throughout the kind of, um, the, the life cycle, um, where, when I heard the word scrum fall, it kind of made me shut it because I know that's a problem. That's where we're at with some of our things at the moment we need to get beyond that. We need >>To be releasing, um, changes more frequently into production and actually being a bit more brave and having the confidence to actually do more testing in production and go straight to production itself. So expect to see much more of that next year. Um, yeah. Thank you. I haven't got any food analogies. Unfortunately we all need some peanut butter and chocolate. All right. It starts to take us home. That's what's that nugget you think everyone needs to have on their agendas? >>That's interesting. Right. So a couple of days ago we had kind of a latest state of the DevOps report, right? And if you read through the report, it's all about the lost city, but it's all about sweet. We still are receiving DevOps as being all about speed. And so to me, the key advice is in order to create kind of a spiritual collocation in order to foster engagement, we have to go back to what is it we're trying to do collectively. We have to go back to tie everything to the business outcome. And so for me, it's absolutely imperative for organizations to start to plot their value streams, to understand how they're delivering value into aligning everything they do from a metrics to deliver it, to flow to those metrics. And only with that, I think, are we going to be able to actually start to really start to align kind of all these roles across the organizations and drive, not just speed, but business outcomes, >>All about business outcomes. I think you guys, the three of you could write a book together. So I'll give you that as food for thought. Thank you all so much for joining me today and our guests. I think this was an incredibly valuable fruitful conversation, and we appreciate all of you taking the time to spiritually co-located with us today, guys. Thank you. Thank you, Lisa. Thank you. Thank you for Jeff Hammond serves Lucio and Glen Martin. I'm Lisa Martin. Thank you for watching the broad cops Broadcom dev ops virtual forum.

>> Narrator: From theCUBE studios in Palo Alto in Boston, connecting with our leaders all around the world. This is theCUBE conversation. >> Welcome back everybody. Jeff Frick here with theCUBE we are in our Palo Alto studios today. We're still getting through COVID, thankfully media was a necessary industry, so we've been able to come in and keep a small COVID crew, but we can still reach out to the community and through the magic of the internet and camera's on laptops, we can reach out and touch base with our friends. So we're excited to have somebody who's talking about and working on kind of the next big edge, the next big cutting thing going on in technology. And that's the internet of things you've heard about it the industrial Internet of Things. There's a lot of different words for it. But the foundation of it is this company it's Intel. We're happy to have joined us Bill Pearson. He is the Vice President of Internet of Things often said IoT for Intel, Bill, great to see you. >> Same Jeff. Nice to be here. >> Yeah, absolutely. So I just was teasing getting ready for this interview, doing a little homework and I saw you talking about Internet of Things in a 2015 interview, actually referencing a 2014 interview. So you've been at this for a while. So before we jump into where we are today, I wonder if you can share, you know, kind of a little bit of a perspective of what's happened over the last five or six years. >> I mean, I think data has really grown at a tremendous pace, which has changed the perception of what IoT is going to do for us. And the other thing that's been really interesting is the rise of AI. And of course we need it to be able to make sense of all that data. So, you know, one thing that's different is today where we're really focused on how do we take that data that is being produced at this rapid rate and really make sense of it so that people can get better business outcomes from that. >> Right, right. But the thing that's so interesting on the things part of the Internet of Things and even though people are things too, is that the scale and the pace of data that's coming off, kind of machine generated activity versus people generated is orders of magnitude higher in terms of the frequency, the variety, and all kind of your classic big data meme. So that's a very different challenge then, you know, kind of the growth of data that we had before and the types of data, 'cause it's really gone kind of exponential across every single vector. >> Absolutely. It has, I mean, we've seen estimates that data is going to increase by about five times as much as it is today, over the next, just a couple years. So it's exponential as you said. >> Right. The other thing that's happened is Cloud. And so, you know, kind of breaking the mold of the old mold roar, all the compute was either in your mini computer or data center or mainframe or on your laptop. Now, you know, with Cloud and instant connectivity, you know, it opens up a lot of different opportunities. So now we're coming to the edge and Internet of Things. So when you look at kind of edge in Internet of Things, kind of now folding into this ecosystem, you know, what are some of the tremendous benefits that we can get by leveraging those things that we couldn't with kind of the old infrastructure and our old way kind of gathering and storing and acting on data? >> Yeah. So one of the things we're doing today with the edge is really bringing the compute much closer to where all the data is being generated. So these sensors and devices are generating tons and tons of data and for a variety of reasons, we can't send it somewhere else to get processed. You know, there may be latency requirements for that control loop that you're running in your factory or there's bandwidth constraints that you have, or there's just security or privacy reasons to keep it onsite. And so you've got to process a lot of this data onsite and maybe some estimates or maybe half of the data is going to remain onsite here. And when you look at that, you know, that's where you need compute. And so the edge is all about taking compute, bringing it to where the data is, and then being able to use the intelligence, the AI and analytics to make sense of that data and take actions in real time. >> Right, right. But it's a complicated situation, right? 'Cause depending on where that edge is, what the device is, does it have power? Does it not have power? Does it have good connectivity? Does it not have good connectivity? Does it have even the ability to run those types of algorithms or does it have to send it to some interim step, even if it doesn't have, you know, kind of the ability to send it all the way back to the Cloud or all the way back to the data center for latency. So as you kind of slice and dice all these pieces of the chain, where do you see the great opportunity for Intel, where's a good kind of sweet spot where you can start to bring in some compute horsepower and you can start to bring in some algorithmic processing and actually do things between just the itty-bitty sensor at the itty-bitty end of the chain versus the data center that's way, way upstream and far, far away. >> Yeah. Our business is really high performance compute and it's this idea of taking all of these workloads and bringing them in to this high performance compute to be able to run multiple software defined workloads on single boxes, to be able to then process and analyze and store all that data that's being created at the edge, do it in a high performance way. And whether that's a retail smart shelf, for example, that we can do realtime inventory on that shelf, as things are coming and going, or whether it's a factory and somebody's doing, you know,real time defect detection of something moving across their textile line. So all of that comes down to being able to have the compute horsepower, to make sense of the data and do something with it. >> Right, right. So you wouldn't necessarily like in your shelf example that the compute might be done there at the local store or some aggregation point beyond just that actual, you know, kind of sensor that's underneath that one box of tide, if you will. >> Absolutely. Yeah, you could have that on-prem, a big box that does multiple shelves, for example. >> Okay, great. So there's a great example and you guys have the software development kit, you have a lot of resources for developers and in one of the case studies that I just wanted to highlight before we jump into the dev side was I think Audi was the customer. And it really illustrates a point that we talked about a lot in kind of the big data meme, which is, you know, people used to take action on a sample of data after the fact. And I think this case here we're talking about running 1,000 cars a day through this factory, they're doing so many welds, 5 million welds a day, and they would pull one at the end of the day, sample a couple welds and did we have a good day or not? Versus what they're doing now with your technology is actually testing each and every weld as it's being welded, based on data that's coming off the welding machine and they're inspecting every single weld. So I just love you've been at this for a long time. When you talk to customers about what is possible from a business point of view, when you go from after the fact with a sample of data, to in real time with all the data, how that completely changes your view and ability to react to your business. >> Yeah. I mean, it makes people be able to make better decisions in real time. You know, as you've got cameras on things like textile manufacturers or footwear manufacturers, or even these realtime inventory examples you mentioned, people are going to be able to make and can make decisions in real time about how to stock that shelf, what to order about what to pull off the line, am I getting a good product or not? And this has really changed, as you said, we don't have to go back and sample anymore. You can tell right now as that part is passing through your manufacturing line, or as that item is sitting on your shelf, what's happening to it. It's really incredible. >> So let's talk about developers. So you've got a lot of resources available for developers and everyone knows Intel obviously historically in PCs and data centers. And you would do what they call design wins back when I was there, many moons ago, right? You try to get a design win and then, you know, they're going to put your microprocessors and a bunch of other components in a device. When you're trying to work with, kind of Cutting Edge Developers in kind of new fields and new areas, this feels like a much more direct touch to the actual people building the applications than the people that are really just designing the systems of which Intel becomes a core part of. I wonder if you could talk about, you know, the role developers and really Intel's outreach to developers and how you're trying to help them, you know, kind of move forward in this new crazy world. >> Yeah, developers are essential to our business. They're essential to IoT. Developers, as you said, create the applications that are going to really make the business possible. And so we know the value of developers and want to make sure that they have the tools and resources that they need to use our products most effectively. We've done some things around OpenVINO toolkit as an example, to really try and simplify, democratize AI application so that more developers can take advantage of this and, you know, take the ambitions that they have to do something really interesting for their business, and then go put it into action. And the whole, you know, our whole purpose is making sure we can actually accomplish that. >> Right. So let's talk about OPenVINO. It's an interesting topic. So I actually found out what OpeVINO means, Open Visual Inference and Neural Optimization toolkit,. So it's a lot about computer vision. So I will, you know, and computer vision is an interesting early AI application that I think a lot of people are familiar with through Google photos or other things where, you know, suddenly they're putting together little or a highlight movies for you, or they're pulling together all the photos of a particular person or a particular place. So the computer vision is pretty interesting. Inference is a special subset of AI. So I wonder, you know, you guys are way behind OpenVINO. Where do you see the opportunities in visualization? What are some of the instances that you're seeing with the developers out there doing innovative things around computer vision? >> Yeah, there's a whole variety of used cases with computer vision. You know, one that we talked about earlier here was looking at defect detection. There's a company that we work with that has a 360 degree view. They use cameras all around their manufacturing line. And from there, they didn't know what a good part looks like and using inference and OpenVINO, they can tell when a bad part goes through or there's a defect in their line and they can go and pull that and make corrections as needed. We've also seen, you know, use cases like smart shopping, where there's a point of sale fraud detection. We call it, you know, is the item being scanned the same as the item that is actually going through the line. And so we can be much smarter about understanding retail. One example that I saw was a customer who was trying to detect if it was a vodka or potatoes that was being scanned in an automated checkout system. And again, using cameras and OpenVINO, they can tell the difference. >> We haven't talked about a computer testing yet. We're still sticking with computer vision and the natural language processing. I know one of the areas you're interested in and it's going to only increase in importance is education. Especially with what's going on, I keep waiting for someone to start rolling out some national, you know, best practice education courses for kindergartens and third graders and sixth graders. And you know, all these poor teachers that are learning to teach on the fly from home, you guys are doing a lot of work in education. I wonder if you can share, I think your work doing some work with Udacity. What are you doing? Where do you see the opportunity to apply some of this AI and IoT in education? >> Yeah, we launched the Nanodegree with Udacity, and it's all about OpenVINO and Edge AI and the idea is, again, get more developers educated on this technology, take a leader like your Udacity, partner with them to make the coursework available and get more developers understanding using and building things using Edge AI. And so we partnered with them as part of their million developer goal. We're trying to get as many developers as possible through that. >> Okay. And I would be remiss if we talked about IoT and I didn't throw 5G into the conversation. So 5G is a really big deal. I know Intel has put a ton of resources behind it and have been talking about it for a long, long time. You know, I think the huge value in 5G is a lot around IoT as opposed to my handset going faster, which is funny that they're actually releasing 5G handsets out there. But when you look at 5G combined with the other capabilities in IoT, again, how do you see 5G being this kind of step function in ability to do real time analysis and make real time business decisions? >> Well, I think it brings more connectivity certainly and bandwidth and reduces latency. But the cool thing about it is when you look at the applications of it, you know, we talked about factories. A lot of those factors may want to have a private 5G networks that are running inside that factory, running all the machines or robots or things in there. And so, you know, it brings capabilities that actually make a difference in the world of IoT and the things that developers are trying to build. >> That's great. So before I let you go, you've been at this for a while. You've been at Intel for a while. You've seen a lot of big sweeping changes, kind of come through the industry, you know, as you sit back with a little bit of perspective, and it's funny, even IoT, like you said, you've been talking about it for five years and 5G we've been been waiting for it, but the waves keep coming, right? That's kind of the fun of being in this business. As you sit there where you are today, you know, kind of looking forward the next couple of years, couple of four or five years, you know, what has just surprised you beyond compare and what are you still kind of surprised that's it's still a little bit lagging that you would have expected to see a little bit more progress at this point. >> You know, to me the incredible thing about the computing industry is just the insatiable demand that the world has for compute. It seems like we always come up with, our customers always come up with more and more uses for this compute power. You know, as we've talked about data and the exponential growth of data and now we need to process and analyze and store that data. It's impressive to see developers just constantly thinking about new ways to apply their craft and, you know, new ways to use all that available computing power. And, you know, I'm delighted 'cause I've been at this for a while, as you said, and I just see this continuing to go far as far as the eye can see. >> Yeah, yeah. I think you're right. There's no shortage of opportunity. I mean, the data explosion is kind of funny. The data has always been there, we just weren't keeping track of it before. And the other thing that as I look at Jira, Internet of Things, kind of toolkit, you guys have such a broad portfolio now where a lot of times people think of Intel pretty much as a CPU company, but as you mentioned, you got to FPGAs and VPUs and Vision Solutions, stretch applications Intel has really done a good job in terms of broadening the portfolio to go after, you know, kind of this disparate or kind of sharding, if you will, of all these different types of computer applications have very different demands in terms of power and bandwidth and crunching utilization to technical (indistinct). >> Yeah. Absolutely the various computer architectures really just to help our customers with the needs, whether it's high power or low performance, a mixture of both, being able to use all of those heterogeneous architectures with a tool like OpenVINO, so you can program once, right once and then run your application across any of those architectures, help simplify the life of our developers, but also gives them the compute performance, the way that they need it. >> Alright Bill, well keep at it. Thank you for all your hard work. And hopefully it won't be five years before we're checking in to see how far this IoT thing is going. >> Hopefully not, thanks Jeff. >> Alright Bill. Thanks a lot. He's bill, I'm Jeff. You're watching theCUBE. Thanks for watching, we'll see you next time. (upbeat music)

>> Narrator: From theCUBE Studios in Palo Alto in Boston, connecting with thought leaders around the globe, these are Cloud Native Insights. >> Hi, I'm Stu Miniman the host of cloud native insights. And when we started this weekly program, we look at Cloud Native and you know, what does that mean? And of course, one of the most important topics in IT coming into 2020 was security. And once the global pandemic hit, security went from the top issue to oh my gosh, it's even more important. I've said a few times on the program while most people are working from home, it did not mean that the bad actors went home, we've actually seen an increase in the need for security. So really happy to be able to dig in and talk about what is Cloud Native security, and what should that mean to users? And to help me dig into this important topic, happy to welcome back to the program one of our CUBE alumni Dan Hubbard, he is the CEO of Lacework. Dan thanks so much for joining us. >> Thanks Stu. Happy to be here. >> Alright, so we don't want to argue too much on the Cloud Native term, I agree with you and your team. It's a term that like cloud before, it doesn't necessarily have a lot of meaning. But when we talk about modernization, we talked about customers leveraging the opportunity in innovation and cloud security of course is super important. You know most of us probably remember back, you go back a few years and it's like, "Oh well I adopt cloud. "It's secure, right? "I mean, it should just be built into my platform. "And I should have to think about that." Well, I don't think there's anybody out there at least hopefully there's not anybody out there that thinks that anything that I go to will just be inherently fully secure. So give us a little bit if you would, you know where you see us here in 2020 security's a complex landscape. What are you seeing? >> Yeah, so you know a lot of people as you said, used to talk about what's called the shared responsibility model, which was the cloud provider is responsible for a bunch of things. Like the physical access to the data center, the network, the hypervisor and you know that the core file system and operating system and then you're responsible for everything else that you could configure. But there's something that's not talked about as much. And that's kind of the shared irresponsibility model that's happening within companies where developers are saying they're not responsible for security saying that they're moving too fast. And so what we are seeing is that you know, as people migrate to the cloud or of course are born in the cloud, this notion of DevSecOps, or you know SecDevOps whatever you want to call it, is really about the architecture and the organization. It's not just about technology, and it's not just about people. And it's more about layer seven and eight, than it is about layer one to three. And so there's a bunch of trends that we're seeing in successful companies and customers and prospects will be seeing the market around how do they get to that level of cooperation between the security and the developers in the operation teams? >> Yeah Dan, first of all fully agree with what you're saying. I know when I go to like serverless.com they've got everybody chanting that security is everyone's responsibility. You know I think back to DevOps as a trend, when I read the Phoenix project it was, oh hey, the security is not something that you do bolt on, we're looking at after it's something that you need to shift into everyone thinking about it. Security is just going to be baked in along the process all the way. So the DevOps fail us when it comes to security, why do we need DevSecOps? You know why are you know as you say seven and eight the you know, political and organizational challenges still so much of an issue you know, decades into this discussion? >> Yeah. You know I think there's a few moving parts here and kind of post COVID is even more interesting is that companies have incredibly strategic initiatives to build applications that are core to their business. And in post COVID it's almost existential to their business. If you think of you know, markets like retail and hospitality and restaurants you know, they have to figure out how to digitize and how to deliver their business without potentially physical you know, access to two locations. So as that speed has happened, some of the safety has been left behind. And it's easy to say you have to kind of you know, one of our mantras is to run with speed and safety. But it's kind of hard to run with scissors you know, and be safe at the same time. So some of it is just speed. And the other is that unfortunately, the security people in many ways and the security products and a lot of the security solutions that are out there, the incumbents if you will, are trying to deliver their current solution in a cloud way. So they're doing sometimes it's called Cloud built or you know what I call Cloud washing and they're delivering a system that's not applicable to the modern infrastructure in the modern way that developers are building. So then you have a clash between the teams of like, "Hey I want to do this." And then I'd be like, "No you can't do that get out of our way. "This is strategic to the business." So a lot of it has just been you know, kind of combination of all those factors. >> Alright so Dan, we'll go back to Cloud Native security, you talked about sometimes people are Cloud washing, or they're just taking what they had putting it in the cloud. Sometimes it's just, oh hey we've got a SaaS model on this. Other times I hear cloud native security, and it just means hey I've got some hooks into Containers or Kubernetes. What does modern security look like? Help us understand a little bit. You mentioned some of the you know, legacy vendors what they're doing. I see lots of new security startups, some in you know specifically in that, you know, Kubernetes space. There's already been some acquisitions there. So you know, what do you see out there? You know what's good, what's bad in the trends that you're seeing? >> Yeah so I think the one thing that we really believe is that this is such a large problem that you have to be 100% focused on it. You know if you're doing this, you know, securing your infrastructure and securing your modern applications, and doing other parts of the business whether it's you know securing the endpoints of the laptops of the company and the firewall and authentication and all kinds of other things you have competing interests. So focus is pretty key. And it's obviously a very large addressable problem. What the market is telling us is a few things. The first one is that automation is critical. They may not have as many people to solve the problem. And the problem set is moving at such a scale that it's very, very hard to keep up. So a lot of people ask me you know, what do I worry about? You know, how do I stay awake at night? Or how do I get to sleep? And really the things I'm worried most about in the way where I spend most of my time on the product side is about how fast are builders building? Not necessarily about the bad guys. Now the bad guys are coming and they're doing all kinds of innovative and interesting things. But usually it starts off with the good guys and how they're deploying and how they're building. And you know, the cloud providers literally are releasing API's and new acronyms almost weekly it seems. So like new technology is being created such a scale. So automation the ability to adapt to that is one key message that we hear from the customers. The other is that it has to solve or go across multiple categories. So although things like Kubernetes and Containers are very popular today. The cloud security tackle and challenges is much more complex than that. You've got infrastructure as code, you've got server lists, you've got kind of fragmented workloads, whether some are Containers, some are VMs, maybe some are armies and then some are Kubernetes. So you've got a very fragmented world out there, and all of it needs to be secured. And then the last one is probably the most consistent theme we're hearing is that as DevOps becomes involved, because they know the application and the stack much better than security, it has to fit into your modern workflow of DevOps. So that means you know, deep integrations into Jira and Slack and PagerDuty and New Relic and Datadog are a lot more important in integrating to your you know, Palo Alto firewall and your Cisco IDs system and your endpoint you know antivirus. So those are the real key trends that we're seeing from the customers. >> Yeah Dan, you bring up a really important point, leveraging automation. I'm wondering what you're hearing from customers, because there definitely is a little bit of concern, especially if you take something like security and say, okay well, automation. Is that something that I'm just going to let the system do it? Or is it giving me to getting me to a certain point that then a human makes the final decision and enacts what's going to happen there? Where are we along that journey? >> Yeah, so I think of automation in two lenses. The first lens is efficacy, which is you know do I have to write rules? And do I have to tune train and alter the system over time? Or can it do that on my behalf? Or is there a combination of both? So the notion of people writing rules and building rules is very, very hard in this world because things are moving so quickly. You know, what is the KMS you know threat surface? The threat attacks are just changing. And typically what happens when you write rules is they're either too narrow and you messed up or they're too broad you just get way too much noise. So there's automating the efficacy of the system. That's one that's really critical. The other one that is becoming more important is in the past it was called enforcement. And this is how do I automate a response to your efficacy. And in this scenario it were very, very early days. Some vendors have come out and said you know, we can do full remediation and blocking. And typically what happens is the DevOps team kind of gives the Heisman to the security team it says, "No, you're not doing that." You know this is my production servers, and my infrastructure that's you know running our business, you can't block anything without us knowing about it. So I think we're really early. I believe that you know we're going to move to a world that's more about orchestration and automation, where there's a set of parameters where you can orchestrate certain things or maybe an ops assist mode. You know for example, we have some customers that will send our alerts to Slack, then they have a Slack bot and they say, "Okay, is it okay that Bob just opened "an S3 bucket in this region, yes or no?" No, and then it runs a serverless function and closes it. So there's kind of a what we call driver assist mode versus you know full you know, no one behind the steering wheel today. But I think it's going to mature over time. >> Yeah, Dan one of the other big challenges customer has is that their environments are even more fragmented than they would in the past. So often they're leveraging multiple cloud providers, multiple SaaS providers then they have their hosting providers. And security is something that I need to have holistically across these environments but not have to worry about okay, do I have the skill set and understanding between those environments? Hopefully you know that's something you see out there and want to understand, you know how the security industry in general and maybe Lacework specifically is helping customers, get their arms a little bit more around that multi cloud challenge if you will? >> Yeah. So I totally agree things are you know, I think we have this Silicon Valley, West Coast bias that the world is all you know, great. And it says to utopia Kubernetes, modern infrastructure, everything runs up and down, and it's all you know super easy. The reality is much different. Even in the most sophisticated sets of infrastructure in the most sophisticated customers are very fragmented and diverse. The other challenge that security runs into is security in the past a lot of traditional security mindsets are all about point in time. And they're really all about inventory. So you know, I know used to be able to ask, you know a security person, how many servers do you have? Where are they? What are they doing this? They say, "Oh, you know we have 10 racks with 42 servers in each rack. "And here's our IP addresses." Nowadays, the answer is kind of like, "I don't know what time is it you know, "how busy is a service?" It's very ephemeral. So you have to have a system which can adapt with the ephemeral nature of everything. So you know in the past it was really difficult to spin up, say 10,000 servers in a Asia data center for four hours to do research you know. Security probably know if that's happening, you know they would know through a number of different ways could make big change control window would be really hard they have to ship the units, they bake them in you know, et cetera. Nowadays that's like three lines of code. So the security people have to know and get visibility into the changes and have an engine which can determine those changes and what the risk profile of those in near real time. >> Yeah it's the what we've seen is the monitoring companies out there now talking all about observability. Its real time, it's streamings. You know it reminds me of you know my physics. So you know Heisenberg's uncertainty principle when you try to measure something, you already can't because it's already changed. So what does that mean-- >> Dan: Yeah. >> You know what does security look like in my you know, real time serverless ever changing world? You know, how is it that we are going to be able to stay secure? >> Yeah, so I think there are some really positive trends. The first one is that this is kind of a reboot. So this is kind of a restart. You know there are things we've learned in the past that we can bring forward but it's also an opportunity to kind of clean the slate and think about how we can rebuild the infrastructure. The first kind of key one is that over time security in the traditional data center started understanding less and less about the application over time, what they did was they built this big fortress around it, some called it defense in depth you know, the Security Onion whatever you want to call it you know, the M&M'S. But they were really lacking in the understanding of the application. So now security really has to understand the application because that's the core of what's important. And that allows them to be smarter about what are the changes in their environment, and if those are good, bad or indifferent. The other thing that I think is interesting is that compliance was kind of a dirty word that no one really wanted to talk about. It was kind of this boring thing or auditors would show up once every six months go through a very complex checklist and say you're okay. Now compliance is actually very sophisticated. And the ability to look at your configuration in near real time and understand if you are compliant or following best practices is real. And we do that for our customers all the time. You know we can tell them how they're doing against the compliance standard within a you know, a minute timeframe. And we can tell that they're drifting in and out of that. And the last one and the one that I think most are excited about is really the journey towards least privileges and minimizing the scope of your attack surface within your developers and their access in your infrastructure. Now it's... We're pretty far from there, it's an easy thing to say it's a pretty hard thing to do. But getting towards and driving towards that journey of least privilege I think is where most people are looking to go. >> Alright Dan, I want to go back to something that we talked about early in the conversation, that relationship with the cloud providers themselves, so you know talking AWS, Azure, Google Cloud and the like. How should customers be thinking about how they manage security, dealing with them dealing with companies like Lacework and the ecosystem you mentioned in companies like Datadog and the New Relic? You know how do they sort through and manage how they can maintain those relationships? >> So there's kind of the layer eight relationships, of course which are starting you know in particular with the cloud providers, it's a lot more about bottoms up relationships and very technical understanding of product and features, than it is about being on the golf course, and you know eating steak dinners. And that's very different you know, security and buying IT infrastructure was very relationship driven in the past. Now you really especially with SaaS and subscriptions, you're really proving out your technology every day. You know I say kind of trust is built on consistent positive results over time. So you really have to have trust within your solution and within that service and that trust is built on obviously a lot of that go to market business side. But more often than not it's now being built on the ability for that solution to get better over time because it's a subscription. You know how do you deliver more features and increase value to the customer as you do more things over time? So that's really, really important. The other one is like, how do I integrate the technology together? And I believe it's more important for us to integrate our stack with the cloud provider with the adjacent spaces like APM and metrics and monitoring and with open source, because open source really is a core component to this. So how do we have the API's and integrations and the hooks and the visibility into all of those is really, really important for our customers in the market? >> Well Dan as I said at the beginning, security is such an important topic to everyone out there. You know we've seen from practitioners we talked to for the last few years not only is it a top issue it's a board level discussion for pretty much every company out there. So I want to give you the final word as to in today's you know modern era, what advice do you give to users out there to make sure that they are staying as secure as possible? >> Yeah so you know first and foremost, people often say, "Hey you know, when we build our business, "you know, it'd be a good problem to start have to worry "about customers and you know, "all kinds of people using the service. "And you know, we'll worry about security then." And it's easy lip service to say start it as early as possible. The reality is sometimes it's hard to do that. You've got all kinds of competing interests, you're trying to build a business and an application and everything else depending obviously, the maturity of your organization. I would say that this is a great time to kind of crawl, walk, run. And you don't have to think about it. If you're building in the cloud you don't have to think of the end game you know right away, you can kind of stair step into that. So you know my suggestion to people that are moving into the cloud is really think about compliance and configuration best practices first and visibility, and then start thinking of the more complex things like triage alerts and how does that fit into my workflow? How do I look at breaches down the line? Now for the more mature orgs that are taking, you know an application or a new application or Stack and just dropping it in, those are the ones that should really think about how do I fit security into this new world order? And how do I make it as part of the design process? And it's not about how do I take my existing security stack and move it over? That's like taking, you know a centralized application moving to the cloud and calling it cloud. You know if you're going to build in the cloud, you have to secure it the same way that you're building it in a modern way. So really think about you know, modern, you know new generation vendors and solutions and a combination of kind of your provider, maybe some open source and then a service, of course like Lacework. >> Alright well Dan Hubbard, thank you so much for helping us dig into this important topic Cloud Native security, pleasure talking with you. >> Thank you. Have a great day. >> And I'm Stu Miniman your hosts for Cloud Native Insights and looking forward to hearing more of your Cloud Native Insights in the future. (upbeat music)

>> Narrator: From theCUBE studios in Palo Alto in Boston, connecting with all leaders all around the world, this is theCUBE conversation. >> Hi, everybody, this is Dave Vellante, and welcome to this episode of Startup Insights. Andrew Lau is here with me, he's the co founder and CEO of a relatively new company called Jellyfish. They focus on engineering management, which is kind of a new space that we want to present to you, Andrew, great to see you, thanks for coming on. >> Hey, Dave, thanks for having me on. >> So when I see co founder and a title I always ask why did you start a company? What's your Why? >> So the three co founders myself, Dave Gourley, and Phil Braden,we actually met geez more than 20 years ago, at a company called Endeca. We had the chance to kind of bring the proverbial band back together, just 'cause rare is the chance to work with great people. And for us, Jellyfish really was coming out of our own experiences. All three of us grew our careers running big engineering teams big product teams. We realized how hard it was to really lead those teams and connect them to the business at scale. And that's the problem we just got together to solve. >> Yeah, so interesting right, and Endeca, another East Coast company, great exit, brought by Oracle. And of course, when you say in Endeca, I always think okay, Jellyfish, you guys in the search business, but you're not in engineering management. What is engineering management? And, you know, how does it relate to some of your past experiences? >> Great question, well, so engineering management engineering management sector or management platform, as we talked about, really comes down to how we can facilitate the tools to make leading big teams easier, right, we've realized that as you get to larger teams, teams bigger than 50, bigger than 100 engineers, it's really hard to understand what the team is doing really hard to make sure that they're working their best in making sure they're pointed in the right direction. And even Furthermore, how to connect that to the business and how to make sure the business successful after the team dies. And as for us, is really around making tools and processes available that they really help accelerate the act of leading big teams. >> So when you think about it, I mean, it's really the problem you're solving is visibility, kind of what's going on in engineering and providing metrics is that a sort of a fair, high level? >> I think it's a perfect high level statement. When we actually got together and talked about the problem space we all saw as we lead these big teams. We quickly drew an analogy to you know, when I started my career in the late 90s, there was a time before Salesforce and CRM were pervasive, right And so really quickly, we drew a quick and easy analogy that said, like, hey, why isn't there a Salesforce for engineering? That is, why isn't it that same leadership and executive visibility into how a team is progressing and to make sure it's aligned with the business? >> Well, when you think about it, right, Salesforce, what 1999 we had the first Salesforce clouding before the real cloud hit, you know, we certainly have marketing clouds now capital Management clouds, customer service management clouds, why not an engineering cloud right? >> I think it's probably you follow that map there, I think we've seen the last 20 years, that clouds actually kind of progressed from sales to marketing to success to HR, as you pointed out, I think the last bastion in organizations is the engineering team right? It just so happens right now that engineering teams probably are the most strategic, if not the most expensive team in most companies roster. And so really, providing visibility is really necessary at this time. >> So I don't run engineering it's Silicon Angle by my co-CEO and partner John furrier does. But I'm always like asking him like, hey, what are you guys working on? What are the deliverables? What am I going to get it and when? how we do it on quality, and so forth and so I think just scanning your website, reading some of your blogs, these are some of the things that you really focus on. You wrote a five part series, I think, I don't know if I'm dying to see part five but four parts are out. I want to dig into some of those, I think they're in, I think you called it, you know, five things that you should present to the board. >> Yeah, yeah Like, you know, a great question around like, there's a series were four or five in two, I'm talking about what are five slides that heads of engineering should be showing to their boards or even their executive management teams? You know, early on in the process, I'm aware, before we actually started the company, we did a ton of research, talking with leaders at scale, trying to ask them, like, how do you manage your team? How do you connect into the business? And out of those conversations fell, you know, asking folks like, well, what slides do you show your board meeting? And and the answer was, there wasn't ubiquitous answer. People were looking for answers and so we really synthesized a number of different leaders that we thought were really successful in this world, and really put together this series to talk about like, what are the metrics that people should be measuring? >> Well, let's talk more about some of those metrics so you know, I mentioned so what am I going to get? When what are some of the other key things that people are focused on? I mean, obviously, quality, where I'm spending my money. But what are some of the ones that you're seeing, aligning with business and really driving business outcomes? >> So okay, so part one I think you talked about right which is, you know, what are you getting? What got shipped out the door? what's coming down the pipe? I think job one for a leader of engineering and Product is to talk about what's coming out, right in the same way that sales job is actually hit a revenue number and talk about the pipeline coming down the way it's an engineers job to actually ship new product that you can sell your users can engage with so that's definitely slide one. Slide two, you already alluded to here two is about quality, right? I think if you're shipping product that actually can't hold quality in the eyes of your customers, it won't last very long. So I think it's really important to show command of quality and actually show metrics that actually measure quality over time in the lens of your customer. Slide three, I think we're really talking about alignment. Making sure that your team is spending your dollars, time and effort in the right way that actually aligns with what the business wants to right? So examples might be, if you're a company that splits time between enterprise and SMB efforts, well, making sure that the features the team is working on actually aligned to the strategy of the company, right? And you can't do that if you don't measure that. And then slide four is really around capturing broad level productivity. But is the team healthy moving forward? And then the last of which is really the preview coming up for the next segment here is going to be about really around the team in hiring right. How is the team holding up? How's the morale? And how's the actual hiring pipeline looking and ramp? All right in terms of new employees right, it's really the people side of the engineering leadership. >> Cool, thanks for teasing that a little bit. I'm glad to hear it's not just about productivity, 'cause there are tools out there that can measure developer productivity, but seems like you're taking a broader approach and building a platform to really take a more end to end sort of lifecycle view. >> Yeah, I think we really look about, think about it as look productive is really important. I think it's necessary but not sufficient. You can talk about a boat, you can roll faster. But if you're rolling the boat in the wrong direction, who cares? So it's as important to make sure that alignments in place and actually making sure that the rest of communication and context is really in place to make sure that team succeeds and the lens of the business. >> Andrew I'm interested in the market, I mean, my sense is engineering management is sort of new, very new, actually, although we talked about productivity being sort of one of the metrics and there are tools out there, but how do you look at this market? Is there a big whale that you're targeting? Or is this more of a Greenfield up? >> I think really, it's a Greenfield opportunity. I think if you were to you know, people don't wake up right now with a quadrant they look at or a wave that they're actually measuring on at the moment. It doesn't say isn't coming. I mean, if you look at this as a baseball game, we're probably getting one and defining market right now. And you can tell because if you look across the vendor space, I think there's a lot of variety of different solutions in these places. And so, from my view, you know, there's room right now just to innovate and describe what this platform really is going to be right and that's what the next few years are going to look like and defining this market. >> Did the unknown nature of the market? Was that was that a challenge in terms of your race? >> I actually think that's actually the opportunity. I think both as founders and and our investors, I think really, whenever you have these Greenfield opportunities, it helps you create big opportunities, I either grow you know, this better than anyone, I think, define the markets are very clear in the box you're trying to fill in, it's a race to do that. You know, this space here is a space ripe for innovation. It needs innovation, both in product and process and how you going to market and the story will be told five years from now. >> So I want to talk about your go to market but before we do, so one of the things you got to do when you're doing your investor pitches, you got to figure out the total available market your served market. I mean, how did you do that? What can you share with us about the TAM? >> I mean, okay, there's the quantitative answer, right, you could pick apart all the companies in the world count all the software engineers and I can tell you, it's going to be a big number, right You can also map it to other large software engineering companies like Atlassian, or even Microsoft and talk about the markets there. But I think, you know, look, the world has moved far for long with that like, what's the word every company is a software company now? I think it's not a necessary part of the pitch anymore. I think everyone intuits the TAM is large, because even air conditioning companies now have hundreds of software engineers, It's no longer this niche thing like it was 20 years ago. I think literally, you know, every company in the planet could be a potential customer of Jellyfish in the future. >> You know I feel like some sometimes if you can actually size the TAM, it's maybe a negative in your race because if the TAM is just so obviously large then investors say hey, okay, check the markets huge, and that's what they want to see. >> And I think part of it too, is like we've seen the last five years, not just has you know, every company become a software company. This also means the engineering departments and how they recruit have been really scrutinized. Everybody needs more wants more engineers, they're hard to get and expensive. I think everyone's realizing like, because of both of those things, everyone cares a lot more, It's no longer this, you know, small number of people have low cost. It's actually just an expensive investment, a strategic one. And I want to make sure everyone wants to make sure it's pointing in the right direction now. >> So there's a lot of people in our community, young people get, you know, either just graduated from college or been out for, you know, 567 years working at a company and feel like they want to do their own thing. And they're always interested in how you did it, how you got started, how you ascended the company where you know how you seated it, I think, I think you guys started in 2017, I think you've raised $12 million. But take us back to the beginning, how did you and your co founders get launched, you know, how did you see the company and bootstrap it? So I mean, I, I think for us, like we're lucky to have actually been through all three of us through a number of different startups. So I think this is for us coming with a lot of awareness of actually how to build the company, we had the chance, you know, in at the talent 2016 to actually get the proverbial band back together. We hadn't worked together in probably shy of 15 years. But I think we really respected the chance to do so. And so we got together and said, like, hey, let's see if there's an opportunity for us to do something together. And so that was a real journey, you know, we pushed through a number of different concepts, we largely fell into this one simply because of our backgrounds, right, it is an area that we actually bring some personal expertise to and our networks bring it that way, but also some passion around wanting to actually solve it. So I think it's probably at the end of 16 that we actually said, like, hey, this is a space we might be interested in. We actually spent I think the the first couple months of 17 just interviewing every VPE CTO CEO, exactly we could find I think we probably talked to north of 65 technology leaders in that time period. Largely just actually asked him like, hey, what do you think about this space, this idea? What do you do instead? In fact, tell me not to start the startup, I don't want to invest x years of my life to find out that there's a better solution out there. The part that was I think amazing was that everyone was interviews, everybody kind of stopped at the end of it and leaned in and said, "hey, can you do me a favor? Can you write up the, whatever your notes on this and just send me the actual answer? So at that moment, we knew that, you know, we didn't have the solution yet. But we knew there's pain out there an opportunity to actually solve something, we weren't the only ones that actually identified that. And so that became the mission, which is how do we make people in this seat? How do we make their lives better, right? And, you know, sliding forward that you know, concurrent with actually, the early checks coming in, we actually call those same folks back and said, hey, can we work with you to build the product? So from a philosophical standpoint, we really believe in actually building with our customers, right, and so, from the first moment, you know, pre product, you know, pre code, we sat down with those same people and said, hey, let's work with you. let's do things by hand, let's do with your data, just to make sure that we understand what we're building a use case that you care about. >> Okay, so you co-created really with the customers as you actually started generating revenue, kind of a sell design build model, is that right, or? >> Yeah, I want to think of a much more of a, as an alpha product development, right, I think, you know, our philosophy on that early on, let's say June of 17. It was, look, we'll do your manual, we'll do your board decks for you, we'll do your management team slides, we'll do your metrics will do your capitalization, right. We'll do whatever you need on a manual basis, as long as we can work with you and your data. And you know, because we always had an eye on building the platform there. And so behind the scenes, of course, we're automating all of this. But that helps make sure that the use cases that we're building for were things they actually needed, we're going to use. >> Did you find you had to leave a lot on the cutting room floor? In other words, a lot of times when you take that approach, and you kind of try to generate maybe early revenue from customers, you sometimes get sick, especially in the enterprise, you get sucked into specials and some of your custom work that might not scale across the the other organizations. You guys obviously experienced, was that something you guys put a lot of thought into and how did you manage that? >> Look, I don't think it's magic, I think we were aware. To your point we've done this a few times, so at least we knew the pitfalls, like yeah, so some stuff has been left on the cutting for in the sense that we probably, you know, pushed harder on areas that we push less on today. I don't think anything was abandoned. I think part of it is that, you know, there's two sides of it, right, which is, if you're able to think about where you want to go, which is building a platform, you can always take any engagement and trade it off and say like, hey, is this something we want to build? Does this make sense for actually leading us towards the long platform story? you don't have to do every opportunity that comes along. So I think you need to thread the needle and I should take advantage of working with customers, but also making sure you have an eye for where you're going the North star so you can pick and choose which project you want to work on or which customers you know you want to work with because end of the day products are really the byproduct of who you work for who you serve as who you actually build for. And so we're very conscious along the way to choose the right individuals, the right partners that helps shape the product over time. >> And you guys had some some engineering chops and your own Andy Jassy says there's no compression algorithm, the algorithm for experience and then maybe that's an Amazon thing, but I hear him all the time. So you know, they had that to your advantage. And so, okay, so i got your website I see you've got customers so you know, you're well into your journey here. You've got product market fit, I presume you've got you know, your SaaS model your pricing down, but where are you in your sort of journey and you're phasing? >> I mean, geez, those words all change quite a bit these days, I would actually say product market fit is never a binary thing, that the constant journey, right so I would say that we're always working on that because the markets are always moving. And we have a market that is changing month on month and a quarter on quarter, right, so I never want to declare victory on that. Because that's going to get left behind. I think in terms of our journey, like we have on the team right now is 27 people normally based in downtown Boston, We're all working from home at the moment. You know, we have, you know, a sales team in place now of I think six, seven folks now. So we're in market actually pushing this forward. But, you know, I think for us, we're out there really kind of scaling the story right now. I think we've had some tremendous customers we had the chance to work with, we have a product that we're really proud of. I think we just need to put more units out there and more customers to actually make them more successful. So I think anything we're really in the act of repeating every function of the organization right now. It's really kind of build it up. So okay, so I mean, normally when you do a startup, you go to your friends, first the people in your core circles, you get them to that's I'm sure you did something similar. You're obviously beyond that phase of six to seven salespeople, you're starting to scale up, you've probably got a good sense as to that types of salespeople that you're looking for. And then now you're trying to figure out okay, it sounds like how much do I spend on marketing? How does that affect sales productivity? And then how do we scale that whole thing up and then go hyper scale and build a moat and all that other good stuff. >> Yeah, I think you're exactly right, I mean, I think we're at the point now where we can actually start making trade offs, right, like, you know, like, you know, do we actually add a additional salesperson? Do we actually invest in marketing programs? Do we actually build out more strategic product? I think you know, we are still the point we have to make trade offs, right, but the business is mature enough that we can make trade offs right, that makes sense. >> So let's talk about customers, I mean, maybe you could give us some examples, some of your, your favorite examples how you've impacted their business. >> Well, I think if you look at actually our website, I think there's a few case studies up there, I think there's building them up there, there's like salsify books at toast. I think all three of those actually really talk about different kinds of use cases around how we actually affect them. So One of which will we're really helping them actually on alignment, right and making sure that their team is working on the most important things, And, and in those situations, when you're working on the most important thing, you're really kind of essentially getting opportunity cost of engineers and making sure that they're driving towards things that you really will help the business, right so if you're, if you're looking at it that way, you're finding engineers that can help you progress faster, but you're building more product faster, because you can focus the energy where the team is going. And so that's case one, I think another case is really is around, making sure around quantitative management visibility, right, making sure that the team is visible in the metrics and in their output to make sure that they're performing their best, right, and that might mean everything from automated performance management to just making sure that people aren't left behind and making sure that they're the team is actually healthy in their function. And then the last of which is really around capitalization, which is a financial process and really automating that, that's out of the house which is, you know, capitalization requires a traditionally, engineering leaders have to manually fill out spreadsheets for finance, and for the accounting team to make sure that they're actually able to account for where the team effort is going, and then it can actually capitalize it correctly when we treat it from a financial perspective. And so we automate that process that just makes everyone's lives easier. So you're no longer manually data on a week by week basis. >> So I may have obviously seen some of your product and some of the outputs but your your SaaS based model, you know, cloud pricing, all that sort of modern, you know, approaches and business practices. And but what else can you tell us about sort of your, pricing model and how you're going to market? >> Yeah, so I think, you know, we are a SaaS hosted application. We also have, you know, open source agents that have been deployed on premise to actually, you know, whether to work with complex network architectures or deal specific redaction concerns, so we got to operate an on premise environments in that way. You mentioned our pricing is SaaS base we broadly price annually, you know, from broad strokes perspective, it's relative to the the size of the engineering team. Very simply like a, an engineering team of 300 people is a lot more complicated than engineering team of 30 people, right, put it that way. And the pricing reflects that. And then to your question around, like, what else I can talk about? Well, you know, I made the analogy earlier around like they were trying differentiating what Salesforce did for sales. What I mean by that really is providing that executive and leadership visibility to that department, right, if you're looking at the innovation that Salesforce brought in the early aughts, it was really getting stuff out of notebooks into the cloud through manual data entry, and in contemporary sales, I think there's less of that these days, it's all through plugins and voice recognition and stuff. And in the same way, in the engineering side, we're not in the business of actually asking for new data entry. In fact, we connect with systems engineers already using You know, the JIRA is the GitHub to get labs, all they know that their testing tools and their CI tools and then all of those things really we emit what we call engineering signals. So the engineers don't do anything differently. We collect that data, we connect to those systems, we clean that data, normalize and contextualize it with respect to business data. And that's actually where the insight actually comes from. Because if you just look at the raw engineering data by itself, there's not a lot you can do with it, right you know, I joked with you in our kind of earlier conversation, which is, you might look at, you know, your 300 get or request your engineers are produced thing. Like, it doesn't really help you figure out if you're going to do great this quarter, right? And so for us, we really bring that in contextualize it and make sure that you understand it in a business context, to talk about like, hey, is the team accelerating and being successful in the ways that we need the business needs them to do. >> Yeah, you're so right I get a stream of those every day every week and I open them up and I go, okay, I don't know. There's people that work in sort of last sort of topic areas is I want to understand where you want to take this thing. I think I'm writing that you've raised about $12 million, you obviously got a big market seems like you've got a great product. I mean, if I'm you, I'm throwing gasoline on the fire, I want to run the table, you got to create the market . So that's sometimes kind of expensive. Where do you want to take this thing? >> I mean, look, this may sound hubris bowl, but our ambitions are to build a large multi billion dollar standalone software company. And and I think, you know, part of the reason why I say it that way is that I think it's important to have a North star, right. It's important to have a North star to make sure that we're all headed in the right direction. We get the right team members, actually, as we grow the team, and then we actually capitalize it accordingly. I think if you look at the analogy, we started out earlier around sales and marketing. Every time someone's actually cracked that leadership visibility, for each function, there has been a multi billion dollar opportunity there, if not, a multi, you know multi multi billion dollar opportunity out there. So I don't think it's a overly anim facies that where we're going. But I think there's a lot of work to get from here to there. >> Yeah, I mean, I didn't ask you directly about the competition, I did ask you if there's a big whale, is there a big entity, you know, like a database, guys, is they want to target oracle, for example. And I looked around and I, I really didn't see it. It really does look like a Greenfield opportunity, which is absolutely enormous. I mean, I think I'm getting that right. >> Yeah, I think you're right on, and look at I think there are going to be more small players actually entering the market. Like I think whenever we look at new markets, and as they actually kind of build momentum, that always happens. And so of course, I you know, in that sense, I want competition to be here. But right now, I really don't focus on that. I think as for us, It's really about our product in the hands of our customers, how we make them successful. And then let's rinse and rinse and repeat that over and over again to more and more companies. >> Yeah, you don't have to compete you guys have to create. Andrew, great to have you on thanks so much for sharing your insights on your company, good luck with Jellyfish and come back anytime you know, in the future would love to track your progress and see how you're doing. >> Right Dave, thank you so much for having me here and I hope you, your family or team are staying healthy and all this and I look forward to next time. okay, and thank you for watching everybody this is Dave Vellante for theCUBE and startup insights. We'll see you next time. (upbeat music)

>> Narrator: From the Cube studios in Palo Alto and Boston, this is and episode in the Remote Works Citrix virtual series. >> Hello everybody, my name is Dave Vellante, and welcome to this cube conversation. You know, for the last several weeks, we've been interviewing key executives to really try to understand how they're responding to the COVID-19 crisis. And one of the key areas that we've been reporting on is the so called work from home offset, and I'll explain that in a little bit. But there are two great executives from Citrix that I'm really pleased to have on Donna Kimmel is the Executive Vice President and Chief chief people officer. Donna, great to see you. Thanks for coming on. >> Thank you. >> And she she's joined by Meerah Rajavel, who's the CIO of Citrix. Meerah, thank you as well. >> Thank you. >> So, I mean, this thing has been amazing. We've been doing a lot of research and it just obviously came out of the blue guys, if you would actually bring up that that chart. I want to set up the conversation here. This is something that we've been reporting on for a while. This is an ETR survey from about 1300 CIOs and IT practitioners that we asked them, how is your budget going to change in 2020 as a result of COVID? And you can see the red, we all know the story in the red, it's ugly. But surprisingly, about 35% of the respondents said no change. They're actually going to plow ahead. But what's even more surprising was 20 plus percent, about 21% said we're actually going to spend more. And so you can see from the data, that it's actually would be a lot worse, we're not for the green. Now, the reality is that green is a function really have worked from home infrastructure. And guys, that's something that I really want to talk to you about today. So, Donna, let me start with you. I mean, this is we're always talking about people, people process and technology. I mean, we went from put your toe in the water with work from home infrastructure, to all in. Your thoughts I mean, this is just overnight. >> Absolutely, you know, I think when I think about remote work and working from home, it is really not business as usual and probably was the biggest change that businesses have experienced, even in my career and many others. You know this was pretty much thrust upon us the work from home. And we realized that it requires new ways of thinking and behaving and operating. Our home offices quickly became kitchen tables and basements and bathrooms and bedrooms. And, in addition to it, not necessarily being set up the way that we would normally set it up if we knew we were going to work from home. It also didn't generally involve caring for family members at the same time. And so, most people thought for the first couple of weeks well, I can get through this. You know, for, it's not an extended period of time, but the reality is it's become an extended period of time. And I think ultimately, you know when we step back and think we're as humans, we're all survivors, and we're resilient. And there's a number of ways that, you know, we can help our employees as they make the adjustment that was really sort of pushed on them. >> Now, the executives that I've been talking to they, to a person start with, look, the safety and health of our team is the most important. So you obviously had to communicate that. Donna, I wonder if you could talk about sort of the priorities, you know what is it the cadence of your communication? The transparency of your communication? What really was your kind of first move, if you will? >> Yeah, absolutely. I think for us, one of the first things we had to step back and think about is, who are we what is what is our culture, what's important to us and we recognize it Citrix, it's our talent that makes the business successful. So we to show understand as much of the experience as possible that are that our employees are having, and really come at it from, I think a place of, of empathy. Listening to what's important to them, thinking about what's going to enable them to be successful because when our employees are successful, they truly drive success and a great experience for our customers. They're the ones out there helping to support our customers to support our sales partners, and certainly, ultimately, our communities. But when we think about this, we're thinking about the challenges, the opportunities, trying to develop plans and programs, and making sure that we have continuous information that is provided to our employees. And I think part of it you know we'll have an opportunity to talk with Meerah as well. When we step back, we think about kind of three things from a future of work perspective, we always think about the culture of the organization. Which is the embodiment of the values, the who we are and what we do. All of this clearly is grounded in the business objectives. So the first piece is our is our culture. The second piece is our physical space. So what is our environment like that enables us to be as productive as possible. And then the third piece is our digital space. If you can think about all of those almost as a Venn diagram, and that really puts the employee at the center. When we think about what's going to enable our employees to be successful, we think about that in a very holistic way. And so culture is sorry, did you want to-- >> Oh no please. >> Yeah. Culture for us is really grounded in our ability to drive trust in the organization. It's about that human connection. Because the more we can be connected with each other's managers to employees and peers to employees, the better off we are, people will feel less isolated. Because without that face to face, it makes it, and face to face and I'll say in person makes it a lot more difficult. The second piece that we focus on is that physical environment. And I think for many employees because they were thrust into the situation when they compare it to the work environment, when you're in the office, there's almost a professional feel, in that work environment and so employees feel a fair amount of pressure to try to create that same professionalism in their home. And the reality is, it's hard to do that. So it puts a lot of pressure on employees when they recognize that the whole family is quarantined with them, right? There's homeschooling going on. There's no childcare or eldercare. There's interruptions at inopportune times, barking dogs and cats walking across keyboards and family members doing drive-bys while the video cameras on and I think one of the things that we've been able to do is to help employees feel comfortable with that's who you are, that's our humanity. And the more we can help people feel comfortable about creating that physical space that's open and welcoming. That really helps drive that experience. And then the third piece, as I mentioned, is the digital space. And that's really where the partnership with Meerah comes in is so, so important, do they have the right tools and technology at home to be able to drive that experience? And for us, you know as Meerah and I have talked that partnership between IT and HR is critical. We're almost like the new BFFs in order to drive variance to enable our employees to be as productive as possible in this work from home. >> All right, so Meerah, let's let's get into that. So once you've established the safety, the health of your your employees, obviously financial flexibility and runway and the like their physical digital space. Now, you're really under a microscope with the tech. Now, of course, Citrix has been in this business for decades. So you know a lot about this, but nonetheless, this is really new. You were thrust into it overnight. Your thoughts on on how you responded and you know kind of where we're at in that journey. >> Absolutely, absolutely. So one other thing Donna mentioned, right, the three aspects when we moved to work from home, the biggest piece of this aspect that made it like for example, she was telling, I mean, I myself, we are in transition. I'm moving from Austin, Texas to Florida when it is all in the middle. I'm right now in the middle of my transition, I'm not settled my new house. And literally I'm doing this interview with the sitting my laptop on top of cereal boxes right now. That's actually something that I empathize clearly with my employees. So the physical space when we are in an office location is not any more that we can control. So the digital space need to really compensate for the physical space. The culture is something I think we are very lucky being in Citrix, the notion of what we have been always been talking about remote work, and employee experience, we have got that ingrained. So when we have to go into this remote workspace, work force culture, the culture is something that I would say we had some foundation to stand on. But IT has to come in, it's not an easy job because we want to give people the ability to do they what they want to do in a productive fashion. But now digital need to compensate for the physical, you know efficiencies that are possibly lacking in a home environment. So I looked at it from three C's, right? It starts with connectivity, right? Connectivity being are we providing the right kind of connectivity, which is to a secure connection. At the end of the day my job here is to make the employee productive and secure at the same time. It's not just about the productivity, but also wrap it up with a greater experience. So we start looking at connectivity from a security point of view, from performance point of view, using you know technologies like SDVAN and maximizing their performance to the nearest, how we can, you know break out the circuits to maximize performance for our employees. We also need to take into account that there are countries we went into the last mile to understand where the true problem is. Because if you go to Asia, there are so many countries, you know even if we can provide superior experience, their experience is very dependent on the local connectivity. So we need to look at, okay, how do we ensure our heavy duty applications are in a way optimized so it doesn't become a productivity tip for the employee. The second is if you think about productivity for employees, and it's all about information sharing and content sharing, right? So I call the second C is the content. The ability for the employee to have the right data at the right place. So they can make decisions and they can be productive. So using things like whether it is your ShareFile or your OneDrive or your collaboration platform JIRA, it doesn't matter, but you have to really make sure that data and information are available. And we focused on making sure that we are streamlined that and communicating about that very vocally like to Donna's point. The third C we looked at was collaboration, right? I mean, that's actually where, we are now compensating for the physical touch with a digital touch. So that includes things like your audio conferencing platform, your video conferencing platform, your ability to bring these different facets together, right? I mean, the ability to share, a ability to whiteboard I had last week, three days off site, and it was a complete virtual off site with nine hours of working session. And we used all kinds of tools that literally we had digital stickies to move around that integrated into our video conferencing platform that integrated into our conference sharing platform. So whatever we are doing, these are all connected. At the end of the day I truly felt like you know what i can contribute to not you know adding to the carbon footprint of the globe, because we have people from all over the globe, all of a sudden, I'm getting feedback from employees saying now the playing field is completely level down, people who have been remote users before they felt they had a short stick. Now everybody's same. In fact, my staff actually talked to one of my permanent remote employees and say, hey, what is the tips that I can use from you to make sure I'm productive, right? So I see the culture aspect is super important. That's actually bringing us together, but it is from a technology and digital point of view, bringing your, you know connectivity, content and collaboration in a way that it's going to be secure and in a way that we are looking at it with the aspect of your culture and from the employee shoes is a super important thing from a technology point of view. >> So Donna, you mentioned the sort of BFF between between HR and IT now, of course, HR IT have always had a relationship but it really has been around that Human Capital Managers Software, whether it was simplified and efficient onboarding or certain, change management functions. What have you been able to learn from that relationship and apply and what's new? >> You know, I think, what we're doing together what Meerah and I and the IT in the HR organizations are really doing together is truly understanding what it means to enable productivity for employees. And when you think about having the right tools to enable employees to be productive, doing that in alignment with the culture of the organization, what is it that drives our sense of meaning and accomplishment? And then being able to do it in a way both in a physical environment whether that physical environment is in the office or if it is remote. We do Look collectively together at the change management, how do you get employees to adopt new ways of doing things? And utilize that and learn from it. So we experiment with certain types of productivity tools, as Meerah was, was talking about, which ones worked, which ones needed to change, what worked for some teams and didn't work for others, when she and I can do that together, and our departments can do that together that enables us to truly drive productivity across the organization. >> Yeah, I would probably add one more thing to what Donna said. I mean, one of the thing is, also if you think about it, you know the human resource, the talent organization has a much better understanding of the culture of the subcultures, right? I mean, I've never been in a company even when it's 1000 people company, you have subcultures. And HR is in involved in the culture of those subcultures as we are going through. From IT point of view, we look at it from user personas, okay? So a salesperson who's actually always on road or always like more of a remote worker versus an engineering person. I mean, we are a software company and R&D persona requires a different set of productivity tools, compared to a salesperson compared to an executive compared to an executive assistant, right? So for us, it's actually bringing that different functional line of business. And that type of personas. And HR is absolutely crucial because as we are looking at it, we're saying, hey, what is the success for this organization, and what's the culture of that organization and one of the primary job roles and we don't do just with HR but HR gives us so much you know content to get jumpstart, then when we engage with the real users, we are not going with a blank sheet of paper we are going with something that they can react to and they can add to it. So we are doing a design thinking with them with something they can begin start together rather than you know white canvas and telling, tell me what do you want? I mean, he's asked, what do you want, you'll be getting, you know finding the sky on the moon. >> Well, it's a good thing you have those virtual stickies to help with that design thinking, right? You know, one of the things that I've been been saying is that, you know we've never seen obviously anything like this before a forced shutdown to the economy, which is why we're going to remember it. And like 911, you know post 911 we are going to see some things here that that have permanence, bad post GDPR for example, it required, certain changes. So, Donna, I want to begin start with you. Just it's ironic that, you know we're starting a new decade with this crisis. We're not just going to go back and revert the 2019 there's not just going to be some, you know all of a sudden, everything is rosy again, it's not. There's going to be certain permanent changes. How much have you thought about that? And do you have any visibility on what those are going to be? >> Yeah, you know when I stepped back and I think about this, and I think a large part of it has to do with much of what Meerah was just talking about in terms of design thinking. It's really, I think, for all of us, it's coming back to recognize that this became almost a forced opportunity to focus on business continuity. And how do we think about what's right for us as we move forward? But the design of that is based on what is right? What's the context for that particular business? What's the culture of that organization? What are the products and services that, you know that business provides? What are the subcultures in the organization? So, for me, it really does step back to say, look, we need to focus on business continuity. And now we have a couple of new models where you know in the past, it would be really easy for managers to say, you know I don't think my team can work remotely or your job isn't possible to do remotely. And now what we're finding in many businesses is that many jobs can actually be done remotely if they're provided the right tools and the right resources. So for me it, I step back and say, as we think about the business continuity going forward, there is a new way to work. It is a combination of finding that flexibility between working in the office and remote work and providing the right tools that enable employees to be able to do it successfully. >> You know, Meerah, this notion that Don is bringing up of business continuance, I've sort of been noodling on this and thinking that going forward, one of the things that will change is that companies might be willing to sub optimize near term performance to put in better business resiliency. Now at the same time, I know how CEOs thing and they say, okay great, we're going to make that investment. Yeah, fine. We'll maybe sacrifice some short term performance, but I had a really interesting conversation recently with a chief data officer said you don't have to sacrifice necessarily, with with data in this new era, there actually are ways in which you can both drive business resilience and drive productivity and ultimately profitability. What's your thinking on on that sort of imbalance or balance, if you will? >> I agree with that statement. Because to me, you know today's business we need to look at I mean, especially with the cloud and some of the new technologies that we have, I mean, even I see this thing coming out of COVID there's going to be industries that are going to come out new business models that are going to emerge, right? I mean, think about telemedicine, we have been very, very hesitant about telemedicine for decades now. I mean, that's not a new concept, but we have been very hesitant. we said, I have to see the doctor. But today, pretty much everybody except for if you're seriously injured, you're getting telemedicine. That industry is going to work, right? So to me the statement you made is absolutely, absolutely, and for me, it's actually an opportunity coming out of an adversity that's going to come out. When I think about it, the most important thing I see is the businesses that are going to be successful. That's why even HR, you know partnership is even more greater. The businesses that has talent with digital dexterity are the ones that are going to win, right? I mean, regardless, you know whether you're in HR, whether you're in finance, whether you're in IT, you're in R&D, you're in manufacturing doesn't matter. Your digital dexterity of your company really makes you whether you win in the market, or you're you're one of those dinosaurs in the market, right? And how do you bring those together? That's a cultural change. That's actually educating, right? I mean, we don't want to leave, we already have talent shortage, and we don't want Want to leave a generation of population behind and focused on only the millennials and others because I mean, recently I've been going through the scaled agile framework, which is a lean agile and I really love the word of lean agile, lean has a lot of economies of scale. Agile brings a lot of agility. When you bring them together, you get both. And that's exactly what we need to do with our talent, bring the vision and bring this digital dexterity that we need to bring there. How we get it from a productivity? Of course, we want to be respectful of privacy. But as we have been going through we have been looking at different productivity metrics looking at, you know what is the usage pattern of our employees, how much code checking they've done? How was my MTTR being, I mean, in my organization, I've been looking at the velocity of our transaction processing and our issue resolution SLA times. And we also even, you know had a little because I think at the end of the day, we human we actually We are social animals, we need that patch. And we cannot forget, we are not mechanical, we are human. So we need that empathy and we need that emotional side of it. So we have been both qualitatively and quantitatively checking with our workforce, how they're feeling about it, and also looking at the data to see if the productivity is telling the story, what people are talking about. And to our surprise, you know 66% of our population, when we did this pulse survey said, they feel more productive in this situation, because many of them commented that, you know the time they save from not commuting, or the feel, just the sense of spending a little bit more time with the family is actually giving them that extra boost. And they can really do a work life integration, not like a work life balance they need to do. And we also heard about 11% felt pretty much they're in the same range. And but I also want to recognize it's not for everyone, right? I mean, we do have folks who are in manufacturing, they need to patch the physical things. And those jobs in certain days need to be, more physical. So there's about 3-5%, depending on your job function said, you know what I need access to the lab because I really deal with changing my connectivity, changing my or a dislike for the customer, I'm repairing their board, I really need to see that, those are the ones where we find kind of, you know absolute physical touch is required. >> You know, in a way, I mean, we're kind of lucky in the technology business talk about the digital transformation. I've been saying this is going to accelerate a lot of digital transformations. But for us, you look at the Cube, we've been up remote studios, no problem. You're a software company, you've already really transitioned largely to a subscription model so you can code remotely, but there are some industries in particular industries, where you guys sell a lot of product, I think about healthcare, you mentioned telemedicine, Meerah, financial services, defense, big users of VDI, they're highly regulated and secure industries. And while it's not, you know your main thrust, you talk to your peers and in those industries. So, and I've always said, you know some of these industries really haven't digitally transformed, they're actually kind of complacent. My feeling is that this is going to really accelerate, you know some of those-- >> Absolutely. Industries that haven't transformed and haven't been disrupted. I wonder if you could both comment from both a technology perspective and a people perspective. >> You know, I think, I think from the people perspective, it's really about mindset. And it and recognizing that how we approach these new problems and needs new ways of thinking about getting work done, is all about what our minds block us from thinking. And this pushed us into a situation where we've been able to demonstrate roles that we did not think could ever be done remotely, can actually be done remotely. And so for me, it is about a mindset shift. It's about enabling the dialogue sort of having the courage to have that dialogue inside of the organization to understand, again, what's the business context? What can we do in a more flexible way? And how do we continue to serve our customers the best that we can? >> I think for me, it comes down to you know protection is always an extinction, right? I mean, if you're trying to protect a current model, and if you're trying to be saying, you know, you don't want to be the dinosaur. Things are going to change and being proactive about the change and embracing the change will let you to some extent influence and control that change versus being the change being done to you. In this particular case, to me looking at it to see especially with today's technology around, you know manufacturing industry is probably going to see a lot of remote hands as well with IoT and robotics coming in. And I see that is going to be one area, you may see a drip down on type of talent that's getting extinct. On the other side, we are going to continue to see the demand on technology is going to continue to go up and especially which is already shortage. I mean, if I remember the last survey from KPMG, in December, the CIO survey said 60% of the CIOs responded, they are having challenges with the you know filling the roles and I also remember the other one is around Korn Ferry survey of technology talent shortage. By 2030, the expectation is we're going to leave around 8.7 billion or $7 trillion of revenue on the table and 85% will be unfulfilled. I mean, this is a time for, you know really how do you ensure there are industries that are going to transform which means there are certain skills, people need to reskill. I mean, even in technology that reskill and upskill is going to be a constant thing that's actually it's nobody is there, you know spark from that one, in my opinion in today's world. so that reskill and upskill is going to be the ones who are going to embrace that they're going to be in a bigger way and taking advantage of these transitions and transformations. I also think there are areas that we may see what we call the hype may have a broader adoption. So you'd mentioned about the chief data officer talking about how data can come in, I mean, I see automation accelerating and data is going to be a core component of acceleration. And you will see more and more you know things around how measurements becomes important as a start that leads to you know more data modeling that leads to more automation, that cycle is going to accelerate the influence of AI is going to accelerate even further than when we have said. I mean, I just wish some of the areas where, you know we have been slow in that option if you would have accelerated some of the challenges we are dealing with now with capacity, we wouldn't have been having problems. I mean, then I did a reflection with my team. The one of the highest one ranked by my leadership was we should have accelerated accelerated automation more. >> Well, I think what are some really, really interesting and deep points, but really no industry is safe, from disruption and in really Meerah to your points. If you're just paving the cow path, you're going to be in trouble. If you're trying to protect the past from the future, you're going to get disrupted. And I feel like you guys really have a good handle on this. And it's our pleasure to be able to post an interview such experts like yourselves, really appreciate you sharing your insights and your experience with with our audience. I mean, we're kind of all in this together. So thank you, Donna, Meerah, thanks so much for coming on the Cube. >> Thank you so much. >> Thank you for having us. >> You're welcome and thank you for watching everybody. This is Dave Vellante for the Cube. For my CXO series we will see you next time. (upbeat music)

>> From the Cube studios in Palo Alto in Boston, this is a Citrix Virtual Series examining the realities of a remote work world. >> Hello everybody, my name is Dave Vellante, and welcome to this cube conversation. You know, for the last several weeks, we've been interviewing key executives to really try to understand how they're responding to the COVID-19 crisis. And one of the key areas that we've been reporting on is the so called work from home offset, and I'll explain that in a little bit. But there are two great executives from Citrix that I'm really pleased to have on Donna Kimmel is the Executive Vice President and Chief chief people officer. Donna, great to see you. Thanks for coming on. >> Thank you. >> And she she's joined by Meerah Rajavel, who's the CIO of Citrix. Meerah, thank you as well. >> Thank you. >> So, I mean, this thing has been amazing. We've been doing a lot of research and it just obviously came out of the blue guys, if you would actually bring up that that chart. I want to set up the conversation here. This is something that we've been reporting on for a while. This is an ETR survey from about 1300 CIOs and IT practitioners that we asked them, how is your budget going to change in 2020 as a result of COVID? And you can see the red, we all know the story in the red, it's ugly. But surprisingly, about 35% of the respondents said no change. They're actually going to plow ahead. But what's even more surprising was 20 plus percent, about 21% said we're actually going to spend more. And so you can see from the data, that it's actually would be a lot worse, we're not for the green. Now, the reality is that green is a function really have worked from home infrastructure. And guys, that's something that I really want to talk to you about today. So, Donna, let me start with you. I mean, this is we're always talking about people, people process and technology. I mean, we went from put your toe in the water with work from home infrastructure, to all in. Your thoughts I mean, this is just overnight. >> Absolutely, you know, I think when I think about remote work and working from home, it is really not business as usual and probably was the biggest change that businesses have experienced, even in my career and many others. You know this was pretty much thrust upon us the work from home. And we realized that it requires new ways of thinking and behaving and operating. Our home offices quickly became kitchen tables and basements and bathrooms and bedrooms. And, in addition to it, not necessarily being set up the way that we would normally set it up if we knew we were going to work from home. It also didn't generally involve caring for family members at the same time. And so, most people thought for the first couple of weeks well, I can get through this. You know, for, it's not an extended period of time, but the reality is it's become an extended period of time. And I think ultimately, you know when we step back and think we're as humans, we're all survivors, and we're resilient. And there's a number of ways that, you know, we can help our employees as they make the adjustment that was really sort of pushed on them. >> Now, the executives that I've been talking to they, to a person start with, look, the safety and health of our team is the most important. So you obviously had to communicate that. Donna, I wonder if you could talk about sort of the priorities, you know what is it the cadence of your communication? The transparency of your communication? What really was your kind of first move, if you will? >> Yeah, absolutely. I think for us, one of the first things we had to step back and think about is, who are we what is what is our culture, what's important to us and we recognize it Citrix, it's our talent that makes the business successful. So we to show understand as much of the experience as possible that are that our employees are having, and really come at it from, I think a place of, of empathy. Listening to what's important to them, thinking about what's going to enable them to be successful because when our employees are successful, they truly drive success and a great experience for our customers. They're the ones out there helping to support our customers to support our sales partners, and certainly, ultimately, our communities. But when we think about this, we're thinking about the challenges, the opportunities, trying to develop plans and programs, and making sure that we have continuous information that is provided to our employees. And I think part of it you know we'll have an opportunity to talk with Meerah as well. When we step back, we think about kind of three things from a future of work perspective, we always think about the culture of the organization. Which is the embodiment of the values, the who we are and what we do. All of this clearly is grounded in the business objectives. So the first piece is our is our culture. The second piece is our physical space. So what is our environment like that enables us to be as productive as possible. And then the third piece is our digital space. If you can think about all of those almost as a Venn diagram, and that really puts the employee at the center. When we think about what's going to enable our employees to be successful, we think about that in a very holistic way. And so culture is sorry, did you want to-- >> Oh no please. >> Yeah. Culture for us is really grounded in our ability to drive trust in the organization. It's about that human connection. Because the more we can be connected with each other's managers to employees and peers to employees, the better off we are, people will feel less isolated. Because without that face to face, it makes it, and face to face and I'll say in person makes it a lot more difficult. The second piece that we focus on is that physical environment. And I think for many employees because they were thrust into the situation when they compare it to the work environment, when you're in the office, there's almost a professional feel, in that work environment and so employees feel a fair amount of pressure to try to create that same professionalism in their home. And the reality is, it's hard to do that. So it puts a lot of pressure on employees when they recognize that the whole family is quarantined with them, right? There's homeschooling going on. There's no childcare or eldercare. There's interruptions at inopportune times, barking dogs and cats walking across keyboards and family members doing drive-bys while the video cameras on and I think one of the things that we've been able to do is to help employees feel comfortable with that's who you are, that's our humanity. And the more we can help people feel comfortable about creating that physical space that's open and welcoming. That really helps drive that experience. And then the third piece, as I mentioned, is the digital space. And that's really where the partnership with Meerah comes in is so, so important, do they have the right tools and technology at home to be able to drive that experience? And for us, you know as Meerah and I have talked that partnership between IT and HR is critical. We're almost like the new BFFs in order to drive variance to enable our employees to be as productive as possible in this work from home. >> All right, so Meerah, let's let's get into that. So once you've established the safety, the health of your your employees, obviously financial flexibility and runway and the like their physical digital space. Now, you're really under a microscope with the tech. Now, of course, Citrix has been in this business for decades. So you know a lot about this, but nonetheless, this is really new. You were thrust into it overnight. Your thoughts on on how you responded and you know kind of where we're at in that journey. >> Absolutely, absolutely. So one other thing Donna mentioned, right, the three aspects when we moved to work from home, the biggest piece of this aspect that made it like for example, she was telling, I mean, I myself, we are in transition. I'm moving from Austin, Texas to Florida when it is all in the middle. I'm right now in the middle of my transition, I'm not settled my new house. And literally I'm doing this interview with the sitting my laptop on top of cereal boxes right now. That's actually something that I empathize clearly with my employees. So the physical space when we are in an office location is not any more that we can control. So the digital space need to really compensate for the physical space. The culture is something I think we are very lucky being in Citrix, the notion of what we have been always been talking about remote work, and employee experience, we have got that ingrained. So when we have to go into this remote workspace, work force culture, the culture is something that I would say we had some foundation to stand on. But IT has to come in, it's not an easy job because we want to give people the ability to do they what they want to do in a productive fashion. But now digital need to compensate for the physical, you know efficiencies that are possibly lacking in a home environment. So I looked at it from three C's, right? It starts with connectivity, right? Connectivity being are we providing the right kind of connectivity, which is to a secure connection. At the end of the day my job here is to make the employee productive and secure at the same time. It's not just about the productivity, but also wrap it up with a greater experience. So we start looking at connectivity from a security point of view, from performance point of view, using you know technologies like SDVAN and maximizing their performance to the nearest, how we can, you know break out the circuits to maximize performance for our employees. We also need to take into account that there are countries we went into the last mile to understand where the true problem is. Because if you go to Asia, there are so many countries, you know even if we can provide superior experience, their experience is very dependent on the local connectivity. So we need to look at, okay, how do we ensure our heavy duty applications are in a way optimized so it doesn't become a productivity tip for the employee. The second is if you think about productivity for employees, and it's all about information sharing and content sharing, right? So I call the second C is the content. The ability for the employee to have the right data at the right place. So they can make decisions and they can be productive. So using things like whether it is your ShareFile or your OneDrive or your collaboration platform JIRA, it doesn't matter, but you have to really make sure that data and information are available. And we focused on making sure that we are streamlined that and communicating about that very vocally like to Donna's point. The third C we looked at was collaboration, right? I mean, that's actually where, we are now compensating for the physical touch with a digital touch. So that includes things like your audio conferencing platform, your video conferencing platform, your ability to bring these different facets together, right? I mean, the ability to share, a ability to whiteboard I had last week, three days off site, and it was a complete virtual off site with nine hours of working session. And we used all kinds of tools that literally we had digital stickies to move around that integrated into our video conferencing platform that integrated into our conference sharing platform. So whatever we are doing, these are all connected. And the end of the day I truly felt like you know what i can contribute to not you know adding to the carbon footprint of the globe, because we have people from all over the globe, all of a sudden, I'm getting feedback from employees saying now the playing field is completely level down, people who have been remote users before they felt they had a short stick. Now everybody's same. In fact, my staff actually talked to one of my permanent remote employees and say, hey, what is the tips that I can use from you to make sure I'm productive, right? So I see the culture aspect is super important. That's actually bringing us together, but it is from a technology and digital point of view, bringing your, you know connectivity, content and collaboration in a way that it's going to be secure and in a way that we are looking at it with the aspect of your culture and from the employee shoes is a super important thing from a technology point. >> So Donna, you mentioned the sort of BFF between between HR and IT now, of course, HR IT have always had a relationship but it really has been around that Human Capital Managers Software, whether it was simplified and efficient onboarding or certain, change management functions. What have you been able to learn from that relationship and apply and what's new? >> You know, I think, what we're doing together what Meerah and I and the IT in the HR organizations are really doing together is truly understanding what it means to enable productivity for employees. And when you think about having the right tools to enable employees to be productive, doing that in alignment with the culture of the organization, what is it that drives our sense of meaning and accomplishment? And then being able to do it in a way both in a physical environment whether that physical environment is in the office or if it is remote. We do Look collectively together at the change management, how do you get employees to adopt new ways of doing things? And utilize that and learn from it. So we experiment with certain types of productivity tools, as Meerah was, was talking about, which ones worked, which ones needed to change, what worked for some teams and didn't work for others, when she and I can do that together, and our departments can do that together that enables us to truly drive productivity across the organization. >> Yeah, I would probably add one more thing to what Donna said. I mean, one of the thing is, also if you think about it, you know the human resource, the talent organization has a much better understanding of the culture of the subcultures, right? I mean, I've never been in a company even when it's 1000 people company, you have subcultures. And HR is in involved in the culture of those subcultures as we are going through. From IT point of view, we look at it from user personas, okay? So a salesperson who's actually always on road or always like more of a remote worker versus an engineering person. I mean, we are a software company and R&D persona requires a different set of productivity tools, compared to a salesperson compared to an executive compared to an executive assistant, right? So for us, it's actually bringing that different functional line of business. And that type of personas. And HR is absolutely crucial because as we are looking at it, we're saying, hey, what is the success for this organization, and what's the culture of that organization and one of the primary job roles and we don't do just with HR but HR gives us so much you know content to get jumpstart, then when we engage with the real users, we are not going with a blank sheet of paper we are going with something that they can react to and they can add to it. So we are doing a design thinking with them with something they can begin start together rather than you know white canvas and telling, tell me what do you want? I mean, he's asked, what do you want, you'll be getting, you know finding the sky on the moon. >> Well, it's a good thing you have those virtual stickies to help with that design thinking, right? You know, one of the things that I've been been saying is that, you know we've never seen obviously anything like this before a forced shutdown to the economy, which is why we're going to remember it. And like 911, you know post 911 we are going to see some things here that that have permanence, bad post GDPR for example, it required, certain changes. So, Donna, I want to begin start with you. Just it's ironic that, you know we're starting a new decade with this crisis. We're not just going to go back and revert the 2019 there's not just going to be some, you know all of a sudden, everything is rosy again, it's not. There's going to be certain permanent changes. How much have you thought about that? And do you have any visibility on what those are going to be? >> Yeah, you know when I stepped back and I think about this, and I think a large part of it has to do with much of what Meerah was just talking about in terms of design thinking. It's really, I think, for all of us, it's coming back to recognize that this became almost a forced opportunity to focus on business continuity. And how do we think about what's right for us as we move forward? But the design of that is based on what is right? What's the context for that particular business? What's the culture of that organization? What are the products and services that, you know that business provides? What are the subcultures in the organization? So, for me, it really does step back to say, look, we need to focus on business continuity. And now we have a couple of new models where you know in the past, it would be really easy for managers to say, you know I don't think my team can work remotely or your job isn't possible to do remotely. And now what we're finding in many businesses is that many jobs can actually be done remotely if they're provided the right tools and the right resources. So for me it, I step back and say, as we think about the business continuity going forward, there is a new way to work. It is a combination of finding that flexibility between working in the office and remote work and providing the right tools that enable employees to be able to do it successfully. >> You know, Meerah, this notion that Don is bringing up of business continuance, I've sort of been noodling on this and thinking that going forward, one of the things that will change is that companies might be willing to sub optimize near term performance to put in better business resiliency. Now at the same time, I know how CEOs thing and they say, okay great, we're going to make that investment. Yeah, fine. We'll maybe sacrifice some short term performance, but I had a really interesting conversation recently with a chief data officer said you don't have to sacrifice necessarily, with with data in this new era, there actually are ways in which you can both drive business resilience and drive productivity and ultimately profitability. What's your thinking on on that sort of imbalance or balance, if you will? >> I agree with that statement. Because to me, you know today's business we need to look at I mean, especially with the cloud and some of the new technologies that we have, I mean, even I see this thing coming out of COVID there's going to be industries that are going to come out new business models that are going to emerge, right? I mean, think about telemedicine, we have been very, very hesitant about telemedicine for decades now. I mean, that's not a new concept, but we have been very hesitant. we said, I have to see the doctor. But today, pretty much everybody except for if you're seriously injured, you're getting telemedicine. That industry is going to work, right? So to me the statement you made is absolutely, absolutely, and for me, it's actually an opportunity coming out of an adversity that's going to come out. When I think about it, the most important thing I see is the businesses that are going to be successful. That's why even HR, you know partnership is even more greater. The businesses that has talent with digital dexterity are the ones that are going to win, right? I mean, regardless, you know whether you're in HR, whether you're in finance, whether you're in IT, you're in R&D, you're in manufacturing doesn't matter. Your digital dexterity of your company really makes you whether you win in the market, or you're you're one of those dinosaurs in the market, right? And how do you bring those together? That's a cultural change. That's actually educating, right? I mean, we don't want to leave, we already have talent shortage, and we don't want Want to leave a generation of population behind and focused on only the millennials and others because I mean, recently I've been going through the scaled agile framework, which is a lean agile and I really love the word of lean agile, lean has a lot of economies of scale. Agile brings a lot of agility. When you bring them together, you get both. And that's exactly what we need to do with our talent, bring the vision and bring this digital dexterity that we need to bring there. How we get it from a productivity? Of course, we want to be respectful of privacy. But as we have been going through we have been looking at different productivity metrics looking at, you know what is the usage pattern of our employees, how much code checking they've done? How was my MTTR being, I mean, in my organization, I've been looking at the velocity of our transaction processing and our issue resolution SLA times. And we also even, you know had a little because I think at the end of the day, we human we actually We are social animals, we need that patch. And we cannot forget, we are not mechanical, we are human. So we need that empathy and we need that emotional side of it. So we have been both qualitatively and quantitatively checking with our workforce, how they're feeling about it, and also looking at the data to see if the productivity is telling the story, what people are talking about. And to our surprise, you know 66% of our population, when we did this pulse survey said, they feel more productive in this situation, because many of them commented that, you know the time they save from not commuting, or the feel, just the sense of spending a little bit more time with the family is actually giving them that extra boost. And they can really do a work life integration, not like a work life balance they need to do. And we also heard about 11% felt pretty much they're in the same range. And but I also want to recognize it's not for everyone, right? I mean, we do have folks who are in manufacturing, they need to patch the physical things. And those jobs in certain days need to be, more physical. So there's about 3-5%, depending on your job function said, you know what I need access to the lab because I really deal with changing my connectivity, changing my or a dislike for the customer, I'm repairing their board, I really need to see that, those are the ones where we find kind of, you know absolute physical touch is required. >> You know, in a way, I mean, we're kind of lucky in the technology business talk about the digital transformation. I've been saying this is going to accelerate a lot of digital transformations. But for us, you look at the Cube, we've been up remote studios, no problem. You're a software company, you've already really transitioned largely to a subscription model so you can code remotely, but there are some industries in particular industries, where you guys sell a lot of product, I think about healthcare, you mentioned telemedicine, Meerah, financial services, defense, big users of VDI, they're highly regulated and secure industries. And while it's not, you know your main thrust, you talk to your peers and in those industries. So, and I've always said, you know some of these industries really haven't digitally transformed, they're actually kind of complacent. My feeling is that this is going to really accelerate, you know some of those-- >> Absolutely. Industries that haven't transformed and haven't been disrupted. I wonder if you could both comment from both a technology perspective and a people perspective. >> You know, I think, I think from the people perspective, it's really about mindset. And it and recognizing that how we approach these new problems and needs new ways of thinking about getting work done, is all about what our minds block us from thinking. And this pushed us into a situation where we've been able to demonstrate roles that we did not think could ever be done remotely, can actually be done remotely. And so for me, it is about a mindset shift. It's about enabling the dialogue sort of having the courage to have that dialogue inside of the organization to understand, again, what's the business context? What can we do in a more flexible way? And how do we continue to serve our customers the best that we can? >> I think for me, it comes down to you know protection is always an extinction, right? I mean, if you're trying to protect a current model, and if you're trying to be saying, you know, you don't want to be the dinosaur. Things are going to change and being proactive about the change and embracing the change will let you to some extent influence and control that change versus being the change being done to you. In this particular case, to me looking at it to see especially with today's technology around, you know manufacturing industry is probably going to see a lot of remote hands as well with IoT and robotics coming in. And I see that is going to be one area, you may see a drip down on type of talent that's getting extinct. On the other side, we are going to continue to see the demand on technology is going to continue to go up and especially which is already shortage. I mean, if I remember the last survey from KPMG, in December, the CIO survey said 60% of the CIOs responded, they are having challenges with the you know filling the roles and I also remember the other one is around Korn Ferry survey of technology talent shortage. By 2030, the expectation is we're going to leave around 8.7 billion or $7 trillion of revenue on the table and 85% will be unfulfilled. I mean, this is a time for, you know really how do you ensure there are industries that are going to transform which means there are certain skills, people need to reskill. I mean, even in technology that reskill and upskill is going to be a constant thing that's actually it's nobody is there, you know spark from that one, in my opinion in today's world. so that reskill and upskill is going to be the ones who are going to embrace that they're going to be in a bigger way and taking advantage of these transitions and transformations. I also think there are areas that we may see what we call the hype may have a broader adoption. So you'd mentioned about the chief data officer talking about how data can come in, I mean, I see automation accelerating and data is going to be a core component of acceleration. And you will see more and more you know things around how measurements becomes important as a start that leads to you know more data modeling that leads to more automation, that cycle is going to accelerate the influence of AI is going to accelerate even further than when we have said. I mean, I just wish some of the areas where, you know we have been slow in that option if you would have accelerated some of the challenges we are dealing with now with capacity, we wouldn't have been having problems. I mean, then I did a reflection with my team. The one of the highest one ranked by my leadership was we should have accelerated accelerated automation more. >> Well, I think what are some really, really interesting and deep points, but really no industry is safe, from disruption and in really Meerah to your points. If you're just paving the cow path, you're going to be in trouble. If you're trying to protect the past from the future, you're going to get disrupted. And I feel like you guys really have a good handle on this. And it's our pleasure to be able to post an interview such experts like yourselves, really appreciate you sharing your insights and your experience with with our audience. I mean, we're kind of all in this together. So thank you, Donna, Meerah, thanks so much for coming on the Cube. >> Thank you so much. >> Thank you for having us. >> You're welcome and thank you for watching everybody. This is Dave Vellante for the Cube. For my CXO series we will see you next time. (upbeat music)