>>Thank you for joining us for accelerate next event. I hope you're enjoying it so far. I know you've heard about the industry challenges the I. T. Trends HP strategy from leaders in the industry and so today what we wanna do is focus on going deep on workload solutions. So in the most important workload solutions, the ones we always get asked about and so today we want to share with you some best practices, some examples of how we've helped other customers and how we can help you. All right with that. I'd like to start our panel now and introduce chris idler, who's the vice president and general manager of the element. Chris has extensive solution expertise, he's led HP solution engineering programs in the past. Welcome chris and Mark Nickerson, who is the Director of product management and his team is responsible for solution offerings, making sure we have the right solutions for our customers. Welcome guys, thanks for joining me. >>Thanks for having us christa. >>Yeah, so I'd like to start off with one of the big ones, the ones that we get asked about all the time, what we've been all been experienced in the last year, remote work, remote education and all the challenges that go along with that. So let's talk a little bit about the challenges that customers have had in transitioning to this remote work and remote education environments. >>Uh So I I really think that there's a couple of things that have stood out for me when we're talking with customers about V. D. I. Um first obviously there was a an unexpected and unprecedented level of interest in that area about a year ago and we all know the reasons why, but what it really uncovered was how little planning had gone into this space around a couple of key dynamics. One is scale. Um it's one thing to say, I'm going to enable V. D. I. For a part of my work force in a pre pandemic environment where the office was still the central hub of activity for work. It's a completely different scale. When you think about okay I'm going to have 50, 60, 80, maybe 100 of my workforce now distributed around the globe. Um Whether that's in an educational environment where now you're trying to accommodate staff and students in virtual learning, Whether that's in the area of things like Formula one racing, where we had the desire to still have events going on. But the need for a lot more social distancing. Not as many people able to be trackside but still needing to have that real time experience. This really manifested in a lot of ways and scale was something that I think a lot of customers hadn't put as much thought into. Initially the other area is around planning for experience a lot of times the V. D. I. Experience was planned out with very specific workloads are very specific applications in mind. And when you take it to a more broad based environment, if we're going to support multiple functions, multiple lines of business, there hasn't been as much planning or investigation that's gone into the application side. And so thinking about how graphically intense some applications are. Uh one customer that comes to mind would be Tyler I. S. D. Who did a fairly large rollout pre pandemic and as part of their big modernization effort, what they uncovered was even just changes in standard Windows applications Had become so much more graphically intense with Windows 10 with the latest updates with programs like Adobe that they were really needing to have an accelerated experience for a much larger percentage of their install base than they had counted on. So, um, in addition to planning for scale, you also need to have that visibility into what are the actual applications that are going to be used by these remote users? How graphically intense those might be. What's the logging experience going to be as well as the operating experience. And so really planning through that experience side as well as the scale and the number of users is kind of really two of the biggest, most important things that I've seen. >>You know, Mark, I'll just jump in real quick. I think you covered that pretty comprehensively there and it was well done. The a couple of observations I've made, one is just that um, V. D. I suddenly become like mission critical for sales. It's the front line, you know, for schools, it's the classroom, you know, that this isn't Uh cost cutting measure or uh optimization in IT. measure anymore. This is about running the business in a way it's a digital transformation. One aspect of about 1000 aspects of what does it mean to completely change how your business does. And I think what that translates to is that there's no margin for error, right? You know, you really need to to deploy this in a way that that performs, that understands what you're trying to use it for. That gives that end user the experience that they expect on their screen or on their handheld device or wherever they might be, whether it's a racetrack classroom or on the other end of a conference call or a boardroom. Right? So what we do in the engineering side of things when it comes to V. D. I. R. Really understand what's a tech worker, What's a knowledge worker? What's the power worker? What's a gP really going to look like? What time of day look like, You know, who's using it in the morning, Who is using it in the evening? When do you power up? When do you power down? Does the system behave? Does it just have the, it works function and what our clients can can get from H. P. E. Is um you know, a worldwide set of experiences that we can apply to, making sure that the solution delivers on its promises. So we're seeing the same thing you are christa, We see it all the time on beady eye and on the way businesses are changing the way they do business. >>Yeah. It's funny because when I talked to customers, you know, one of the things I heard that was a good tip is to roll it out to small groups first so you can really get a good sense of what the experiences before you roll it out to a lot of other people and then the expertise. Um It's not like every other workload that people have done before. So if you're new at it make sure you're getting the right advice expertise so that you're doing it the right way. Okay. One of the other things we've been talking a lot about today is digital transformation and moving to the edge. So now I'd like to shift gears and talk a little bit about how we've helped customers make that shift and this time I'll start with chris. >>All right Hey thanks. Okay so you know it's funny when it comes to edge because um the edge is different for every customer and every client and every single client that I've ever spoken to of. H. P. S. Has an edge somewhere. You know whether just like we were talking about the classroom might be the edge. But I think the industry when we're talking about edges talking about you know the internet of things if you remember that term from not too not too long ago you know and and the fact that everything is getting connected and how do we turn that into um into telemetry? And I think Mark is going to be able to talk through a a couple of examples of clients that we have in things like racing and automotive. But what we're learning about Edge is it's not just how do you make the Edge work? It's how do you integrate the edge into what you're already doing? And nobody's just the edge. Right. And so if it's if it's um ai ml dl there that's one way you want to use the edge. If it's a customer experience point of service, it's another, you know, there's yet another way to use the edge. So, it turns out that having a broad set of expertise like HP does, um, to be able to understand the different workloads that you're trying to tie together, including the ones that are running at the, at the edge. Often it involves really making sure you understand the data pipeline. What information is at the edge? How does it flow to the data center? How does it flow? And then which data center, which private cloud? Which public cloud are you using? Um, I think those are the areas where we, we really sort of shine is that we we understand the interconnectedness of these things. And so, for example, Red Bull, and I know you're going to talk about that in a minute mark, um the racing company, you know, for them the edges, the racetrack and, and you know, milliseconds or partial seconds winning and losing races, but then there's also an edge of um workers that are doing the design for the cars and how do they get quick access? So, um, we have a broad variety of infrastructure form factors and compute form factors to help with the edge. And this is another real advantage we have is that we we know how to put the right piece of equipment with the right software. And we also have great containerized software with our admiral container platform. So we're really becoming um, a perfect platform for hosting edge centric workloads and applications and data processing. Uh, it's uh um all the way down to things like a Superdome flex in the background. If you have some really, really, really big data that needs to be processed and of course our workhorse reliance that can be configured to support almost every combination of workload you have. So I know you started with edge christa but and and we're and we nail the edge with those different form factors, but let's make sure, you know, if you're listening to this, this show right now, um make sure you you don't isolate the edge and make sure they integrated with um with the rest of your operation, Mark, you know, what did I miss? >>Yeah, to that point chris I mean and this kind of actually ties the two things together that we've been talking about here at the Edge has become more critical as we have seen more work moving to the edge as where we do work, changes and evolves. And the edge has also become that much more closer because it has to be that much more connected. Um, to your point talking about where that edge exists, that edge can be a lot of different places. Um, but the one commonality really is that the edge is an area where work still needs to get accomplished. It can't just be a collection point and then everything gets shipped back to a data center back to some other area for the work. It's where the work actually needs to get done. Whether that's edge work in a used case like V. D. I. Or whether that's edge work. In the case of doing real time analytics, you mentioned red bull racing, I'll bring that up. I mean, you talk about uh, an area where time is of the essence, everything about that sport comes down to time. You're talking about wins and losses that are measured as you said in milliseconds. And that applies not just to how performance is happening on the track, but how you're able to adapt and modify the needs of the car, adapt to the evolving conditions on the track itself. And so when you talk about putting together a solution for an edge like that, you're right. It can't just be, here's a product that's going to allow us to collect data, ship it back someplace else and and wait for it to be processed in a couple of days, you have to have the ability to analyze that in real time. When we pull together a solution involving our compute products are storage products or networking products. When we're able to deliver that full package solution at the edge, what you see results like a 50 decrease in processing time to make real time analytic decisions about configurations for the car and adapting to real time test and track conditions. >>Yeah, really great point there. Um, and I really love the example of edge and racing because I mean that is where it all every millisecond counts. Um, and so important to process that at the edge. Now, switching gears just a little bit. Let's talk a little bit about um some examples of how we've helped customers when it comes to business agility and optimizing the workload for maximum outcome for business agility. Let's talk about some things that we've done to help customers with that >>mark, give it a >>shot. >>Uh, So when we, when we think about business agility, what you're really talking about is the ability to implement on the fly to be able to scale up and scale down the ability to adapt to real time changing situations. And I think the last year has been, has been an excellent example of exactly how so many businesses have been forced to do that. Um I think one of the areas that I think we've probably seen the most ability to help with customers in that agility area is around the space of private and hybrid clouds. Um if you take a look at the need that customers have to be able to migrate workloads and migrate data between public cloud environments, app development environments that may be hosted on site or maybe in the cloud, the ability to move out of development and into production and having the agility to then scale those application rollouts up, having the ability to have some of that. Um some of that private cloud flexibility in addition to a public cloud environment is something that is becoming increasingly crucial for a lot of our customers. >>All right, well, we could keep going on and on, but I'll stop it there. Uh, thank you so much Chris and Mark. This has been a great discussion. Thanks for sharing how we help other customers and some tips and advice for approaching these workloads. I thank you all for joining us and remind you to look at the on demand sessions. If you want to double click a little bit more into what we've been covering all day today, you can learn a lot more in those sessions. And I thank you for your time. Thanks for tuning in today.

>> From theCUBE Studios in Palo Alto, in Boston connecting with alt leaders all around the world, This is a CUBE conversation. >> Hello everybody, this is Dave Vellante and welcome to this Breaking Analysis. I'm here with Erik Bradley, who's the managing director of ETR and runs their VEN program. Erik good to see you. >> Very nice to see you too Dave. Hope you're doing well. >> Yeah, I'm doing okay hanging in there. You know, you guys in New York are fighting the battle. Looks like we're making some progress here so, you know, all the best, you and your family and the wider community. I'm really excited to have you on today because I had the pleasure of sitting in on a CIO/ CISO panel last week. And we're going to explain sort of what that's all about, but one of the things ETR does that I really like is they go deeper with anecdotal information and it's almost like in-depth interviews in these round tables. So they compliment their quarterly surveys, and their other drill down surveys, with other anecdotal information for people in their community. So it's a tried and true survey practice that adds some color to the dataset. So guys if you bring up the agenda, I want to share with the audience what we're going to talk about today. So, we'll talk a little bit about, you know we just did intros, I want to ask Erik, what ETR VENN is and then we'll go through some of the guests, but if we go back to Erik, explain a little bit about VENN and the whole process and how you guys do that. >> Yeah sure, we should hire you for marketing. You just did a great job, actually, describing that, but about three years ago what we decided was, ETR does an amazing job collecting the data. It can tell you what's happening, who it's happening to and when it's happening. But it can't always tell you why it's happened. So leveraging a lot of my background in twenty-plus years in journalism and institutional Wall Street research, we decided to take the ETR community, the people that actually take the surveys, and start doing interviews with them and start doing events with them. And enable to doing that, we're basically just trying to compliment the survey findings and the data. So what we always say is that ETR will always give you the quantitative answer and VENN will give you the qualitative answer. >> Now guys, let's bring up the agenda slide again, let's take a look at the folks that participated in the round table. Now, for ETR's clients, they actually know the names and the titles and well the company that these guys work for. We've anonymized it for the public. But you had a CIO of a Global Auto Supplier, a CISO of a Diversified Holdings Firm, who actually had some hospitality exposure but also some government contract manufacturing exposure. Chief Architect of a Software ISV and a VP and CISO of a Global Hospitality Resort Chain. So you had three out of the for, Erik, were really in industries that are getting hit hard. Obviously the software company maybe a little bit better. But maybe you can add some color to that. >> Well actually the software company, unfortunately, was getting hit hard as well because they're a software ISV that actually plays into the manufacturing space as well. So, this particular panel of CIOs and CISOs were actually in a very hard hit industries. And are going to make sure we do two more follow-ups with different industry verticals to make sure we're getting a little bit of a wider berth and collect all of that information in a better way. But coming back to this particular call, the whole reason we did this, and as you know, you spoke to my colleague and friend, Sagar Kadakia, who is the Director of Research for ETR, and we were nimble enough to actually change our survey while it was in the field, to start collecting data on what the real-time impact was on the COVID-19 pandemic. We were able to take that information, extrapolate it, and then say okay let's start reading out to these people and dig deeper. Find out why it's happening and even more so, is it permanent? And which vendors are going to win and which vendors might lose from it. So that was the whole reason we set up the series of calls. We've only conducted on so far. We have another one this coming Tuesday as well with four entirely new panelists that are going to be from different industry verticals because, as you astutely pointed out, these verticals were very hard hit and not all of them are as hard as others. So it's important to get a wider cross-section. >> So, guys let's take a look at some of the budget impacts the anecdotal evidence that we gathered here. So let me just scan through it and then Erik, I'll ask you to comment. So, you know, like Erik said, some hard hit industries. All major projects, anything sort of next-generation, have been essentially shelved. That was the ISV. And then another one, we cut at least 70% of the big projects moving forward. He mentioned ServiceNow actually calls them out, but the ServiceNow is a SaaS company they'll probably, you know, weather the storm here. But he did say we've put that on hold. The best comment, you know, "As-a-service has Saved our SaaS." (Erik laughs) That one's great. And then we're going to get into some of the networking commentary. Some really interesting things about how to support the work from home. You know, kind of shifting from a hardened top into remote workers. And then a lot of commentary on security. So, you know, that's sort of a high level scan and there's just so much information here Erik, but maybe you could sort of summarize on some of that commentary. >> Yeah, we should definitely dig into each of those sectors a little more, but to summarize what we're seeing here was the real winners and losers are clear. Not everyone was prepared to have a work from home strategy. Not everyone was prepared to send their workers out. Their VPN wasn't, they didn't have enough bandwidth. So there was a real quick uptick in spending, but longer term we're starting to see that these changes will become more permanent. So the real winners and losers right now, we're going to see on the loser's side traditional networking. The MPLS networking is in a lot of trouble according to all the data and the commentary that we're seeing. It's expensive, it's difficult to ramp to up bandwidth as quickly as you need and it doesn't support remote. So we're seeing that lose out and the winners there are in the SD-WAN space. It's going to be impossible to ignore that going forward and some of our CIO and CISO panelists said that change will be permanent. Also, we're seeing, at the same time, what they were calling a "SaaS and Cloud". Now, we know these trends obviously were already happening but they're being exacerbated. They're happening even more quickly and more strong. And I don't see that changing any time soon. That, of course, is at the expense of network, I'm sorry, data centers. Whether it be your own or hosted. Which has huge ramifications on on-prem hardware. Even the firewall providers. So what we're seeing here is obviously we know things are going to be impacted by this situation. We didn't necessarily expect all of our community members and IT decision-makers to talk about them being possibly permanent. So that on a high level was something that was extremely interesting. And the last one that I would bring up is that as we make this shift towards working from home, towards remote access, you also have to align yourself with the security that can support that. And one of the things that we're seeing in our data side on ETR, is a widening bifurcation between the next-generation security vendors and the more traditional security or the legacy security players. That bifurcation just keeps getting wider and wider and this situation could be the last straw. >> So I want to follow up on a couple of those things. You're talking about sort of the network shift you know, towards the SD-WAN. What people have described to me is that they got a, you know, a hardened top. It's a hierarchical network. It's very well understood and it's safe, right? And now all of a sudden you got all those remote workers and so you've got to completely soft of rethink your whole network architecture. The other thing I want to drill into is your Cloud commentary. There's a comment that I saw, Erik, that really stood out. One of the folks said, "I would like to see the data centers "be completely deleted, if you will, or closed down." I think we're going to see, you know, a lot more of this obviously. Not only from the standpoint of, and you heard this a lot, the kind of paid by the drink. But just generally getting rid of all that sort of so-called non-differentiated heavy-lifting as we often hear about. >> That is a extreme comment. I don't think everyone feels that way. But, yes, the comment was made and we've heard the comment from other people. As you and I both know, the larger the enterprise the harder that is to go completely SaaS. But yeah, when a situation like this has and see the inflexibility of their on-prem infrastructure, yes it becomes something that really has to be addressed and it can become a permanent change. I was also shocked about that comment. That gentleman also stated that his executives outside of the ITs area, the CEO, the CFO, had never ever, ever wanted to discuss Cloud. They did not want to discuss work from home. They did not want to discuss remote access. He said that conversation has changed immediately and to the credit of the actual IT companies out there, the technology companies, they're doing everything they can with this opportunity to make that happen. >> Yeah, and so you're right the whole work from home conversation. To your point earlier, Erik, big chunks of COVID, the post-COVID world are going to remain permanent. Guys bring up the SaaS slide if you will. The SaaS commentary, "As-a-Service Saved our SaaS." "The wittiest quip award" going to the ETR. You know, but you had, what's very interesting to hear folks, in fact I think somebody even called out, "Hey," you know, "we expected Oracle to," you know, "be auditing us but they're actually being supportive "as is IBM." Salesforce was an interesting common, Erik. One of the folks said they would share accounts on-prem, but when they all do the work from home they had to actually buy some more. You also got Cisco with big props. Microsoft was called out. A lot of organizations actually allowing them to defer payments. So the SaaS vendors actually got very high marks didn't they? >> They really did and even I wrote that summary and it was difficult to write that about Oracle because we all know that they're infamous for auditing their own customers in 2009 right after we came out of financial crisis. They have notoriously been a-- I don't know if they found religion and they decided to be nice to their customers, but every-single person mentioned them as one of the vendors that was actually helping. That was very shocking. And we all know that when bad situations happen people become opportunistic. And right now it's really seeming that the SaaS vendors understand that they need a longterm relationship with these customers and they're being altruistic instead. Which is really nice. >> Yeah I think that anybody with a Cloud realizes that hey, we have an opportunity here that the lifetime value of that customer, whereas maybe in 2009 when Oracle didn't have a Cloud, they had to get people in a headlock to try to persevere their, you know, income statement. Let's go to the networking drill down guys, that next slide because Fortinet, some of the things we've been reporting on is the sort of divergence in evaluations between Fortinet and Palo Alto before this whole thing hit, Fortinet has done a really good job with its Cloud offerings. Palo Alto struggles a little bit with trying to figure out the sales compensation, is maybe a little bit behind. Although both companies got strong props and I've talked to a number of customers, Palo Alto is going to be in the mix. Fortinet, from a Cloud standpoint, seems to be doing quite well? Obviously networking, Cisco is the big gorilla there. But we also got call outs from guys like Trend Micro which was interesting, from some of the folks. So, your thoughts on this Erik. >> Yeah, I'll start on the networking side because this is something that I've really, I've dug into quite amount, in not only this panel, but a lot of interviews and it really seems as if as networking refresh starts to come up, and it's coming up with a lot of large enterprises, when your network refresh comes up people are going to do an RFP for SD-WAN. They are sick and tired of paying MPLS network vendors and they really want to look at something else. That was even prior to this situation. Now what we're hearing is this is a permanent change. I particularly had one person say, I wanted to find this quote real quickly if I can, but basically they basically saying that, "From a permanency perspective, the freedom from MTLS "will reduce our networks spend by over half "while more than doubling or tripling our bandwidth." You can't ignore that. You're going to save me money and triple my bandwidth, and hey by the way, my refresh is due. It's something that's coming and it's going to happen. And yes, you mentioned the few right? There's Viptela, there's Velocloud, there's some big players like Cisco. The Palo Alto just acquired CloudGenix in the midst of all of this. They just went and got an SD-WAN player themselves. And they just keep acquiring a portfolio to shift from their on-prem to next-generation. It's going to take some time, because 70% plus of their revenues is still on-prem hardware, but I do believe that their portfolio that they're creating is the way the world is moving. And that's just one comment on the traditional networking versus the next-generation SD-WAN. >> And the customers have indicated, you know it's not easy just to get off of their MPLS network. I mean it takes time, it's like slowly pulling of the bandaid. But, like many things, COVID-19 is sort of accelerating that. We haven't talked about digital transformation. That came up as a maybe more strategic initiative. But one that very clearly has legs. >> You know, David, it's very simple. You just said it. People, when things are going well and they're comfortable, they don't change. And that's the same for an enterpriser company. Hey, everything's great, our revenue's fine. Why would we do this? We'll worry about that next year. Then something like this happens and you realize wow, we've been dragging our feet. That digital transformation that we've been talking about, and we've been a little bit slow to accept, we need to accept it, we need to move now. And yes, it was another one of the major themes and it sounds silly for researchers like you and I because we know this is a theme. We know Clouded option is there, we know digital transformation is there. But, there are still a lot of people that haven't moved as quickly as they should and this is going to be that final catalyst to get them there, without a doubt. Quickly on your point of Fortinet, I was actually very impressed with the commentary that came from that because Fortinet is sometimes one of those names that you think of that maybe plays in a smaller pool or isn't as big as some of the 800 pound gorillas out there. But in other other interviews besides this I've heard the phrase coined of "Forti-everything". So through RND and through acquisition, Fortinet has really expanded the portfolio and right now is their time to shine because when you have smaller satellite, you know, offices and branches that you need to connect, they're really, really good at it. And you don't always want to call a Palo Alto and pay that price when you have smaller branch offices. And I actually, I was glad you brought up Fortinet because it's not a name that we get to herald that often and it was deserving from this panel. >> Yeah and, you know, companies that can secure gateways, secure endpoints, obviously going to have momentum. Zscaler came up, you know I think that, and I'll tell ya, looking at, I've done a couple of breaking analysis on security and Fortinet has been strong in two dimensions. You know ETR is, as our audience is I think getting to know. We really look at two key metrics. One is net score, which is a measure of spending momentum, and the other is market share, which is a measure of pervasiveness. And companies like Fortinet, in security, show up on both of those dimensions so it's notable. >> Yes, it certainly is, it is. And I'm glad you brought up Zscaler too. Very recently by client request, we did a very in-depth research on Zscaler versus Palo Alto Prisma Access and they were very interested. This was before all this happened, you know. Does Palo Alto have a chance of catching up, taking share from Zscaler. And I've had the pleasure, myself, personally hosting Jay the CEO of Zscaler at an event in New York City. And I have nothing but incredible respect for the company. But what we found out through this research is Zscaler, at the moment, their technology is still ahead, according to their answers. There's no doubt. However, there doesn't seem to be any real secret sauce that will stop Palo Alto from catching up. So we do believe the parody of feature set will shrink over time. And then it will come down to Palo Alto obviously has a wider and user base. Now, what's happening today might change that. Because if I had to make a decision right now, for my company on secure web gateway, I'm still probably going to go to Zscaler. It's the name. If I had to choose that in a year from now, Palo Alto might have had a better chance. So in this panel, as you brought up, Zscaler was mentioned numerous times as just the wave of the future. Along with CASB brokers right? Whether you're talking about a Netskoper or Forcepointer. All those people that also play in CASB space to secure your access. Zero trust is no longer a marketing-hype term. It is real and it is becoming more real by the week. >> And so, I want to kind of end on one of the other comments that really struck me because we're constantly talking about okay, do you go with a portfolio of a suite of services or do you go with best of breed? What about startups? Are startups more risky in a crisis like this? And one of your panelists, I just love this comment, he said, "One of things that I've always done," he said, "You always hear about the guy, "oh we're going to go to the gardener, we're going to "check out the magic water, we'll pick out three guys "in the upper right hand corner and test them out." He says, "One of the things I always like to do, "I'll pick two from the upper right "and I'll take one from the lower left." One of the emerging, text, "And I'll give em a shot." It won't win every time, but then he called out FireEye as one of the organizations that he found early that gave them competitive advantage. >> Right. >> Love that comment. >> It's a great comment. And honestly if you're in charge of procurement you'd be stupid not to do that. Not only just to see what the technology is, but now I can play you off the big guys because I have negotiating leverage and I can say oh, well I could always just take their contract. So it's silly not to do it from a business perspective. But from technology perspective, what we kept hearing from these people with the smaller vendors. My partner Peter Steube, my colleague and I, we did the host together, we asked this question really believing that the financial insecurity of the moment and the times would make smaller vendors not viable. We heard the exact opposite. What our panelists said was, "No, I'd be happy "to work with a smaller vendor right now "because they're going to give me pricing flexibility, "they're going to work with me right now. "I don't need to pay them upfront "because we're seeing a permanent shift from CapEx to OpEX, "and the smaller vendors are willing to work with me and I can pay them later." So we were actually surprised to hear that and glad to hear it because, to connect to your other point, the other person who was talking about security and the platform approach versus best of breed, he said "Listen, platform approaches you're already "with the vendor, you can bundle a little bit. "But the problem is, if you're just going to acquire "a new technology every time there's a new threat, "the bad guys are just going to switch the threat. "And you can't acquire indefinitely. "So therefore, best of breed with security "will always beat platform." And that's kind of a message to Palo Alto and Cisco, in my opinion, because they seem to be the ones fighting that out. Even Microsoft now, trying to say they're a platform approach in security. >> Well and this says to me the security business, as we predicted, is going to stay fragmented because you're still going to get that best of breed. You know, just like Cloud is going to be fragmented and it's, you know, multiple vendors. Ever since I've been in this business people are trying to consolidate the number of vendors, but technology moves so quickly, it gives competitive advantage. Erik, awesome! Thank you so much for joining us. I'm looking forward to next Tuesday with the next vendor and love to have you back and talk about it anytime. You're a great guest, thanks so much. >> Certainly, I'll do my best to get a better AV connection the next time guys, I apologize for that. But it was great talking to you tonight. >> Hey we're all learning, you know so, thank you everybody for watching, this is Dave Vellante for theCUBE and we'll see you next time. (upbeat music)

(upbeat music) >> Hi, everyone. Welcome to theCUBE's coverage of International Women's Day. I'm John Furrier, host of theCUBE. As part of International Women's Day, we're featuring some of the leading women in business technology from developer to all types of titles and to the executive level. And one topic that's really important is called Getting a Seat at the Table, board makeup, having representation at corporate boards, private and public companies. It's been a big push. And former technology operating executive and corporate board member, she's a board machine Sue Barsamian, formerly with HPE, Hewlett Packard. Sue, great to see you. CUBE alumni, distinguished CUBE alumni. Thank you for coming on. >> Yes, I'm very proud of my CUBE alumni title. >> I'm sure it opens a lot of doors for you. (Sue laughing) We're psyched to have you on. This is a really important topic, and I want to get into the whole, as women advance up, and they're sitting on the boards, they can implement policy and there's governance. Obviously public companies have very strict oversight, and not strict, but like formal. Private boards have to operate, be nimble. They don't have to share all their results. But still, boards play an important role in the success of scaled up companies. So super important, that representation there is key. >> Yes. >> I want to get into that, but first, before we get started, how did you get into tech? How did it all start for you? >> Yeah, long time ago, I was an electrical engineering major. Came out in 1981 when, you know, opportunities for engineering, if you were kind, I went to Kansas State as an undergrad, and basically in those days you went to Texas and did semiconductors. You went to Atlanta and did communication satellites. You went to Boston or you went to Silicon Valley. And for me, that wasn't too hard a choice. I ended up going west and really, I guess what, embarked on a 40 year career in Silicon Valley and absolutely loved it. Largely software, but some time on the hardware side. Started out in networking, but largely software. And then, you know, four years ago transitioned to my next chapter, which is the corporate board director. And again, focused on technology software and cybersecurity boards. >> For the folks watching, we'll cut through another segment we can probably do about your operating career, but you rose through the ranks and became a senior operating executive at the biggest companies in the world. Hewlett Packard Enterprise, Hewlett Packard Enterprise and others. Very great career, okay. And so now you're kind of like, put that on pause, and you're moving on to the next chapter, which is being a board director. What inspired you to be a board director for multiple public companies and multiple private companies? Well, how many companies are you on? But what's the inspiration? What's the inspiration? First tell me how many board ships you're on, board seats you're on, and then what inspired you to become a board director? >> Yeah, so I'm on three public, and you are limited in terms of the number of publics that you can do to four. So I'm on three public, and I'm on four private from a tech perspective. And those range from, you know, a $4 billion in revenue public company down to a 35 person private company. So I've got the whole range. >> So you're like freelancing, I mean, what is it like? It's a full-time job, obviously. It's a lot of work involved. >> Yeah, yeah, it's. >> John: Why are you doing it? >> Well, you know, so I retired from being an operating executive after 37 years. And, but I loved, I mean, it's tough, right? It's tough these days, particularly with all the pressures out there in the market, not to mention the pandemic, et cetera. But I loved it. I loved working. I loved having a career, and I was ready to back off on, I would say the stresses of quarterly results and the stresses of international travel. You have so much of it. But I wasn't ready to back off from being involved and engaged and continuing to learn new things. I think this is why you come to tech, and for me, why I went to the valley to begin with was really that energy and that excitement, and it's like it's constantly reinventing itself. And I felt like that wasn't over for me. And I thought because I hadn't done boards before I retired from operating roles, I thought, you know, that would fill the bill. And it's honestly, it has exceeded expectations. >> In a good way. You feel good about where you're at and. >> Yeah. >> What you went in, what was the expectation going in and what surprised you? And were there people along the way that kind of gave you some pointers or don't do this, stay away from this. Take us through your experiences. >> Yeah, honestly, there is an amazing network of technology board directors, you know, in the US and specifically in the Valley. And we are all incredibly supportive. We have groups where we get together as board directors, and we talk about topics, and we share best practices and stories, and so I underestimated that, right? I thought I was going to, I thought I was going to enter this chapter where I would be largely giving back after 37 years. You've learned a little bit, right? What I underestimated was just the power of continuing to learn and being surrounded by so many amazing people. When, you know, when you do, you know, multiple boards, your learnings are just multiplied, right? Because you see not just one model, but you see many models. You see not just one problem, but many problems. Not just one opportunity, but many opportunities. And I underestimated how great that would be for me from a learning perspective and then your ability to share from one board to the other board because all of my boards are companies who are also quite close to each other, the executives collaborate. So that has turned out to be really exciting for me. >> So you had the stressful job. You rose to the top of the ranks, quarterly shot clock earnings, and it's hard charging. It's like, it's like, you know, being an athlete, as we say tech athlete. You're a tech athlete. Now you're taking that to the next level, which is now you're juggling multiple operational kind of things, but not with super pressure. But there's still a lot of responsibility. I know there's one board, you got compensation committee, I mean there's work involved. It's not like you're clipping coupons and having pizza. >> Yeah, no, it's real work. Believe me, it's real work. But I don't know how long it took me to not, to stop waking up and looking at my phone and thinking somebody was going to be dropping their forecast, right? Just that pressure of the number, and as a board member, obviously you are there to support and help guide the company and you feel, you know, you feel the pressure and the responsibility of what that role entails, but it's not the same as the frontline pressure every quarter. It's different. And so I did the first type. I loved it, you know. I'm loving this second type. >> You know, the retirement, it's always a cliche these days, but it's not really like what people think it is. It's not like getting a boat, going fishing or whatever. It's doing whatever you want to do, that's what retirement is. And you've chose to stay active. Your brain's being tested, and you're working it, having fun without all the stress. But it's enough, it's like going the gym. You're not hardcore workout, but you're working out with the brain. >> Yeah, no, for sure. It's just a different, it's just a different model. But the, you know, the level of conversations, the level of decisions, all of that is quite high. Which again, I like, yeah. >> Again, you really can't talk about some of the fun questions I want to ask, like what's the valuations like? How's the market, your headwinds? Is there tailwinds? >> Yes, yes, yes. It's an amazing, it's an amazing market right now with, as you know, counter indicators everywhere, right? Something's up, something's down, you know. Consumer spending's up, therefore interest rates go up and, you know, employment's down. And so or unemployment's down. And so it's hard. Actually, I really empathize with, you know, the, and have a great deal of respect for the CEOs and leadership teams of my board companies because, you know, I kind of retired from operating role, and then everybody else had to deal with running a company during a pandemic and then running a company through the great resignation, and then running a company through a downturn. You know, those are all tough things, and I have a ton of respect for any operating executive who's navigating through this and leading a company right now. >> I'd love to get your take on the board conversations at the end if we have more time, what the mood is, but I want to ask you about one more thing real quick before we go to the next topic is you're a retired operating executive. You have multiple boards, so you've got your hands full. I noticed there's a lot of amazing leaders, other female tech athletes joining boards, but they also have full-time jobs. >> Yeah. >> And so what's your advice? Cause I know there's a lot of networking, a lot of sharing going on. There's kind of a balance between how much you can contribute on the board versus doing the day job, but there's a real need for more women on boards, so yet there's a lot going on boards. What's the current state of the union if you will, state of the market relative to people in their careers and the stresses? >> Yeah. >> Cause you left one and jumped in all in there. >> Yeah. >> Some can't do that. They can't be on five boards, but they're on a few. What's the? >> Well, and you know, and if you're an operating executive, you wouldn't be on five boards, right? You would be on one or two. And so I spend a lot of time now bringing along the next wave of women and helping them both in their career but also to get a seat at the table on a board. And I'm very vocal about telling people not to do it the way I do it. There's no reason for it to be sequential. You can, you know, I thought I was so busy and was traveling all the time, and yes, all of that was true, but, and maybe I should say, you know, you can still fit in a board. And so, and what I see now is that your learnings are so exponential with outside perspective that I believe I would've been an even better operating executive had I done it earlier. I know I would've been an even better operating executive had I done it earlier. And so my advice is don't do it the way I did it. You know, it's worked out fine for me, but hindsight's 2020, I would. >> If you can go back and do a mulligan or a redo, what would you do? >> Yeah, I would get on a board earlier, full stop, yeah. >> Board, singular, plural? >> Well, I really, I don't think as an operating executive you can do, you could do one, maybe two. I wouldn't go beyond that, and I think that's fine. >> Yeah, totally makes sense. Okay, I got to ask you about your career. I know technical, you came in at that time in the market, I remember when I broke into the business, very male dominated, and then now it's much better. When you went through the ranks as a technical person, I know you had some blockers and definitely some, probably some people like, well, you know. We've seen that. How did you handle that? What were some of the key pivot points in your journey? And we've had a lot of women tell their stories here on theCUBE, candidly, like, hey, I was going to tell that professor, I'm going to sit in the front row. I'm going to, I'm getting two degrees, you know, robotics and aerospace. So, but they were challenged, even with the aspiration to do tech. I'm not saying that was something that you had, but like have you had experience like that, that you overcome? What were those key points and how did you handle them and how does that help people today? >> Yeah, you know, I have to say, you know, and not discounting that obviously this has been a journey for women, and there are a lot of things to overcome both in the workforce and also just balancing life honestly. And they're all real. There's also a story of incredible support, and you know, I'm the type of person where if somebody blocked me or didn't like me, I tended to just, you know, think it was me and like work harder and get around them, and I'm sure that some of that was potentially gender related. I didn't interpret it that way at the time. And I was lucky to have amazing mentors, many, many, many of whom were men, you know, because they were in the positions of power, and they made a huge difference on my career, huge. And I also had amazing female mentors, Meg Whitman, Ann Livermore at HPE, who you know well. So I had both, but you know, when I look back on the people who made a difference, there are as many men on the list as there are women. >> Yeah, and that's a learning there. Create those coalitions, not just one or the other. >> Yeah, yeah, yeah, absolutely. >> Well, I got to ask you about the, well, you brought up the pandemic. This has come up on some interviews this year, a little bit last year on the International Women's Day, but this year it's resonating, and I would never ask in an interview. I saw an interview once where a host asked a woman, how do you balance it all? And I was just like, no one asked men that. And so it's like, but with remote work, it's come up now the word empathy around people knowing each other's personal situation. In other words, when remote work happened, everybody went home. So we all got a glimpse of the backdrop. You got, you can see what their personal life was on Facebook. We were just commenting before we came on camera about that. So remote work really kind of opened up this personal side of everybody, men and women. >> Yeah. >> So I think this brings this new empathy kind of vibe or authentic self people call it. Is remote work an opportunity or a threat for advancement of women in tech? >> It's a much debated topic. I look at it as an opportunity for many of the reasons that you just said. First of all, let me say that when I was an operating executive and would try to create an environment on my team that was family supportive, I would do that equally for young or, you know, early to mid-career women as I did for early to mid-career men. And the reason is I needed those men, you know, chances are they had a working spouse at home, right? I needed them to be able to share the load. It's just as important to the women that companies give, you know, the partner, male or female, the partner support and the ability to share the love, right? So to me it's not just a woman thing. It's women and men, and I always tried to create the environment where it was okay to go to your soccer game. I knew you would be online later in the evening when the kids were in bed, and that was fine. And I think the pandemic has democratized that and made that, you know, made that kind of an everyday occurrence. >> Yeah the baby walks in. They're in the zoom call. The dog comes in. The leaf blower going on the outside the window. I've seen it all on theCUBE. >> Yeah, and people don't try to pretend anymore that like, you know, the house is clean, the dog's behaved, you know, I mean it's just, it's just real, and it's authentic, and I think that's healthy. >> Yeah. >> I do, you know, I also love, I also love the office, and you know, I've got a 31 year old and a soon to be 27 year old daughter, two daughters. And you know, they love going into the office, and I think about when I was their age, how just charged up I would get from being in the office. I also see how great it is for them to have a couple of days a week at home because you can get a few things done in between Zoom calls that you don't have to end up piling onto the weekend, and, you know, so I think it's a really healthy, I think it's a really healthy mix now. Most tech companies are not mandating five days in. Most tech companies are at two to three days in. I think that's a, I think that's a really good combination. >> It's interesting how people are changing their culture to get together more as groups and even events. I mean, while I got you, I might as well ask you, what's the board conversations around, you know, the old conferences? You know, before the pandemic, every company had like a user conference. Right, now it's like, well, do we really need to have that? Maybe we do smaller, and we do digital. Have you seen how companies are handling the in-person? Because there's where the relationships are really formed face-to-face, but not everyone's going to be going. But now certain it's clearly back to face-to-face. We're seeing that with theCUBE as you know. >> Yeah, yeah. >> But the numbers aren't coming back, and the numbers aren't that high, but the stakeholders. >> Yeah. >> And the numbers are actually higher if you count digital. >> Yeah, absolutely. But you know, also on digital there's fatigue from 100% digital, right? It's a hybrid. People don't want to be 100% digital anymore, but they also don't want to go back to the days when everybody got on a plane for every meeting, every call, every sales call. You know, I'm seeing a mix on user conferences. I would say two-thirds of my companies are back, but not at the expense level that they were on user conferences. We spend a lot of time getting updates on, cause nobody has put, interestingly enough, nobody has put T&E, travel and expense back to pre-pandemic levels. Nobody, so everybody's pulled back on number of trips. You know, marketing events are being very scrutinized, but I think very effective. We're doing a lot of, and, you know, these were part of the old model as well, like some things, some things just recycle, but you know, there's a lot of CIO and customer round tables in regional cities. You know, those are quite effective right now because people want some face-to-face, but they don't necessarily want to get on a plane and go to Las Vegas in order to do it. I mean, some of them are, you know, there are a lot of things back in Las Vegas. >> And think about the meetings that when you were an operating executive. You got to go to the sales kickoff, you got to go to this, go to that. There were mandatory face-to-faces that you had to go to, but there was a lot of travel that you probably could have done on Zoom. >> Oh, a lot, I mean. >> And then the productivity to the family impact too. Again, think about again, we're talking about the family and people's personal lives, right? So, you know, got to meet a customer. All right. Salesperson wants you to get in front of a customer, got to fly to New York, take a red eye, come on back. Like, I mean, that's gone. >> Yeah, and oh, by the way, the customer doesn't necessarily want to be in the office that day, so, you know, they may or may not be happy about that. So again, it's and not or, right? It's a mix. And I think it's great to see people back to some face-to-face. It's great to see marketing and events back to some face-to-face. It's also great to see that it hasn't gone back to the level it was. I think that's a really healthy dynamic. >> Well, I'll tell you that from our experience while we're on the topic, we'll move back to the International Women's Day is that the productivity of digital, this program we're doing is going to be streamed. We couldn't do this face-to-face because we had to have everyone fly to an event. We're going to do hundreds of stories that we couldn't have done. We're doing it remote. Because it's better to get the content than not have it. I mean it's offline, so, but it's not about getting people to the event and watch the screen for seven hours. It's pick your interview, and then engage. >> Yeah. >> So it's self-service. So we're seeing a lot, the new user experience kind of direct to consumer, and so I think there will be an, I think there's going to be a digital first class citizen with events, so that that matches up with the kind of experience, but the offline version. Face-to-face optimized for relationships, and that's where the recruiting gets done. That's where, you know, people can build these relationships with each other. >> Yeah, and it can be asynchronous. I think that's a real value proposition. It's a great point. >> Okay, I want to get, I want to get into the technology side of the education and re-skilling and those things. I remember in the 80s, computer science was software engineering. You learned like nine languages. You took some double E courses, one or two, and all the other kind of gut classes in school. Engineering, you had the four class disciplines and some offshoots of specialization. Now it's incredible the diversity of tracks in all engineering programs and computer science and outside of those departments. >> Yeah. >> Can you speak to the importance of STEM and the diversity in the technology industry and how this brings opportunity to lower the bar to get in and how people can stay in and grow and keep leveling up? >> Yeah, well look, we're constantly working on how to, how to help the incoming funnel. But then, you know, at a university level, I'm on the foundation board of Kansas State where I got my engineering degree. I was also Chairman of the National Action Council for Minorities in Engineering, which was all about diversity in STEM and how do you keep that pipeline going because honestly the US needs more tech resources than we have. And if you don't tap into the diversity of our entire workforce, we won't be able to fill that need. And so we focused a lot on both the funnel, right, that starts at the middle school level, particularly for girls, getting them in, you know, the situation of hands-on comfort level with coding, with robot building, you know, whatever gives them that confidence. And then keeping that going all the way into, you know, university program, and making sure that they don't attrit out, right? And so there's a number of initiatives, whether it's mentoring and support groups and financial aid to make sure that underrepresented minorities, women and other minorities, you know, get through the funnel and stay, you know, stay in. >> Got it. Now let me ask you, you said, I have two daughters. You have a family of girls too. Is there a vibe difference between the new generation and what's the trends that you're seeing in this next early wave? I mean, not maybe, I don't know how this is in middle school, but like as people start getting into their adult lives, college and beyond what's the current point of view, posture, makeup of the talent coming in? >> Yeah, yeah. >> Certain orientations, do you see any patterns? What's your observation? >> Yeah, it's interesting. So if I look at electrical engineering, my major, it's, and if I look at Kansas State, which spends a lot of time on this, and I think does a great job, but the diversity of that as a major has not changed dramatically since I was there in the early 80s. Where it has changed very significantly is computer science. There are many, many university and college programs around the country where, you know, it's 50/50 in computer science from a gender mix perspective, which is huge progress. Huge progress. And so, and to me that's, you know, I think CS is a fantastic degree for tech, regardless of what function you actually end up doing in these companies. I mean, I was an electrical engineer. I never did core electrical engineering work. I went right into sales and marketing and general management roles. So I think, I think a bunch of, you know, diverse CS graduates is a really, really good sign. And you know, we need to continue to push on that, but progress has been made. I think the, you know, it kind of goes back to the thing we were just talking about, which is the attrition of those, let's just talk about women, right? The attrition of those women once they got past early career and into mid-career then was a concern, right? And that goes back to, you know, just the inability to, you know, get it all done. And that I am hopeful is going to be better served now. >> Well, Sue, it's great to have you on. I know you're super busy. I appreciate you taking the time and contributing to our program on corporate board membership and some of your story and observations and opinions and analysis. Always great to have you and call you a contributor for theCUBE. You can jump on on one more board, be one of our board contributors for our analysts. (Sue laughing) >> I'm at capacity. (both laughing) >> Final, final word. What's the big seat at the table issue that's going well and areas that need to be improved? >> So I'll speak for my boards because they have made great progress in efficiency. You know, obviously with interest rates going up and the mix between growth and profitability changing in terms of what investors are looking for. Many, many companies have had to do a hard pivot from grow at all costs to healthy balance of growth and profit. And I'm very pleased with how my companies have made that pivot. And I think that is going to make much better companies as a result. I think diversity is something that has not been solved at the corporate level, and we need to keep working it. >> Awesome. Thank you for coming on theCUBE. CUBE alumni now contributor, on multiple boards, full-time job. Love the new challenge and chapter you're on, Sue. We'll be following, and we'll check in for more updates. And thank you for being a contributor on this program this year and this episode. We're going to be doing more of these quarterly, so we're going to move beyond once a year. >> That's great. (cross talking) It's always good to see you, John. >> Thank you. >> Thanks very much. >> Okay. >> Sue: Talk to you later. >> This is theCUBE coverage of IWD, International Women's Day 2023. I'm John Furrier, your host. Thanks for watching. (upbeat music)

>> TheCUBE's live coverage is made possible by funding from Dell Technologies. Creating technologies that drive human progress. (upbeat music) >> Welcome back to Spain, everybody. We're here at the Theater Live and MWC 23. You're watching theCUBE's Continuous Coverage. This is day two. I'm Dave Vellante with my co-host, Dave Nicholson. Lisa Martin is also in the house. John Furrier out of our Palo Alto studio covering all the news. Check out silicon angle.com. Okay, we're going to dig into the core infrastructure here. We're going to talk a little bit about servers. Manya Rastogi is here. She's in technical marketing at Dell Technologies. And Abdel Bagegni is technical program manager at the Telecom Infra Project. Folks, welcome to theCUBE. Good to see you. >> Thank you. >> Abdel, what is the Telecom Infras Project? Explain to our audience. >> Yeah. So the Telecom Infra Project is a US based non-profit organization community that brings together different participants, suppliers, vendors, operators SI's together to accelerate the adoption of open RAN and open interface solutions across the globe. >> Okay. So that's the mission is open RAN adoption. And then how, when was it formed? Give us the background and some of the, some of the milestones so far. >> Yeah. So the telecom infra project was established five years ago from different vendor leaders and operators across the globe. And then the mission was to bring different players in to work together to accelerate the adoption of, of open RAN. Now open RAN has a lot of potential and opportunities, but in the same time there's challenges that we work together as a community to facilitate those challenges and overcome those barriers. >> And we've been covering all week just the disaggregation of the network. And you know, we've seen this movie sort of before playing out now in, in telecom. And Manya, this is obviously a compute intensive environment. We were at the Dell booth earlier this morning poking around, beautiful booth, lots of servers. Tell us what your angle is here in this marketplace. >> Yeah, so I would just like to say that Dell is kind of leading or accelerating the innovation at the telecom edge with all these ruggedized servers that we are offering. So just continuing the mission, like Abdel just mentioned for the open RAN, that's where a lot of focus will be from these servers will be, so XR 8000, it's it's going to be one of the star servers for telecom with, you know, offering various workloads. So it can be rerun, open run, multi access, edge compute. And it has all these different features with itself and the, if we, we can talk more about the performance gains, how it is based on the Intel CPUs and just try to solve the purpose like along with various vendors, the whole ecosystem solve this challenge for the open RAN. >> So Manya mentioned some of those infrastructure parts. Does and do, do you say TIP or T-I-P for short? >> Abdel: We say TIP. >> TIP. >> Abdel: T-I-P is fine as well. >> Does, does, does TIP or T-I-P have a certification process or a, or a set of guidelines that someone like Dell would either adhere to or follow to be sort of TIP certified? What does that look like? >> Yeah, of course. So what TIP does is TIP accredits what solutions that actually work in a real commercial grade environment. So what we do is we bring the different players together to come up with the most efficient optimized solution. And then it goes through a process that the community sets the, the, the criteria for and accepts. And then once this is accredited it goes into TIP exchange for other operators and the participants and the industry to adopt. So it's a well structured process and it's everything about how we orchestrate the industry to come together and set those requirements and and guidelines. Everything starts with a use case from the beginning. It's based on operators requirements, use cases and then those use cases will be translated into a solution that the industry will approve. >> So when you say operator, I can think of that sort of traditionally as the customer side of things versus the vendor side of things. Typically when organizations get together like TIP, the operator customer side is seeking a couple of things. They want perfect substitutes in all categories so that they could grind vendors down from a price perspective but they also want amazing innovation. How do you, how do you deliver both? >> Yeah, I mean that's an excellent question. We be pragmatic and we bring all players in one table to discuss. MNO's want this, vendors can provide a certain level and we bring them together and they discuss and come up with something that can be deployed today and future proof for the future. >> So I've been an enterprise technology observer for a long time and, you know, I saw the, the attempt to take network function virtualization which never really made much of an impact, but it was a it was the beginning of the enterprise players really getting into this market. And then I would see companies, whether it was Dell or HPE or Cisco, they'd take an X 86 server, put a cool name on it, edge something, and throw it over the fence and that didn't work so well. Now it's like, Manya. We're starting to get serious. You're building relationships. >> Manya: Totally. >> I mentioned we were at the Dell booth you're actually building purpose built systems now for this, this segment. Tell us what's different about this market and the products that you're developing for this market than say the commercial enterprise. >> So you are absolutely right, like, you know, kind of thinking about the journey, there has been a lot of, it has been going for a long time for all these improvements and towards going more open disaggregated and overall that kind of environment and what Dell brings together with our various partners and particularly if you talk about Intel. So these servers are powered by the players four gen intel beyond processors. And so what Intel is doing right now is providing us with great accelerators like vRAN Boost. So it increases performance like doubles what it was able to do before. And power efficiency, it has been an issue for a long, long time and it still continues but there is some improvement. For example 20% reduction overall with the power savings. So that's a step forward in that direction. And then we have done some of our like own testing as well with these servers and continuing that, you know it's not just telecom but also going towards Edge or inferencing like all these comes together not just X 30,000 but for example XR 56 10, 70, 76 20. So these are three servers which combines together to like form telecom and Edge and covers altogether. So that's what it is. >> Great, thank you. So Abdel, I mean I think generally people agree that in the fullness of time all radio access networks are going to be open, right? It's just a matter of okay, how do we get there? How do we make sure that it has the same, you know, quality of service characteristics. So where are we on on that, that journey from your perspective? And, and maybe you could project what, what it's going to look like over this decade. 'Cause it's going to take, you know, years. >> It's going to take a bit of time to mature and be a kind of a plug and play different units together. I think there was a lot, there was a, was a bit of over-promising in a few, in the last few years on the acceleration of open RAN deployment. That, well, a TIP is trying to do is trying to realize the pragmatic approach of the open run deployment. Now we know the innovation cannot happen when you have a kind of closed interfaces when you allow small players to be within the market and bring the value to, to the RAN areas. This is where the innovation happens. I think what would happen on the RAN side of things is that it would be driven by use cases and the operators. And the minute that the operators are no longer can depend on the closed interface vendors because there's use cases that fulfill that are requires some open RAN functionality, be the, the rig or the SMO layers and the different configurations of the rUSE getting the servers to the due side of things. This kind of modular scalability on this layer is when the RAN will, the Open RAN, would boost. This would happen probably, yeah. >> Go ahead. >> Yeah, it would happen in, in the next few years. Not next year or the year after but definitely something within the four to five years from now. >> I think it does feel like it's a second half of the decade and you feel like the, the the RAN intelligent controller is going to be a catalyst to actually sort of force the world into this open environment. >> Let's say that the Rick and the promises that were given to, to the sun 10 years ago, the Rick is realizing it and the closed RAN vendors are developing a lot on the Rick side more than the other parts of the, of the open RAN. So it will be a catalyst that would drive the innovation of open RAN, but only time will tell. >> And there are some naysayers, I mean I've seen some you know, very, very few, but I've seen some works that, oh the economics aren't there. It'll, it'll never get there. What, what do you, what do you say to that? That, that it won't ever, open RAN won't ever be as cost effective as you know, closed networks. >> Open RAN will open innovations that small players would have the opportunity to contribute to the, to the RAN space. This opportunity is not given to small players today. Open RAN provides this kind of opportunity and given that it's a path for innovation, then I would say that, you know, different perspectives some people are making sure that, you know the status quo is the way forward. But it would certainly put barriers on on innovation and this is not the way forward. >> Yeah. You can't protect the past in the future. My own personal opinion is, is that it doesn't have to be comparable from a, from a TCO perspective it can be close enough. It's the innovative, same thing with like you watch the, the, the adoption of Cloud. >> Exactly. >> Like cloud was more expensive it's always more expensive to rent, but people seem to be doing public Cloud, you know, because of the the innovation capabilities and the developer capabilities. Is that a fair analogy in this space, do you think? >> I mean this is what all technologies happens. >> Yeah. >> Right? It starts with a quite costly and then the the cost will start dropping down. I mean the, the cost of, of a megabyte two decades ago is probably higher than what it costly terabyte. So this is how technology evolves and it's any kind of comparison, either copper or even the old generation, the legacy generations could be a, a valid comparison. However, they need to be at a market demand for something like that. And I think the use cases today with what the industry is is looking for have that kind of opportunity to pull this kind of demand. But, but again, it needs to go work close by the what happens in the technology space, be it, you know we always talk about when we, we used to talk about 5G, there was a lot of hypes going on there. But I think once it realized in, in a pragmatic, in a in a real life situation, the minutes that governments decide to go for autonomous vehicles, then you would have limitations on the current closed RAN infrastructures and you would definitely need something to to top it up on the- >> I mean, 5G needs open RAN, I mean that's, you know not going to happen without it. >> Exactly. >> Yeah, yeah. But, but what is, but what would you say the most significant friction is between here and the open RAN nirvana? What are, what are the real hurdles that need to be overcome? There's obviously just the, I don't want to change we've been doing this the same way forever, but what what are the, what are the real, the legitimate concerns that people have when we start talking about open RAN? >> So I think from a technology perspective it will be solved. All of the tech, I mean there's smart engineers in the world today that will fix, you know these kind of problems and all of the interability, interruptability issues and, and all of that. I think it's about the mindset, the, the interfaces between the legacy core and RAN has been became more fluid today. We don't have that kind of a hard line between these kind of different aspects. We have the, the MEC coming closer to the RAN, we have the RAN coming closer to the Core, and we have the service based architectures in the Core. So these kind of things make it needs a paradigm shift between how operators that would need to tackle the open RAN space. >> Are there specific deployment requirements for open RAN that you can speak to from your perspective? >> For sure and going in this direction, like, you know evolution with the technology and how different players are coming together. Like that's something I wanted to comment from the previous question. And that's where like, you know these servers that Dell is offering right now. Specific functionality requirements, for example, it's it's a small server, it's short depth just 430 millimeters of depth and it can fit anywhere. So things like small form factor, it's it's crucial because if you, it can replace like multiple servers 10 years ago with just one server and you can place it like near a base band unit or to a cell site on top of a roof wherever. Like, you know, if it's a small company and you need this kind of 5G connection it kind of solves that challenge with this server. And then there are various things like, you know increasing thermals for example temperatures. It is classified like, you know kind of compliant with the negative 5 to 55 degree Celsius. And then we are also moving towards, for example negative 20 to 65 degree Celsius. Which is, which is kind of great because in situations where, which are out of our hands and you need specific thermals for those situations that's where it can solve that problem. >> Are those, are those statistics in those measurements different than the old NEB's standards, network equipment building standards? Or are they, are they in line with that? >> It is, it is a next step. Like so most of our servers that we have right now are negative five to five degree Celsius, for especially the extremely rugged server series and this one XR 8,000 which is focused for the, it's telecom inspired so it's focused on those customers. So we are trying to come up like go a step ahead and also like offering this additional temperatures testing and yeah compliance. So, so it is. >> Awesome. So we, I said we were at the booth early today. Looks like some good traffic people poking around at different, you know, innovations you got going. Some of the private network stuff is kind of cool. I'm like how much does that cost? I think I might like one of those, you know, but- >> [Private 5G home network. >> Right? Why not? Guys, great to have you on the show. Thanks so much for sharing. Appreciate it. >> Thank you. >> Thank you so much. >> Okay. For Dave Nicholson and Lisa Martin this is Dave Vellante, theCUBE's coverage. MWC 23 live from the Fida in Barcelona. We'll be right back. (outro music)

(gentle music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase. The theme is Data as Code; The Future of Enterprise Data and Analytics. This is the season two, episode two of the ongoing series of covering the exciting startups in the AWS ecosystem around data analytics and cloud computing. I'm your host, John Furrier. Today we're joined by great guests here. Three guests. Wen Phan, who's a Director of Product Management at Ahana, Satyam Krishna, Engineering Manager at Blinkit, and we have Akshay Agarwal, Senior Engineer at Blinkit as well. We're going to get into the relationship there. Let's get into. We're going to talk about how Blinkit's using open data lake, data house with Presto on AWS. Gentlemen, thanks for joining us. >> Thanks for having us. >> So we're going to get into the deep dive on the open data lake, but I want to just quickly get your thoughts on what it is for the folks out there. Set the table. What is the open data lakehouse? Why it is important? What's in it for the customers? Why are we seeing adoption around this because this is a big story. >> Sure. Yeah, the open data lakehouse is really being able to run a gamut of analytics, whether it be BI, SQL, machine learning, data science, on top of the data lake, which is based on inexpensive, low cost, scalable storage. And more importantly, it's also on top of open formats. And this to the end customer really offers a tremendous range of flexibility. They can run a bunch of use cases on the same storage and great price performance. >> You guys have any other thoughts on what's your reaction to the lakehouse? What is your experience with it? What's going on with Blinkit? >> No, I think for us also, it has been the primary driver of how as a company we have shifted our completely delivery model from us delivering in one day to someone who is delivering in 10 minutes, right? And a lot of this was made possible by having this kind of architecture in place, which helps us to be more open-source, more... where the tools are open-source, we have an open table format which helps us be very modular in nature, meaning we can pick solutions which works best for us, right? And that is the kind of architecture that we want to be in. >> Awesome. Wen, you know last time we chat with Ahana, we had a great conversation around Presto, data. The theme of this episode is Data as Code, which is interesting because in all the conversations in these episodes all around developers, which administrators are turning into developers, there's a developer vibe with data. And with opensource, it's software. Now you've got data taking a similar trajectory as how software development was with code, but the people running data they're not developers, they're administrators, they're operators. Now they're turning into DataOps. So it's kind of a similar vibe going on with branches and taking stuff out of and putting it back in, and testing it. Datasets becoming much more stable, iterating on machine learning algorithm. This is a movement. What's your guys reaction before we get into the relationships here with you guys. But, what's your reaction to this Data as Code movement? >> Yeah, so I think the folks at Blinkit are doing a great job there. I mean, they have a pretty compact data engineering team and they have some pretty stringent SLAs, as well as in terms of time to value and reliability. And what that ultimately translates for them is not only flexibility but reliability. So they've done some very fantastic work on a lot of automation, a lot of integration with code, and their data pipelines. And I'm sure they can give the details on that. >> Yes. Satyam and Akshay, you guys are engineers' software, but this is becoming a whole another paradigm where the frontline coding and or work or engineer data engineering is implementing the operations as well. It's kind of like DevOps for data. >> For sure. Right. And I think whenever you're working, even as a software engineer, the understanding of business is equally important. You cannot be working on something and be away from business, right? And that's where, like I mentioned earlier, when we realized that we have to completely move our stack and start giving analytics at 10 minutes, right. Because when you're delivering in 10 minutes, your leaders want to take decisions in your real-time. That means you need to move with them. You need to move with business. And when you do that, the kind of flexibility these softwares give is what enables the businesses at the end of the day. >> Awesome. This is the really kind of like, is there going to be a book called agile data warehouses? I don't think so. >> I think so. (laughing) >> The agile cloud data. This is cool. So let's get into what you guys do. What is Blinkit up to? What do you guys do? Can you take a minute to explain the company and your product? >> Sure. I'll take that. So Blinkit is India's biggest 10 minute delivery platform. It pioneered the delivery model in the country with over 10 million Indian shopping on our platform, ranging from everything: grocery staples, vegetables, emergency services, electronics, and much more, right. It currently delivers over 200,000 orders every day, and is in a hurry to bring the future of farmers to everyone in India. >> What's the relationship with Ahana and Blinkit? Wen, what's the tie in? >> Yeah, so Blinkit had a pretty well formed stack. They needed a little bit more flexibility and control. They thought a managed service was the way to go. And here at Ahana, we provide a SaaS managed service for Presto. So they engaged us and they evaluated our offering. And more importantly, we're able to partner. As a early stage startup, we really rely on very strong partners with great use cases that are willing to collaborate. And the folks at Blinkit have been really great in helping us push our product, develop our product. And we've been very happy about the value that we've been able to deliver to them as well. >> Okay. So let's unpack the open data lakehouse. What is it? What's under the covers? Let's get into it. >> Sure. So if bring up a slide. Like I said before, it's really a paradigm on being able to run a gamut of analytics on top of the open data lake. So what does that mean? How did it come about? So on the left hand side of the slide, we are coming out of this world where for the last several decades, the primary workhorse for SQL based processing and reporting and dashboarding use cases was really the data warehouse. And what we're seeing is a shift due to the trends in inexpensive scalable storage, cloud storage. The proliferation of open formats to facilitate using this storage to get certain amounts of reliability and performance, and the adoption of frameworks that can operate on top of this cloud data lake. So while here at Ahana, we're primarily focused on SQL workloads and Presto, this architecture really allows for other types of frameworks. And you see the ML and AI side. And like to Satyam's point earlier, offers a great amount of flexibility modularity for many use cases in the cloud. So really, that's really the lakehouse, and people like it for the performance, the openness, and the price performance. >> How's the open-source open side of it playing in the open-source? It's kind of open formats. What is the open-source angle on this because there's a lot of different approaches. I'm hearing open formats. You know, you have data stores which are a big part of seeing that. You got SQL, you mentioned SQL. There's got a mishmash of opportunities. Is it all coexisting? Is it one tool to rule the world or is it interchangeable? What's the open-source angle? >> There's multiple angles and I'll let definitely Satyam add to what I'm saying. This was definitely a big piece for Blinkit. So on one hand, you have the open formats. And what really the open formats enable is multiple compute engines to work on that data. And that's very huge. 'Cause it's open, you're not locked in. I think the other part of open that is important and I think it was important to Blinkit was the governance around that. So in particular Presto is governed by the Linux Foundation. And so, as a customer of open-source technology, they want some assurances for things like how's it governed? Is the license going to change? So there's that aspect of openness that I think is very important. >> Yeah. Blinkit, what's the data strategy here with lakehouse and you guys? Why are you adopting this type of architecture? >> So adding to what... Yeah, I think adding to Wen said, right. When we are thinking in terms of all these OpenStacks, you have got these open table formats, everything which is deployed over cloud, the primary reason there is modularity. It's as simple as that, right. You can plug and play so many different table formats from one thing to another based on the use case that you're trying to serve, so that you get the most value out of data. Right? I'll give you a very simple example. So for us we use... not even use one single table format. It's not that one thing solves for everything, right? We use both Hudi and Iceberg to solve for different use cases. One is good for when you're working for a certain data site. Icebergs works well when you're in the SQL kind of interface, right. Hudi's still trying to reach there. It's going to go there very soon. So having the ability to plug and play different formats based on the use case helps you to grow faster, helps you to take decisions faster because you now you're not stuck on one thing. They will have to implement it. Right. So I think that's what it is great about this data lake strategy. Keeping yourself cost effective. Yeah, please. >> So the enablement is basically use case driven. You don't have to be rearchitecturing for use cases. You can simply plug can play based on what you need for the use case. >> Yeah. You can... and again, you can focus on your business use case. You can figure out what your business users need and not worry about these things because that's where Presto comes in, helps you stitch that data together with multiple data formats, give you the performance that you need and it works out the best there. And that's something that you don't get to with traditional warehouse these days. Right? The kind of thing that we need, you don't get that. >> I do want to add. This is just to riff on what Satyam said. I think it's pretty interesting. So, it really allowed him to take the best-of-breed of what he was seeing in the community, right? So in the case of table formats, you've got Delta, you've got Hudi, you've got Iceberg, and they all have got their own roadmap and it's kind of organic of how these different communities want to evolve, and I think that's great, but you have these end consumers like Blinkit who have different maybe use cases overlapping, and they're not forced to pick one. When you have an open architecture, they can really put together best-of-breed. And as these projects evolve, they can continue to monitor it and then make decisions and continue to remain agile based on the landscape and how it's evolving. >> So the agility is a key point. Flexibility and agility, and time to valuing with your data. >> Yeah. >> All right. Wen, I got to get in to why the Presto is important here. Where does that fit in? Why is Presto important? >> Yeah. For me, it all comes down to the use cases and the needs. And reporting and dashboarding is not going to go away anytime soon. It's a very common use case. Many of our customers like Blinkit come to us for that use case. The difference now is today, people want to do that particular use case on top of the modern data lake, on top of scalable, inexpensive, low cost storage. Right? In addition to that, there's a need for this low latency interactive ability to engage with the data. This is often arises when you need to do things in a ad hoc basis or you're in the developmental phase of building things up. So if that's what your need is. And latency's important and getting your arms around the problems, very important. You have a certain SLA, I need to deliver something. That puts some requirements in the technology. And Presto is a perfect for that ideal use case. It's ideal for that use case. It's distributed, it's scalable, it's in memory. And so it's able to really provide that. I think the other benefit for Presto and why we're bidding on Presto is it works well on the data lakes, but you have to think about how are these organizations maturing with this technology. So it's not necessarily an all or nothing. You have organizations that have maybe the data lake and it's augmented with other analytical data stores like Snowflake or Redshift. So Presto also... a core aspect is its ability to federate or connect and query across different data sources. So this can be a permanent thing. This could also be a transitionary thing. We have some customers that are moving and slowly shifting their data portfolio from maybe all data warehouse into 80% data lake. But it gives that optionality, it gives that ability to transition over a timeframe. But for all those reasons, the latency, the scalability, the federation, is why Presto for this particular use case. >> And you can connect with other databases. It can be purpose built database, could be whatever. Right? >> Sure. Yes, yes. Presto has a very pluggable architecture. >> Okay. Here's the question for the Blinkit team? Why did you choose Presto and what led you to Ahana? >> So I'll take this better, over this what Presto sits well in that reach is, is how it is designed. Like basically, Presto decouples your storage with the compute. Basically like, people can use any storage and Presto just works as a query engine for them. So basically, it has a constant connectors where you can connect with a real-time databases like Pinot or a Druid, along with your warehouses like Redshift, along with your data lake that's like based on Hudi or Iceberg. So it's like a very landscape that you can use with the Presto. And consumers like the analytics doesn't need to learn the SQL or different paradigms of the querying for different sources. They just need to learn a single source. And, they get a single place to consume from. They get a single consumer on their single destination to write on also. So, it's a homologous architecture, which allows you to put a central security like which Presto integrates. So it's also based on open architecture, that's Apache engine. And it has also certain innovative features that you can see based on caching, which reduces a lot of the cost. And since you have further decoupled your storage with the compute, you can further reduce your cost, because now the biggest part of our tradition warehouse is a storage. And the cost goes massively upwards with the amount of data that you've added. Like basically, each time that you add more data, you require more storage, and warehouses ask you to write the data in their own format. Over here since we have decoupled that, the storage cost have gone down. It's literally that your cost that you are writing, and you just pay for the compute, and you can scale in scale out based on the requirements. If you have high traffic, you scale out. If you have low traffic, you scale in. So all those. >> So huge cost savings. >> Yeah. >> Yeah. Cost effectiveness, for sure. >> Cost effectiveness and you get a very good price value out of it. Like for each query, you can estimate what's the cost for you based on that tracking and all those things. >> I mean, if you think about the other classic Iceberg and what's under the water you don't know, it's the hidden cost. You think about the tooling, right, and also, time it takes to do stuff. So if you have flexibility on choice, when we were riffing on this last time we chatted with you guys and you brought it up earlier around, you can have the open formats to have different use cases in different tools or different platforms to work on it. Redshift, you can use Redshift here, or use something over there. You don't have to get locking >> Absolutely. >> Satyam & Akshay: Yeah. >> Locking is a huge problem. How do you guys see that 'cause sounds like here there's not a lot of locking. You got the open formats, and you got choice. >> Yeah. So you get best of the both worlds. Like you get with Ahana or with the Presto, you can get the best of the both worlds. Since it's cloud native, you can easily deploy your clusters very easily within like five minutes. Your cluster is up, you can start working on it. You can deploy multiple clusters for multiple teams. You get also flexibility of adding new connectors since it's open and further it's also much more secure since it's based on cloud native. So basically, you can control your security endpoints very well. So all those things comes in together with this architecture. So you can definitely go more on the lakehouse architecture than warehousing when you want to deliver data value faster. And basically, you get the much more high value out of your data in a sorted template. >> So Satyam, it sounds like the old warehousing was like the application person, not a lot of usage, old, a lot of latency. Okay. Here and there. But now you got more speed to deploy clusters, scale up scale down. Application developers are as everyone. It's not one person. It's not one group. It's whenever you want. So, you got speed. You got more diversity in the data opportunities, and your coding. >> Yeah. I think data warehouses are a way to start for every organization who is getting into data. I don't think data warehousing is still a solution and will be a solution for a lot of teams which are still getting into data. But as soon as you start scaling, as you start seeing the cost going up, as you start seeing the number of use cases adding up, having an open format definitely helps. So, I would say that's where we are also heading into and that's how our journey as well started with Presto as well, why we even thought about Ahana, right. >> (John chuckles) >> So, like you mentioned, one of the things that happened was as we were moving to the lakehouse and the open table format, I think Ahana is one of the first ones in the market to have Hudi as a first class citizen completely supported with all the things which are not even present at the time of... even with Presto, right. So we see Ahana working behind the scenes, improving even some of the things already over the open-source ecosystem. And that's where we get the most value out of Ahana as well. >> This is the convergence of open-source magic and commercialization. Wen, because you think about Data as Code, reminds me, I hear, "Data warehouse, it's not going to go away." But you got cloud scale or scale. It reminds me of the old, "Oh yeah, I have a data center." Well, here comes the cloud. So, doesn't really kill the data center, although Amazon would say that the data center's going to be eliminated. No, you just use it for whatever you need it for. You use it for specific use cases, but everyone, all the action goes to the cloud for scale. The same things happen with data, and look at the open-source community. It's kind of coming together. Data as Code is coming together. >> Yeah, absolutely. >> Absolutely. >> I do want to again to connect on another dot in terms of cost and that. You know, we've been talking a little bit about price performance, but there's an implicit cost, and I think this was also very important to Blinkit, and also why we're offering a managed service. So one piece of it. And it really revolves around the people, right? So outside of the technology, the performance. One thing that Akshay brought up and it's another important piece that I should have highlighted a little bit more is, Presto exposes the ability to interact your data in a widely adopted way, which is basically ANSI SQL. So the ability for your practitioners to use this technology is huge. That's just regular Presto. In terms of a managed service, the guys at Blinkit are a great high performing team, but they have to be very efficient with their time and what they manage. And what we're trying to do is provide leverage for them. So take a lot of the heavy lifting away, but at the same time, figuring out the right things to expose so that they have that same flexibility. And that's been the balancing point that we've been trying to balance at Ahana, but that goes back to cost. How do I total cost of ownership? And that not doesn't include just the actual querying processing time, but the ability for the organization to go ahead and absorb the solution. And what does it cost in terms of the people involved? >> Yeah. Great conversation. I mean, this brings up the question of back in the data center, the cloud days, you had the concept of an SRE, which is now popular, site reliability engineer. One person does all the clusters and manages all the scale. Is the data engineer the new SRE for data? Are we seeing a similar trajectory? Just want to get your reaction. What do you guys think? >> Yes, so I would say, definitely. It depends on the teams and the sizes of that. We are high performing team so each automation takes bits on the pieces of the architecture, like where they want to invest in. And it comes out with the value of the engineer's time and basically like how much they can invest in, how much they need to configure the architecture, and how much time it'll take to time to market. So basically like, this is what I would also highlight as an engineer. I found Ahana like the... I would say as a Presto in a cloud native environment, or I think so there's the one in the market that seamlessly scales and then scales out. And further, with a team of us, I would say our team size like three to four engineers managing cluster day in day out, conferring, tuning and all those things takes a lot of time. And Ahana came in and takes it off our plate and the hands in a solution which works out of box. So that's where this comes in. Ahana it's also based on open-source community. >> So the time of the engineer's time is so valuable. >> Yeah. >> My take on it really in terms of the data engineering being the SRE. I think that can work, it depends on the actual person, and we definitely try to make the process as easy as possible. I think in Blinkit's case, you guys are... There are data platform owners, but they definitely are aware of the pipelines. >> John: Yeah. >> So they have very intimate knowledge of what data engineers do, but I think in their case, you guys, you're managing a ton of systems. So it's not just even Presto. They have a ton of systems and surfacing that interface so they can cater to all the data engineers across their data systems, I think is the big need for them. I know you guys you want to chime in. I mean, we've seen the architecture and things like that. I think you guys did an amazing job there. >> So, and to adding to Wen's point, right. Like I generally think what DevOps is to the tech team. I think, what is data engineer or the data teams are to the data organization, right? Like they play a very similar role that you have to act as a guardrail to ensure that everyone has access to the data so the democratizing and everything is there, but that has to also come with security, right? And when you do that, there are (indistinct) a lot of points where someone can interact with data. We have... And again, there's a mixed match of open-source tools that works well, as well. And there are some paid tools as well. So for us like for visualization, we use Redash for our ad hoc analysis. And we use Tableau as well whenever we want to give a very concise reporting. We have Jupyter notebooks in place and we have EMRs as well. So we always have a mixed batch of things where people can interact with data. And most of our time is spent in acting as that guardrail to ensure that everyone should have access to data, but it shouldn't be exploited, right. And I think that's where we spend most of our time in. >> Yeah. And I think the time is valuable, but that your point about the democratization aspect of it, there seems to be a bigger step function value that you're enabling and needs to be talked out. The 10x engineer, it's more like 50x, right? If you get it done right, the enablement downstream at the scale that we're seeing with this new trend is significant. It's not just, oh yeah, visualization and get some data quicker, there's actually real advantages on a multiple with that engineering. So, and we saw that with DevOps, right? Like, you do this right and then magic happens on the edges. So, yeah, it's interesting. You guys, congratulations. Great environment. Thanks for sharing the insight Blinkit. Wen, great to see you. Ahana again with Presto, congratulations. The open-source meets data engineering. Thanks so much. >> Thanks, John. >> Appreciate it. >> Okay. >> Thanks John. >> Thanks. >> Thanks for having us. >> This season two, episode two of our ongoing series. This one is Data as Code. This is theCUBE. I'm John furrier. Thanks for watching. (gentle music)

[Music] in the 2010s organizations became keenly aware that data would become the key ingredient in driving competitive advantage differentiation and growth but to this day putting data to work remains a difficult challenge for many if not most organizations now as the cloud matures it has become a game changer for data practitioners by making cheap storage and massive processing power readily accessible we've also seen better tooling in the form of data workflows streaming machine intelligence ai developer tools security observability automation new databases and the like these innovations they accelerate data proficiency but at the same time they had complexity for practitioners data lakes data hubs data warehouses data marts data fabrics data meshes data catalogs data oceans are forming they're evolving and exploding onto the scene so in an effort to bring perspective to the sea of optionality we've brought together the brightest minds in the data analyst community to discuss how data management is morphing and what practitioners should expect in 2022 and beyond hello everyone my name is dave vellante with the cube and i'd like to welcome you to a special cube presentation analyst predictions 2022 the future of data management we've gathered six of the best analysts in data and data management who are going to present and discuss their top predictions and trends for 2022 in the first half of this decade let me introduce our six power panelists sanjeev mohan is former gartner analyst and principal at sanjamo tony bear is principal at db insight carl olufsen is well-known research vice president with idc dave meninger is senior vice president and research director at ventana research brad shimon chief analyst at ai platforms analytics and data management at omnia and doug henschen vice president and principal analyst at constellation research gentlemen welcome to the program and thanks for coming on thecube today great to be here thank you all right here's the format we're going to use i as moderator are going to call on each analyst separately who then will deliver their prediction or mega trend and then in the interest of time management and pace two analysts will have the opportunity to comment if we have more time we'll elongate it but let's get started right away sanjeev mohan please kick it off you want to talk about governance go ahead sir thank you dave i i believe that data governance which we've been talking about for many years is now not only going to be mainstream it's going to be table stakes and all the things that you mentioned you know with data oceans data lakes lake houses data fabric meshes the common glue is metadata if we don't understand what data we have and we are governing it there is no way we can manage it so we saw informatica when public last year after a hiatus of six years i've i'm predicting that this year we see some more companies go public uh my bet is on colibra most likely and maybe alation we'll see go public this year we we i'm also predicting that the scope of data governance is going to expand beyond just data it's not just data and reports we are going to see more transformations like spark jaws python even airflow we're going to see more of streaming data so from kafka schema registry for example we will see ai models become part of this whole governance suite so the governance suite is going to be very comprehensive very detailed lineage impact analysis and then even expand into data quality we already seen that happen with some of the tools where they are buying these smaller companies and bringing in data quality monitoring and integrating it with metadata management data catalogs also data access governance so these so what we are going to see is that once the data governance platforms become the key entry point into these modern architectures i'm predicting that the usage the number of users of a data catalog is going to exceed that of a bi tool that will take time and we already seen that that trajectory right now if you look at bi tools i would say there are 100 users to a bi tool to one data catalog and i i see that evening out over a period of time and at some point data catalogs will really become you know the main way for us to access data data catalog will help us visualize data but if we want to do more in-depth analysis it'll be the jumping-off point into the bi tool the data science tool and and that is that is the journey i see for the data governance products excellent thank you some comments maybe maybe doug a lot a lot of things to weigh in on there maybe you could comment yeah sanjeev i think you're spot on a lot of the trends uh the one disagreement i think it's it's really still far from mainstream as you say we've been talking about this for years it's like god motherhood apple pie everyone agrees it's important but too few organizations are really practicing good governance because it's hard and because the incentives have been lacking i think one thing that deserves uh mention in this context is uh esg mandates and guidelines these are environmental social and governance regs and guidelines we've seen the environmental rags and guidelines imposed in industries particularly the carbon intensive industries we've seen the social mandates particularly diversity imposed on suppliers by companies that are leading on this topic we've seen governance guidelines now being imposed by banks and investors so these esgs are presenting new carrots and sticks and it's going to demand more solid data it's going to demand more detailed reporting and solid reporting tighter governance but we're still far from mainstream adoption we have a lot of uh you know best of breed niche players in the space i think the signs that it's going to be more mainstream are starting with things like azure purview google dataplex the big cloud platform uh players seem to be uh upping the ante and and addressing starting to address governance excellent thank you doug brad i wonder if you could chime in as well yeah i would love to be a believer in data catalogs um but uh to doug's point i think that it's going to take some more pressure for for that to happen i recall metadata being something every enterprise thought they were going to get under control when we were working on service oriented architecture back in the 90s and that didn't happen quite the way we we anticipated and and uh to sanjeev's point it's because it is really complex and really difficult to do my hope is that you know we won't sort of uh how do we put this fade out into this nebulous nebula of uh domain catalogs that are specific to individual use cases like purview for getting data quality right or like data governance and cyber security and instead we have some tooling that can actually be adaptive to gather metadata to create something i know is important to you sanjeev and that is this idea of observability if you can get enough metadata without moving your data around but understanding the entirety of a system that's running on this data you can do a lot to help with with the governance that doug is talking about so so i just want to add that you know data governance like many other initiatives did not succeed even ai went into an ai window but that's a different topic but a lot of these things did not succeed because to your point the incentives were not there i i remember when starbucks oxley had come into the scene if if a bank did not do service obviously they were very happy to a million dollar fine that was like you know pocket change for them instead of doing the right thing but i think the stakes are much higher now with gdpr uh the floodgates open now you know california you know has ccpa but even ccpa is being outdated with cpra which is much more gdpr like so we are very rapidly entering a space where every pretty much every major country in the world is coming up with its own uh compliance regulatory requirements data residence is becoming really important and and i i think we are going to reach a stage where uh it won't be optional anymore so whether we like it or not and i think the reason data catalogs were not successful in the past is because we did not have the right focus on adoption we were focused on features and these features were disconnected very hard for business to stop these are built by it people for it departments to to take a look at technical metadata not business metadata today the tables have turned cdo's are driving this uh initiative uh regulatory compliances are beating down hard so i think the time might be right yeah so guys we have to move on here and uh but there's some some real meat on the bone here sanjeev i like the fact that you late you called out calibra and alation so we can look back a year from now and say okay he made the call he stuck it and then the ratio of bi tools the data catalogs that's another sort of measurement that we can we can take even though some skepticism there that's something that we can watch and i wonder if someday if we'll have more metadata than data but i want to move to tony baer you want to talk about data mesh and speaking you know coming off of governance i mean wow you know the whole concept of data mesh is decentralized data and then governance becomes you know a nightmare there but take it away tony we'll put it this way um data mesh you know the the idea at least is proposed by thoughtworks um you know basically was unleashed a couple years ago and the press has been almost uniformly almost uncritical um a good reason for that is for all the problems that basically that sanjeev and doug and brad were just you know we're just speaking about which is that we have all this data out there and we don't know what to do about it um now that's not a new problem that was a problem we had enterprise data warehouses it was a problem when we had our hadoop data clusters it's even more of a problem now the data's out in the cloud where the data is not only your data like is not only s3 it's all over the place and it's also including streaming which i know we'll be talking about later so the data mesh was a response to that the idea of that we need to debate you know who are the folks that really know best about governance is the domain experts so it was basically data mesh was an architectural pattern and a process my prediction for this year is that data mesh is going to hit cold hard reality because if you if you do a google search um basically the the published work the articles and databases have been largely you know pretty uncritical um so far you know that you know basically learning is basically being a very revolutionary new idea i don't think it's that revolutionary because we've talked about ideas like this brad and i you and i met years ago when we were talking about so and decentralizing all of us was at the application level now we're talking about at the data level and now we have microservices so there's this thought of oh if we manage if we're apps in cloud native through microservices why don't we think of data in the same way um my sense this year is that you know this and this has been a very active search if you look at google search trends is that now companies are going to you know enterprises are going to look at this seriously and as they look at seriously it's going to attract its first real hard scrutiny it's going to attract its first backlash that's not necessarily a bad thing it means that it's being taken seriously um the reason why i think that that uh that it will you'll start to see basically the cold hard light of day shine on data mesh is that it's still a work in progress you know this idea is basically a couple years old and there's still some pretty major gaps um the biggest gap is in is in the area of federated governance now federated governance itself is not a new issue uh federated governance position we're trying to figure out like how can we basically strike the balance between getting let's say you know between basically consistent enterprise policy consistent enterprise governance but yet the groups that understand the data know how to basically you know that you know how do we basically sort of balance the two there's a huge there's a huge gap there in practice and knowledge um also to a lesser extent there's a technology gap which is basically in the self-service technologies that will help teams essentially govern data you know basically through the full life cycle from developed from selecting the data from you know building the other pipelines from determining your access control determining looking at quality looking at basically whether data is fresh or whether or not it's trending of course so my predictions is that it will really receive the first harsh scrutiny this year you are going to see some organization enterprises declare premature victory when they've uh when they build some federated query implementations you're going to see vendors start to data mesh wash their products anybody in the data management space they're going to say that whether it's basically a pipelining tool whether it's basically elt whether it's a catalog um or confederated query tool they're all going to be like you know basically promoting the fact of how they support this hopefully nobody is going to call themselves a data mesh tool because data mesh is not a technology we're going to see one other thing come out of this and this harks back to the metadata that sanji was talking about and the catalogs that he was talking about which is that there's going to be a new focus on every renewed focus on metadata and i think that's going to spur interest in data fabrics now data fabrics are pretty vaguely defined but if we just take the most elemental definition which is a common metadata back plane i think that if anybody is going to get serious about data mesh they need to look at a data fabric because we all at the end of the day need to speak you know need to read from the same sheet of music so thank you tony dave dave meninger i mean one of the things that people like about data mesh is it pretty crisply articulates some of the flaws in today's organizational approaches to data what are your thoughts on this well i think we have to start by defining data mesh right the the term is already getting corrupted right tony said it's going to see the cold hard uh light of day and there's a problem right now that there are a number of overlapping terms that are similar but not identical so we've got data virtualization data fabric excuse me for a second sorry about that data virtualization data fabric uh uh data federation right uh so i i think that it's not really clear what each vendor means by these terms i see data mesh and data fabric becoming quite popular i've i've interpreted data mesh as referring primarily to the governance aspects as originally you know intended and specified but that's not the way i see vendors using i see vendors using it much more to mean data fabric and data virtualization so i'm going to comment on the group of those things i think the group of those things is going to happen they're going to happen they're going to become more robust our research suggests that a quarter of organizations are already using virtualized access to their data lakes and another half so a total of three quarters will eventually be accessing their data lakes using some sort of virtualized access again whether you define it as mesh or fabric or virtualization isn't really the point here but this notion that there are different elements of data metadata and governance within an organization that all need to be managed collectively the interesting thing is when you look at the satisfaction rates of those organizations using virtualization versus those that are not it's almost double 68 of organizations i'm i'm sorry um 79 of organizations that were using virtualized access express satisfaction with their access to the data lake only 39 expressed satisfaction if they weren't using virtualized access so thank you uh dave uh sanjeev we just got about a couple minutes on this topic but i know you're speaking or maybe you've spoken already on a panel with jamal dagani who sort of invented the concept governance obviously is a big sticking point but what are your thoughts on this you are mute so my message to your mark and uh and to the community is uh as opposed to what dave said let's not define it we spent the whole year defining it there are four principles domain product data infrastructure and governance let's take it to the next level i get a lot of questions on what is the difference between data fabric and data mesh and i'm like i can compare the two because data mesh is a business concept data fabric is a data integration pattern how do you define how do you compare the two you have to bring data mesh level down so to tony's point i'm on a warp path in 2022 to take it down to what does a data product look like how do we handle shared data across domains and govern it and i think we are going to see more of that in 2022 is operationalization of data mesh i think we could have a whole hour on this topic couldn't we uh maybe we should do that uh but let's go to let's move to carl said carl your database guy you've been around that that block for a while now you want to talk about graph databases bring it on oh yeah okay thanks so i regard graph database as basically the next truly revolutionary database management technology i'm looking forward to for the graph database market which of course we haven't defined yet so obviously i have a little wiggle room in what i'm about to say but that this market will grow by about 600 percent over the next 10 years now 10 years is a long time but over the next five years we expect to see gradual growth as people start to learn how to use it problem isn't that it's used the problem is not that it's not useful is that people don't know how to use it so let me explain before i go any further what a graph database is because some of the folks on the call may not may not know what it is a graph database organizes data according to a mathematical structure called a graph a graph has elements called nodes and edges so a data element drops into a node the nodes are connected by edges the edges connect one node to another node combinations of edges create structures that you can analyze to determine how things are related in some cases the nodes and edges can have properties attached to them which add additional informative material that makes it richer that's called a property graph okay there are two principal use cases for graph databases there's there's semantic proper graphs which are used to break down human language text uh into the semantic structures then you can search it organize it and and and answer complicated questions a lot of ai is aimed at semantic graphs another kind is the property graph that i just mentioned which has a dazzling number of use cases i want to just point out is as i talk about this people are probably wondering well we have relational databases isn't that good enough okay so a relational database defines it uses um it supports what i call definitional relationships that means you define the relationships in a fixed structure the database drops into that structure there's a value foreign key value that relates one table to another and that value is fixed you don't change it if you change it the database becomes unstable it's not clear what you're looking at in a graph database the system is designed to handle change so that it can reflect the true state of the things that it's being used to track so um let me just give you some examples of use cases for this um they include uh entity resolution data lineage uh um social media analysis customer 360 fraud prevention there's cyber security there's strong supply chain is a big one actually there's explainable ai and this is going to become important too because a lot of people are adopting ai but they want a system after the fact to say how did the ai system come to that conclusion how did it make that recommendation right now we don't have really good ways of tracking that okay machine machine learning in general um social network i already mentioned that and then we've got oh gosh we've got data governance data compliance risk management we've got recommendation we've got personalization anti-money money laundering that's another big one identity and access management network and i.t operations is already becoming a key one where you actually have mapped out your operation your your you know whatever it is your data center and you you can track what's going on as things happen there root cause analysis fraud detection is a huge one a number of major credit card companies use graph databases for fraud detection risk analysis tracking and tracing churn analysis next best action what-if analysis impact analysis entity resolution and i would add one other thing or just a few other things to this list metadata management so sanjay here you go this is your engine okay because i was in metadata management for quite a while in my past life and one of the things i found was that none of the data management technologies that were available to us could efficiently handle metadata because of the kinds of structures that result from it but grass can okay grafts can do things like say this term in this context means this but in that context it means that okay things like that and in fact uh logistics management supply chain it also because it handles recursive relationships by recursive relationships i mean objects that own other objects that are of the same type you can do things like bill materials you know so like parts explosion you can do an hr analysis who reports to whom how many levels up the chain and that kind of thing you can do that with relational databases but yes it takes a lot of programming in fact you can do almost any of these things with relational databases but the problem is you have to program it it's not it's not supported in the database and whenever you have to program something that means you can't trace it you can't define it you can't publish it in terms of its functionality and it's really really hard to maintain over time so carl thank you i wonder if we could bring brad in i mean brad i'm sitting there wondering okay is this incremental to the market is it disruptive and replaceable what are your thoughts on this space it's already disrupted the market i mean like carl said go to any bank and ask them are you using graph databases to do to get fraud detection under control and they'll say absolutely that's the only way to solve this problem and it is frankly um and it's the only way to solve a lot of the problems that carl mentioned and that is i think it's it's achilles heel in some ways because you know it's like finding the best way to cross the seven bridges of konigsberg you know it's always going to kind of be tied to those use cases because it's really special and it's really unique and because it's special and it's unique uh it it still unfortunately kind of stands apart from the rest of the community that's building let's say ai outcomes as the great great example here the graph databases and ai as carl mentioned are like chocolate and peanut butter but technologically they don't know how to talk to one another they're completely different um and you know it's you can't just stand up sql and query them you've got to to learn um yeah what is that carlos specter or uh special uh uh yeah thank you uh to actually get to the data in there and if you're gonna scale that data that graph database especially a property graph if you're gonna do something really complex like try to understand uh you know all of the metadata in your organization you might just end up with you know a graph database winter like we had the ai winter simply because you run out of performance to make the thing happen so i i think it's already disrupted but we we need to like treat it like a first-class citizen in in the data analytics and ai community we need to bring it into the fold we need to equip it with the tools it needs to do that the magic it does and to do it not just for specialized use cases but for everything because i i'm with carl i i think it's absolutely revolutionary so i had also identified the principal achilles heel of the technology which is scaling now when these when these things get large and complex enough that they spill over what a single server can handle you start to have difficulties because the relationships span things that have to be resolved over a network and then you get network latency and that slows the system down so that's still a problem to be solved sanjeev any quick thoughts on this i mean i think metadata on the on the on the word cloud is going to be the the largest font uh but what are your thoughts here i want to like step away so people don't you know associate me with only meta data so i want to talk about something a little bit slightly different uh dbengines.com has done an amazing job i think almost everyone knows that they chronicle all the major databases that are in use today in january of 2022 there are 381 databases on its list of ranked list of databases the largest category is rdbms the second largest category is actually divided into two property graphs and rdf graphs these two together make up the second largest number of data databases so talking about accolades here this is a problem the problem is that there's so many graph databases to choose from they come in different shapes and forms uh to bright's point there's so many query languages in rdbms is sql end of the story here we've got sci-fi we've got gremlin we've got gql and then your proprietary languages so i think there's a lot of disparity in this space but excellent all excellent points sanji i must say and that is a problem the languages need to be sorted and standardized and it needs people need to have a road map as to what they can do with it because as you say you can do so many things and so many of those things are unrelated that you sort of say well what do we use this for i'm reminded of the saying i learned a bunch of years ago when somebody said that the digital computer is the only tool man has ever devised that has no particular purpose all right guys we gotta we gotta move on to dave uh meninger uh we've heard about streaming uh your prediction is in that realm so please take it away sure so i like to say that historical databases are to become a thing of the past but i don't mean that they're going to go away that's not my point i mean we need historical databases but streaming data is going to become the default way in which we operate with data so in the next say three to five years i would expect the data platforms and and we're using the term data platforms to represent the evolution of databases and data lakes that the data platforms will incorporate these streaming capabilities we're going to process data as it streams into an organization and then it's going to roll off into historical databases so historical databases don't go away but they become a thing of the past they store the data that occurred previously and as data is occurring we're going to be processing it we're going to be analyzing we're going to be acting on it i mean we we only ever ended up with historical databases because we were limited by the technology that was available to us data doesn't occur in batches but we processed it in batches because that was the best we could do and it wasn't bad and we've continued to improve and we've improved and we've improved but streaming data today is still the exception it's not the rule right there's there are projects within organizations that deal with streaming data but it's not the default way in which we deal with data yet and so that that's my prediction is that this is going to change we're going to have um streaming data be the default way in which we deal with data and and how you label it what you call it you know maybe these databases and data platforms just evolve to be able to handle it but we're going to deal with data in a different way and our research shows that already about half of the participants in our analytics and data benchmark research are using streaming data you know another third are planning to use streaming technologies so that gets us to about eight out of ten organizations need to use this technology that doesn't mean they have to use it throughout the whole organization but but it's pretty widespread in its use today and has continued to grow if you think about the consumerization of i.t we've all been conditioned to expect immediate access to information immediate responsiveness you know we want to know if an uh item is on the shelf at our local retail store and we can go in and pick it up right now you know that's the world we live in and that's spilling over into the enterprise i.t world where we have to provide those same types of capabilities um so that's my prediction historical database has become a thing of the past streaming data becomes the default way in which we we operate with data all right thank you david well so what what say you uh carl a guy who's followed historical databases for a long time well one thing actually every database is historical because as soon as you put data in it it's now history it's no longer it no longer reflects the present state of things but even if that history is only a millisecond old it's still history but um i would say i mean i know you're trying to be a little bit provocative in saying this dave because you know as well as i do that people still need to do their taxes they still need to do accounting they still need to run general ledger programs and things like that that all involves historical data that's not going to go away unless you want to go to jail so you're going to have to deal with that but as far as the leading edge functionality i'm totally with you on that and i'm just you know i'm just kind of wondering um if this chain if this requires a change in the way that we perceive applications in order to truly be manifested and rethinking the way m applications work um saying that uh an application should respond instantly as soon as the state of things changes what do you say about that i i think that's true i think we do have to think about things differently that's you know it's not the way we design systems in the past uh we're seeing more and more systems designed that way but again it's not the default and and agree 100 with you that we do need historical databases you know that that's clear and even some of those historical databases will be used in conjunction with the streaming data right so absolutely i mean you know let's take the data warehouse example where you're using the data warehouse as context and the streaming data as the present you're saying here's a sequence of things that's happening right now have we seen that sequence before and where what what does that pattern look like in past situations and can we learn from that so tony bear i wonder if you could comment i mean if you when you think about you know real-time inferencing at the edge for instance which is something that a lot of people talk about um a lot of what we're discussing here in this segment looks like it's got great potential what are your thoughts yeah well i mean i think you nailed it right you know you hit it right on the head there which is that i think a key what i'm seeing is that essentially and basically i'm going to split this one down the middle is i don't see that basically streaming is the default what i see is streaming and basically and transaction databases um and analytics data you know data warehouses data lakes whatever are converging and what allows us technically to converge is cloud native architecture where you can basically distribute things so you could have you can have a note here that's doing the real-time processing that's also doing it and this is what your leads in we're maybe doing some of that real-time predictive analytics to take a look at well look we're looking at this customer journey what's happening with you know you know with with what the customer is doing right now and this is correlated with what other customers are doing so what i so the thing is that in the cloud you can basically partition this and because of basically you know the speed of the infrastructure um that you can basically bring these together and or and so and kind of orchestrate them sort of loosely coupled manner the other part is that the use cases are demanding and this is part that goes back to what dave is saying is that you know when you look at customer 360 when you look at let's say smart you know smart utility grids when you look at any type of operational problem it has a real-time component and it has a historical component and having predictives and so like you know you know my sense here is that there that technically we can bring this together through the cloud and i think the use case is that is that we we can apply some some real-time sort of you know predictive analytics on these streams and feed this into the transactions so that when we make a decision in terms of what to do as a result of a transaction we have this real time you know input sanjeev did you have a comment yeah i was just going to say that to this point you know we have to think of streaming very different because in the historical databases we used to bring the data and store the data and then we used to run rules on top uh aggregations and all but in case of streaming the mindset changes because the rules normally the inference all of that is fixed but the data is constantly changing so it's a completely reverse way of thinking of uh and building applications on top of that so dave menninger there seemed to be some disagreement about the default or now what kind of time frame are you are you thinking about is this end of decade it becomes the default what would you pin i i think around you know between between five to ten years i think this becomes the reality um i think you know it'll be more and more common between now and then but it becomes the default and i also want sanjeev at some point maybe in one of our subsequent conversations we need to talk about governing streaming data because that's a whole other set of challenges we've also talked about it rather in a two dimensions historical and streaming and there's lots of low latency micro batch sub second that's not quite streaming but in many cases it's fast enough and we're seeing a lot of adoption of near real time not quite real time as uh good enough for most for many applications because nobody's really taking the hardware dimension of this information like how do we that'll just happen carl so near real time maybe before you lose the customer however you define that right okay um let's move on to brad brad you want to talk about automation ai uh the the the pipeline people feel like hey we can just automate everything what's your prediction yeah uh i'm i'm an ai fiction auto so apologies in advance for that but uh you know um i i think that um we've been seeing automation at play within ai for some time now and it's helped us do do a lot of things for especially for practitioners that are building ai outcomes in the enterprise uh it's it's helped them to fill skills gaps it's helped them to speed development and it's helped them to to actually make ai better uh because it you know in some ways provides some swim lanes and and for example with technologies like ottawa milk and can auto document and create that sort of transparency that that we talked about a little bit earlier um but i i think it's there's an interesting kind of conversion happening with this idea of automation um and and that is that uh we've had the automation that started happening for practitioners it's it's trying to move outside of the traditional bounds of things like i'm just trying to get my features i'm just trying to pick the right algorithm i'm just trying to build the right model uh and it's expanding across that full life cycle of building an ai outcome to start at the very beginning of data and to then continue on to the end which is this continuous delivery and continuous uh automation of of that outcome to make sure it's right and it hasn't drifted and stuff like that and because of that because it's become kind of powerful we're starting to to actually see this weird thing happen where the practitioners are starting to converge with the users and that is to say that okay if i'm in tableau right now i can stand up salesforce einstein discovery and it will automatically create a nice predictive algorithm for me um given the data that i that i pull in um but what's starting to happen and we're seeing this from the the the companies that create business software so salesforce oracle sap and others is that they're starting to actually use these same ideals and a lot of deep learning to to basically stand up these out of the box flip a switch and you've got an ai outcome at the ready for business users and um i i'm very much you know i think that that's that's the way that it's going to go and what it means is that ai is is slowly disappearing uh and i don't think that's a bad thing i think if anything what we're going to see in 2022 and maybe into 2023 is this sort of rush to to put this idea of disappearing ai into practice and have as many of these solutions in the enterprise as possible you can see like for example sap is going to roll out this quarter this thing called adaptive recommendation services which which basically is a cold start ai outcome that can work across a whole bunch of different vertical markets and use cases it's just a recommendation engine for whatever you need it to do in the line of business so basically you're you're an sap user you look up to turn on your software one day and you're a sales professional let's say and suddenly you have a recommendation for customer churn it's going that's great well i i don't know i i think that's terrifying in some ways i think it is the future that ai is going to disappear like that but i am absolutely terrified of it because um i i think that what it what it really does is it calls attention to a lot of the issues that we already see around ai um specific to this idea of what what we like to call it omdia responsible ai which is you know how do you build an ai outcome that is free of bias that is inclusive that is fair that is safe that is secure that it's audible etc etc etc etc that takes some a lot of work to do and so if you imagine a customer that that's just a sales force customer let's say and they're turning on einstein discovery within their sales software you need some guidance to make sure that when you flip that switch that the outcome you're going to get is correct and that's that's going to take some work and so i think we're going to see this let's roll this out and suddenly there's going to be a lot of a lot of problems a lot of pushback uh that we're going to see and some of that's going to come from gdpr and others that sam jeeve was mentioning earlier a lot of it's going to come from internal csr requirements within companies that are saying hey hey whoa hold up we can't do this all at once let's take the slow route let's make ai automated in a smart way and that's going to take time yeah so a couple predictions there that i heard i mean ai essentially you disappear it becomes invisible maybe if i can restate that and then if if i understand it correctly brad you're saying there's a backlash in the near term people can say oh slow down let's automate what we can those attributes that you talked about are non trivial to achieve is that why you're a bit of a skeptic yeah i think that we don't have any sort of standards that companies can look to and understand and we certainly within these companies especially those that haven't already stood up in internal data science team they don't have the knowledge to understand what that when they flip that switch for an automated ai outcome that it's it's gonna do what they think it's gonna do and so we need some sort of standard standard methodology and practice best practices that every company that's going to consume this invisible ai can make use of and one of the things that you know is sort of started that google kicked off a few years back that's picking up some momentum and the companies i just mentioned are starting to use it is this idea of model cards where at least you have some transparency about what these things are doing you know so like for the sap example we know for example that it's convolutional neural network with a long short-term memory model that it's using we know that it only works on roman english uh and therefore me as a consumer can say oh well i know that i need to do this internationally so i should not just turn this on today great thank you carl can you add anything any context here yeah we've talked about some of the things brad mentioned here at idc in the our future of intelligence group regarding in particular the moral and legal implications of having a fully automated you know ai uh driven system uh because we already know and we've seen that ai systems are biased by the data that they get right so if if they get data that pushes them in a certain direction i think there was a story last week about an hr system that was uh that was recommending promotions for white people over black people because in the past um you know white people were promoted and and more productive than black people but not it had no context as to why which is you know because they were being historically discriminated black people being historically discriminated against but the system doesn't know that so you know you have to be aware of that and i think that at the very least there should be controls when a decision has either a moral or a legal implication when when you want when you really need a human judgment it could lay out the options for you but a person actually needs to authorize that that action and i also think that we always will have to be vigilant regarding the kind of data we use to train our systems to make sure that it doesn't introduce unintended biases and to some extent they always will so we'll always be chasing after them that's that's absolutely carl yeah i think that what you have to bear in mind as a as a consumer of ai is that it is a reflection of us and we are a very flawed species uh and so if you look at all the really fantastic magical looking supermodels we see like gpt three and four that's coming out z they're xenophobic and hateful uh because the people the data that's built upon them and the algorithms and the people that build them are us so ai is a reflection of us we need to keep that in mind yeah we're the ai's by us because humans are biased all right great okay let's move on doug henson you know a lot of people that said that data lake that term's not not going to not going to live on but it appears to be have some legs here uh you want to talk about lake house bring it on yes i do my prediction is that lake house and this idea of a combined data warehouse and data lake platform is going to emerge as the dominant data management offering i say offering that doesn't mean it's going to be the dominant thing that organizations have out there but it's going to be the predominant vendor offering in 2022. now heading into 2021 we already had cloudera data bricks microsoft snowflake as proponents in 2021 sap oracle and several of these fabric virtualization mesh vendors join the bandwagon the promise is that you have one platform that manages your structured unstructured and semi-structured information and it addresses both the beyond analytics needs and the data science needs the real promise there is simplicity and lower cost but i think end users have to answer a few questions the first is does your organization really have a center of data gravity or is it is the data highly distributed multiple data warehouses multiple data lakes on-premises cloud if it if it's very distributed and you you know you have difficulty consolidating and that's not really a goal for you then maybe that single platform is unrealistic and not likely to add value to you um you know also the fabric and virtualization vendors the the mesh idea that's where if you have this highly distributed situation that might be a better path forward the second question if you are looking at one of these lake house offerings you are looking at consolidating simplifying bringing together to a single platform you have to make sure that it meets both the warehouse need and the data lake need so you have vendors like data bricks microsoft with azure synapse new really to the data warehouse space and they're having to prove that these data warehouse capabilities on their platforms can meet the scaling requirements can meet the user and query concurrency requirements meet those tight slas and then on the other hand you have the or the oracle sap snowflake the data warehouse uh folks coming into the data science world and they have to prove that they can manage the unstructured information and meet the needs of the data scientists i'm seeing a lot of the lake house offerings from the warehouse crowd managing that unstructured information in columns and rows and some of these vendors snowflake in particular is really relying on partners for the data science needs so you really got to look at a lake house offering and make sure that it meets both the warehouse and the data lake requirement well thank you doug well tony if those two worlds are going to come together as doug was saying the analytics and the data science world does it need to be some kind of semantic layer in between i don't know weigh in on this topic if you would oh didn't we talk about data fabrics before common metadata layer um actually i'm almost tempted to say let's declare victory and go home in that this is actually been going on for a while i actually agree with uh you know much what doug is saying there which is that i mean we i remembered as far back as i think it was like 2014 i was doing a a study you know it was still at ovum predecessor omnia um looking at all these specialized databases that were coming up and seeing that you know there's overlap with the edges but yet there was still going to be a reason at the time that you would have let's say a document database for json you'd have a relational database for tran you know for transactions and for data warehouse and you had you know and you had basically something at that time that that resembles to do for what we're considering a day of life fast fo and the thing is what i was saying at the time is that you're seeing basically blur you know sort of blending at the edges that i was saying like about five or six years ago um that's all and the the lake house is essentially you know the amount of the the current manifestation of that idea there is a dichotomy in terms of you know it's the old argument do we centralize this all you know you know in in in in in a single place or do we or do we virtualize and i think it's always going to be a yin and yang there's never going to be a single single silver silver bullet i do see um that they're also going to be questions and these are things that points that doug raised they're you know what your what do you need of of of your of you know for your performance there or for your you know pre-performance characteristics do you need for instance hiking currency you need the ability to do some very sophisticated joins or is your requirement more to be able to distribute and you know distribute our processing is you know as far as possible to get you know to essentially do a kind of brute force approach all these approaches are valid based on you know based on the used case um i just see that essentially that the lake house is the culmination of it's nothing it's just it's a relatively new term introduced by databricks a couple years ago this is the culmination of basically what's been a long time trend and what we see in the cloud is that as we start seeing data warehouses as a checkbox item say hey we can basically source data in cloud and cloud storage and s3 azure blob store you know whatever um as long as it's in certain formats like you know like you know parquet or csv or something like that you know i see that as becoming kind of you know a check box item so to that extent i think that the lake house depending on how you define it is already reality um and in some in some cases maybe new terminology but not a whole heck of a lot new under the sun yeah and dave menger i mean a lot of this thank you tony but a lot of this is going to come down to you know vendor marketing right some people try to co-opt the term we talked about data mesh washing what are your thoughts on this yeah so um i used the term data platform earlier and and part of the reason i use that term is that it's more vendor neutral uh we've we've tried to uh sort of stay out of the the vendor uh terminology patenting world right whether whether the term lake house is what sticks or not the concept is certainly going to stick and we have some data to back it up about a quarter of organizations that are using data lakes today already incorporate data warehouse functionality into it so they consider their data lake house and data warehouse one in the same about a quarter of organizations a little less but about a quarter of organizations feed the data lake from the data warehouse and about a quarter of organizations feed the data warehouse from the data lake so it's pretty obvious that three quarters of organizations need to bring this stuff together right the need is there the need is apparent the technology is going to continue to verge converge i i like to talk about you know you've got data lakes over here at one end and i'm not going to talk about why people thought data lakes were a bad idea because they thought you just throw stuff in a in a server and you ignore it right that's not what a data lake is so you've got data lake people over here and you've got database people over here data warehouse people over here database vendors are adding data lake capabilities and data lake vendors are adding data warehouse capabilities so it's obvious that they're going to meet in the middle i mean i think it's like tony says i think we should there declare victory and go home and so so i it's just a follow-up on that so are you saying these the specialized lake and the specialized warehouse do they go away i mean johnny tony data mesh practitioners would say or or advocates would say well they could all live as just a node on the on the mesh but based on what dave just said are we going to see those all morph together well number one as i was saying before there's always going to be this sort of you know kind of you know centrifugal force or this tug of war between do we centralize the data do we do it virtualize and the fact is i don't think that work there's ever going to be any single answer i think in terms of data mesh data mesh has nothing to do with how you physically implement the data you could have a data mesh on a basically uh on a data warehouse it's just that you know the difference being is that if we use the same you know physical data store but everybody's logically manual basically governing it differently you know um a data mission is basically it's not a technology it's a process it's a governance process um so essentially um you know you know i basically see that you know as as i was saying before that this is basically the culmination of a long time trend we're essentially seeing a lot of blurring but there are going to be cases where for instance if i need let's say like observe i need like high concurrency or something like that there are certain things that i'm not going to be able to get efficiently get out of a data lake um and you know we're basically i'm doing a system where i'm just doing really brute forcing very fast file scanning and that type of thing so i think there always will be some delineations but i would agree with dave and with doug that we are seeing basically a a confluence of requirements that we need to essentially have basically the element you know the ability of a data lake and a data laid out their warehouse we these need to come together so i think what we're likely to see is organizations look for a converged platform that can handle both sides for their center of data gravity the mesh and the fabric vendors the the fabric virtualization vendors they're all on board with the idea of this converged platform and they're saying hey we'll handle all the edge cases of the stuff that isn't in that center of data gradient that is off distributed in a cloud or at a remote location so you can have that single platform for the center of of your your data and then bring in virtualization mesh what have you for reaching out to the distributed data bingo as they basically said people are happy when they virtualize data i i think yes at this point but to this uh dave meningas point you know they have convert they are converging snowflake has introduced support for unstructured data so now we are literally splitting here now what uh databricks is saying is that aha but it's easy to go from data lake to data warehouse than it is from data warehouse to data lake so i think we're getting into semantics but we've already seen these two converge so is that so it takes something like aws who's got what 15 data stores are they're going to have 15 converged data stores that's going to be interesting to watch all right guys i'm going to go down the list and do like a one i'm going to one word each and you guys each of the analysts if you wouldn't just add a very brief sort of course correction for me so sanjeev i mean governance is going to be the maybe it's the dog that wags the tail now i mean it's coming to the fore all this ransomware stuff which really didn't talk much about security but but but what's the one word in your prediction that you would leave us with on governance it's uh it's going to be mainstream mainstream okay tony bear mesh washing is what i wrote down that's that's what we're going to see in uh in in 2022 a little reality check you you want to add to that reality check is i hope that no vendor you know jumps the shark and calls their offering a data mesh project yeah yeah let's hope that doesn't happen if they do we're going to call them out uh carl i mean graph databases thank you for sharing some some you know high growth metrics i know it's early days but magic is what i took away from that it's the magic database yeah i would actually i've said this to people too i i kind of look at it as a swiss army knife of data because you can pretty much do anything you want with it it doesn't mean you should i mean that's definitely the case that if you're you know managing things that are in a fixed schematic relationship probably a relational database is a better choice there are you know times when the document database is a better choice it can handle those things but maybe not it may not be the best choice for that use case but for a great many especially the new emerging use cases i listed it's the best choice thank you and dave meninger thank you by the way for bringing the data in i like how you supported all your comments with with some some data points but streaming data becomes the sort of default uh paradigm if you will what would you add yeah um i would say think fast right that's the world we live in you got to think fast fast love it uh and brad shimon uh i love it i mean on the one hand i was saying okay great i'm afraid i might get disrupted by one of these internet giants who are ai experts so i'm gonna be able to buy instead of build ai but then again you know i've got some real issues there's a potential backlash there so give us the there's your bumper sticker yeah i i would say um going with dave think fast and also think slow uh to to talk about the book that everyone talks about i would say really that this is all about trust trust in the idea of automation and of a transparent invisible ai across the enterprise but verify verify before you do anything and then doug henson i mean i i look i think the the trend is your friend here on this prediction with lake house is uh really becoming dominant i liked the way you set up that notion of you know the the the data warehouse folks coming at it from the analytics perspective but then you got the data science worlds coming together i still feel as though there's this piece in the middle that we're missing but your your final thoughts we'll give you the last well i think the idea of consolidation and simplification uh always prevails that's why the appeal of a single platform is going to be there um we've already seen that with uh you know hadoop platforms moving toward cloud moving toward object storage and object storage becoming really the common storage point for whether it's a lake or a warehouse uh and that second point uh i think esg mandates are uh are gonna come in alongside uh gdpr and things like that to uh up the ante for uh good governance yeah thank you for calling that out okay folks hey that's all the time that that we have here your your experience and depth of understanding on these key issues and in data and data management really on point and they were on display today i want to thank you for your your contributions really appreciate your time enjoyed it thank you now in addition to this video we're going to be making available transcripts of the discussion we're going to do clips of this as well we're going to put them out on social media i'll write this up and publish the discussion on wikibon.com and siliconangle.com no doubt several of the analysts on the panel will take the opportunity to publish written content social commentary or both i want to thank the power panelist and thanks for watching this special cube presentation this is dave vellante be well and we'll see you next time [Music] you

(bright music) >> In the 2010s, organizations became keenly aware that data would become the key ingredient to driving competitive advantage, differentiation, and growth. But to this day, putting data to work remains a difficult challenge for many, if not most organizations. Now, as the cloud matures, it has become a game changer for data practitioners by making cheap storage and massive processing power readily accessible. We've also seen better tooling in the form of data workflows, streaming, machine intelligence, AI, developer tools, security, observability, automation, new databases and the like. These innovations they accelerate data proficiency, but at the same time, they add complexity for practitioners. Data lakes, data hubs, data warehouses, data marts, data fabrics, data meshes, data catalogs, data oceans are forming, they're evolving and exploding onto the scene. So in an effort to bring perspective to the sea of optionality, we've brought together the brightest minds in the data analyst community to discuss how data management is morphing and what practitioners should expect in 2022 and beyond. Hello everyone, my name is Dave Velannte with theCUBE, and I'd like to welcome you to a special Cube presentation, analysts predictions 2022: the future of data management. We've gathered six of the best analysts in data and data management who are going to present and discuss their top predictions and trends for 2022 in the first half of this decade. Let me introduce our six power panelists. Sanjeev Mohan is former Gartner Analyst and Principal at SanjMo. Tony Baer, principal at dbInsight, Carl Olofson is well-known Research Vice President with IDC, Dave Menninger is Senior Vice President and Research Director at Ventana Research, Brad Shimmin, Chief Analyst, AI Platforms, Analytics and Data Management at Omdia and Doug Henschen, Vice President and Principal Analyst at Constellation Research. Gentlemen, welcome to the program and thanks for coming on theCUBE today. >> Great to be here. >> Thank you. >> All right, here's the format we're going to use. I as moderator, I'm going to call on each analyst separately who then will deliver their prediction or mega trend, and then in the interest of time management and pace, two analysts will have the opportunity to comment. If we have more time, we'll elongate it, but let's get started right away. Sanjeev Mohan, please kick it off. You want to talk about governance, go ahead sir. >> Thank you Dave. I believe that data governance which we've been talking about for many years is now not only going to be mainstream, it's going to be table stakes. And all the things that you mentioned, you know, the data, ocean data lake, lake houses, data fabric, meshes, the common glue is metadata. If we don't understand what data we have and we are governing it, there is no way we can manage it. So we saw Informatica went public last year after a hiatus of six. I'm predicting that this year we see some more companies go public. My bet is on Culebra, most likely and maybe Alation we'll see go public this year. I'm also predicting that the scope of data governance is going to expand beyond just data. It's not just data and reports. We are going to see more transformations like spark jawsxxxxx, Python even Air Flow. We're going to see more of a streaming data. So from Kafka Schema Registry, for example. We will see AI models become part of this whole governance suite. So the governance suite is going to be very comprehensive, very detailed lineage, impact analysis, and then even expand into data quality. We already seen that happen with some of the tools where they are buying these smaller companies and bringing in data quality monitoring and integrating it with metadata management, data catalogs, also data access governance. So what we are going to see is that once the data governance platforms become the key entry point into these modern architectures, I'm predicting that the usage, the number of users of a data catalog is going to exceed that of a BI tool. That will take time and we already seen that trajectory. Right now if you look at BI tools, I would say there a hundred users to BI tool to one data catalog. And I see that evening out over a period of time and at some point data catalogs will really become the main way for us to access data. Data catalog will help us visualize data, but if we want to do more in-depth analysis, it'll be the jumping off point into the BI tool, the data science tool and that is the journey I see for the data governance products. >> Excellent, thank you. Some comments. Maybe Doug, a lot of things to weigh in on there, maybe you can comment. >> Yeah, Sanjeev I think you're spot on, a lot of the trends the one disagreement, I think it's really still far from mainstream. As you say, we've been talking about this for years, it's like God, motherhood, apple pie, everyone agrees it's important, but too few organizations are really practicing good governance because it's hard and because the incentives have been lacking. I think one thing that deserves mention in this context is ESG mandates and guidelines, these are environmental, social and governance, regs and guidelines. We've seen the environmental regs and guidelines and posts in industries, particularly the carbon-intensive industries. We've seen the social mandates, particularly diversity imposed on suppliers by companies that are leading on this topic. We've seen governance guidelines now being imposed by banks on investors. So these ESGs are presenting new carrots and sticks, and it's going to demand more solid data. It's going to demand more detailed reporting and solid reporting, tighter governance. But we're still far from mainstream adoption. We have a lot of, you know, best of breed niche players in the space. I think the signs that it's going to be more mainstream are starting with things like Azure Purview, Google Dataplex, the big cloud platform players seem to be upping the ante and starting to address governance. >> Excellent, thank you Doug. Brad, I wonder if you could chime in as well. >> Yeah, I would love to be a believer in data catalogs. But to Doug's point, I think that it's going to take some more pressure for that to happen. I recall metadata being something every enterprise thought they were going to get under control when we were working on service oriented architecture back in the nineties and that didn't happen quite the way we anticipated. And so to Sanjeev's point it's because it is really complex and really difficult to do. My hope is that, you know, we won't sort of, how do I put this? Fade out into this nebula of domain catalogs that are specific to individual use cases like Purview for getting data quality right or like data governance and cybersecurity. And instead we have some tooling that can actually be adaptive to gather metadata to create something. And I know its important to you, Sanjeev and that is this idea of observability. If you can get enough metadata without moving your data around, but understanding the entirety of a system that's running on this data, you can do a lot. So to help with the governance that Doug is talking about. >> So I just want to add that, data governance, like any other initiatives did not succeed even AI went into an AI window, but that's a different topic. But a lot of these things did not succeed because to your point, the incentives were not there. I remember when Sarbanes Oxley had come into the scene, if a bank did not do Sarbanes Oxley, they were very happy to a million dollar fine. That was like, you know, pocket change for them instead of doing the right thing. But I think the stakes are much higher now. With GDPR, the flood gates opened. Now, you know, California, you know, has CCPA but even CCPA is being outdated with CPRA, which is much more GDPR like. So we are very rapidly entering a space where pretty much every major country in the world is coming up with its own compliance regulatory requirements, data residents is becoming really important. And I think we are going to reach a stage where it won't be optional anymore. So whether we like it or not, and I think the reason data catalogs were not successful in the past is because we did not have the right focus on adoption. We were focused on features and these features were disconnected, very hard for business to adopt. These are built by IT people for IT departments to take a look at technical metadata, not business metadata. Today the tables have turned. CDOs are driving this initiative, regulatory compliances are beating down hard, so I think the time might be right. >> Yeah so guys, we have to move on here. But there's some real meat on the bone here, Sanjeev. I like the fact that you called out Culebra and Alation, so we can look back a year from now and say, okay, he made the call, he stuck it. And then the ratio of BI tools to data catalogs that's another sort of measurement that we can take even though with some skepticism there, that's something that we can watch. And I wonder if someday, if we'll have more metadata than data. But I want to move to Tony Baer, you want to talk about data mesh and speaking, you know, coming off of governance. I mean, wow, you know the whole concept of data mesh is, decentralized data, and then governance becomes, you know, a nightmare there, but take it away, Tony. >> We'll put this way, data mesh, you know, the idea at least as proposed by ThoughtWorks. You know, basically it was at least a couple of years ago and the press has been almost uniformly almost uncritical. A good reason for that is for all the problems that basically Sanjeev and Doug and Brad we're just speaking about, which is that we have all this data out there and we don't know what to do about it. Now, that's not a new problem. That was a problem we had in enterprise data warehouses, it was a problem when we had over DoOP data clusters, it's even more of a problem now that data is out in the cloud where the data is not only your data lake, is not only us three, it's all over the place. And it's also including streaming, which I know we'll be talking about later. So the data mesh was a response to that, the idea of that we need to bait, you know, who are the folks that really know best about governance? It's the domain experts. So it was basically data mesh was an architectural pattern and a process. My prediction for this year is that data mesh is going to hit cold heart reality. Because if you do a Google search, basically the published work, the articles on data mesh have been largely, you know, pretty uncritical so far. Basically loading and is basically being a very revolutionary new idea. I don't think it's that revolutionary because we've talked about ideas like this. Brad now you and I met years ago when we were talking about so and decentralizing all of us, but it was at the application level. Now we're talking about it at the data level. And now we have microservices. So there's this thought of have we managed if we're deconstructing apps in cloud native to microservices, why don't we think of data in the same way? My sense this year is that, you know, this has been a very active search if you look at Google search trends, is that now companies, like enterprise are going to look at this seriously. And as they look at it seriously, it's going to attract its first real hard scrutiny, it's going to attract its first backlash. That's not necessarily a bad thing. It means that it's being taken seriously. The reason why I think that you'll start to see basically the cold hearted light of day shine on data mesh is that it's still a work in progress. You know, this idea is basically a couple of years old and there's still some pretty major gaps. The biggest gap is in the area of federated governance. Now federated governance itself is not a new issue. Federated governance decision, we started figuring out like, how can we basically strike the balance between getting let's say between basically consistent enterprise policy, consistent enterprise governance, but yet the groups that understand the data and know how to basically, you know, that, you know, how do we basically sort of balance the two? There's a huge gap there in practice and knowledge. Also to a lesser extent, there's a technology gap which is basically in the self-service technologies that will help teams essentially govern data. You know, basically through the full life cycle, from develop, from selecting the data from, you know, building the pipelines from, you know, determining your access control, looking at quality, looking at basically whether the data is fresh or whether it's trending off course. So my prediction is that it will receive the first harsh scrutiny this year. You are going to see some organization and enterprises declare premature victory when they build some federated query implementations. You going to see vendors start with data mesh wash their products anybody in the data management space that they are going to say that where this basically a pipelining tool, whether it's basically ELT, whether it's a catalog or federated query tool, they will all going to get like, you know, basically promoting the fact of how they support this. Hopefully nobody's going to call themselves a data mesh tool because data mesh is not a technology. We're going to see one other thing come out of this. And this harks back to the metadata that Sanjeev was talking about and of the catalog just as he was talking about. Which is that there's going to be a new focus, every renewed focus on metadata. And I think that's going to spur interest in data fabrics. Now data fabrics are pretty vaguely defined, but if we just take the most elemental definition, which is a common metadata back plane, I think that if anybody is going to get serious about data mesh, they need to look at the data fabric because we all at the end of the day, need to speak, you know, need to read from the same sheet of music. >> So thank you Tony. Dave Menninger, I mean, one of the things that people like about data mesh is it pretty crisply articulate some of the flaws in today's organizational approaches to data. What are your thoughts on this? >> Well, I think we have to start by defining data mesh, right? The term is already getting corrupted, right? Tony said it's going to see the cold hard light of day. And there's a problem right now that there are a number of overlapping terms that are similar but not identical. So we've got data virtualization, data fabric, excuse me for a second. (clears throat) Sorry about that. Data virtualization, data fabric, data federation, right? So I think that it's not really clear what each vendor means by these terms. I see data mesh and data fabric becoming quite popular. I've interpreted data mesh as referring primarily to the governance aspects as originally intended and specified. But that's not the way I see vendors using it. I see vendors using it much more to mean data fabric and data virtualization. So I'm going to comment on the group of those things. I think the group of those things is going to happen. They're going to happen, they're going to become more robust. Our research suggests that a quarter of organizations are already using virtualized access to their data lakes and another half, so a total of three quarters will eventually be accessing their data lakes using some sort of virtualized access. Again, whether you define it as mesh or fabric or virtualization isn't really the point here. But this notion that there are different elements of data, metadata and governance within an organization that all need to be managed collectively. The interesting thing is when you look at the satisfaction rates of those organizations using virtualization versus those that are not, it's almost double, 68% of organizations, I'm sorry, 79% of organizations that were using virtualized access express satisfaction with their access to the data lake. Only 39% express satisfaction if they weren't using virtualized access. >> Oh thank you Dave. Sanjeev we just got about a couple of minutes on this topic, but I know you're speaking or maybe you've always spoken already on a panel with (indistinct) who sort of invented the concept. Governance obviously is a big sticking point, but what are your thoughts on this? You're on mute. (panelist chuckling) >> So my message to (indistinct) and to the community is as opposed to what they said, let's not define it. We spent a whole year defining it, there are four principles, domain, product, data infrastructure, and governance. Let's take it to the next level. I get a lot of questions on what is the difference between data fabric and data mesh? And I'm like I can't compare the two because data mesh is a business concept, data fabric is a data integration pattern. How do you compare the two? You have to bring data mesh a level down. So to Tony's point, I'm on a warpath in 2022 to take it down to what does a data product look like? How do we handle shared data across domains and governance? And I think we are going to see more of that in 2022, or is "operationalization" of data mesh. >> I think we could have a whole hour on this topic, couldn't we? Maybe we should do that. But let's corner. Let's move to Carl. So Carl, you're a database guy, you've been around that block for a while now, you want to talk about graph databases, bring it on. >> Oh yeah. Okay thanks. So I regard graph database as basically the next truly revolutionary database management technology. I'm looking forward for the graph database market, which of course we haven't defined yet. So obviously I have a little wiggle room in what I'm about to say. But this market will grow by about 600% over the next 10 years. Now, 10 years is a long time. But over the next five years, we expect to see gradual growth as people start to learn how to use it. The problem is not that it's not useful, its that people don't know how to use it. So let me explain before I go any further what a graph database is because some of the folks on the call may not know what it is. A graph database organizes data according to a mathematical structure called a graph. The graph has elements called nodes and edges. So a data element drops into a node, the nodes are connected by edges, the edges connect one node to another node. Combinations of edges create structures that you can analyze to determine how things are related. In some cases, the nodes and edges can have properties attached to them which add additional informative material that makes it richer, that's called a property graph. There are two principle use cases for graph databases. There's semantic property graphs, which are use to break down human language texts into the semantic structures. Then you can search it, organize it and answer complicated questions. A lot of AI is aimed at semantic graphs. Another kind is the property graph that I just mentioned, which has a dazzling number of use cases. I want to just point out as I talk about this, people are probably wondering, well, we have relation databases, isn't that good enough? So a relational database defines... It supports what I call definitional relationships. That means you define the relationships in a fixed structure. The database drops into that structure, there's a value, foreign key value, that relates one table to another and that value is fixed. You don't change it. If you change it, the database becomes unstable, it's not clear what you're looking at. In a graph database, the system is designed to handle change so that it can reflect the true state of the things that it's being used to track. So let me just give you some examples of use cases for this. They include entity resolution, data lineage, social media analysis, Customer 360, fraud prevention. There's cybersecurity, there's strong supply chain is a big one actually. There is explainable AI and this is going to become important too because a lot of people are adopting AI. But they want a system after the fact to say, how do the AI system come to that conclusion? How did it make that recommendation? Right now we don't have really good ways of tracking that. Machine learning in general, social network, I already mentioned that. And then we've got, oh gosh, we've got data governance, data compliance, risk management. We've got recommendation, we've got personalization, anti money laundering, that's another big one, identity and access management, network and IT operations is already becoming a key one where you actually have mapped out your operation, you know, whatever it is, your data center and you can track what's going on as things happen there, root cause analysis, fraud detection is a huge one. A number of major credit card companies use graph databases for fraud detection, risk analysis, tracking and tracing turn analysis, next best action, what if analysis, impact analysis, entity resolution and I would add one other thing or just a few other things to this list, metadata management. So Sanjeev, here you go, this is your engine. Because I was in metadata management for quite a while in my past life. And one of the things I found was that none of the data management technologies that were available to us could efficiently handle metadata because of the kinds of structures that result from it, but graphs can, okay? Graphs can do things like say, this term in this context means this, but in that context, it means that, okay? Things like that. And in fact, logistics management, supply chain. And also because it handles recursive relationships, by recursive relationships I mean objects that own other objects that are of the same type. You can do things like build materials, you know, so like parts explosion. Or you can do an HR analysis, who reports to whom, how many levels up the chain and that kind of thing. You can do that with relational databases, but yet it takes a lot of programming. In fact, you can do almost any of these things with relational databases, but the problem is, you have to program it. It's not supported in the database. And whenever you have to program something, that means you can't trace it, you can't define it. You can't publish it in terms of its functionality and it's really, really hard to maintain over time. >> Carl, thank you. I wonder if we could bring Brad in, I mean. Brad, I'm sitting here wondering, okay, is this incremental to the market? Is it disruptive and replacement? What are your thoughts on this phase? >> It's already disrupted the market. I mean, like Carl said, go to any bank and ask them are you using graph databases to get fraud detection under control? And they'll say, absolutely, that's the only way to solve this problem. And it is frankly. And it's the only way to solve a lot of the problems that Carl mentioned. And that is, I think it's Achilles heel in some ways. Because, you know, it's like finding the best way to cross the seven bridges of Koenigsberg. You know, it's always going to kind of be tied to those use cases because it's really special and it's really unique and because it's special and it's unique, it's still unfortunately kind of stands apart from the rest of the community that's building, let's say AI outcomes, as a great example here. Graph databases and AI, as Carl mentioned, are like chocolate and peanut butter. But technologically, you think don't know how to talk to one another, they're completely different. And you know, you can't just stand up SQL and query them. You've got to learn, know what is the Carl? Specter special. Yeah, thank you to, to actually get to the data in there. And if you're going to scale that data, that graph database, especially a property graph, if you're going to do something really complex, like try to understand you know, all of the metadata in your organization, you might just end up with, you know, a graph database winter like we had the AI winter simply because you run out of performance to make the thing happen. So, I think it's already disrupted, but we need to like treat it like a first-class citizen in the data analytics and AI community. We need to bring it into the fold. We need to equip it with the tools it needs to do the magic it does and to do it not just for specialized use cases, but for everything. 'Cause I'm with Carl. I think it's absolutely revolutionary. >> Brad identified the principal, Achilles' heel of the technology which is scaling. When these things get large and complex enough that they spill over what a single server can handle, you start to have difficulties because the relationships span things that have to be resolved over a network and then you get network latency and that slows the system down. So that's still a problem to be solved. >> Sanjeev, any quick thoughts on this? I mean, I think metadata on the word cloud is going to be the largest font, but what are your thoughts here? >> I want to (indistinct) So people don't associate me with only metadata, so I want to talk about something slightly different. dbengines.com has done an amazing job. I think almost everyone knows that they chronicle all the major databases that are in use today. In January of 2022, there are 381 databases on a ranked list of databases. The largest category is RDBMS. The second largest category is actually divided into two property graphs and IDF graphs. These two together make up the second largest number databases. So talking about Achilles heel, this is a problem. The problem is that there's so many graph databases to choose from. They come in different shapes and forms. To Brad's point, there's so many query languages in RDBMS, in SQL. I know the story, but here We've got cipher, we've got gremlin, we've got GQL and then we're proprietary languages. So I think there's a lot of disparity in this space. >> Well, excellent. All excellent points, Sanjeev, if I must say. And that is a problem that the languages need to be sorted and standardized. People need to have a roadmap as to what they can do with it. Because as you say, you can do so many things. And so many of those things are unrelated that you sort of say, well, what do we use this for? And I'm reminded of the saying I learned a bunch of years ago. And somebody said that the digital computer is the only tool man has ever device that has no particular purpose. (panelists chuckle) >> All right guys, we got to move on to Dave Menninger. We've heard about streaming. Your prediction is in that realm, so please take it away. >> Sure. So I like to say that historical databases are going to become a thing of the past. By that I don't mean that they're going to go away, that's not my point. I mean, we need historical databases, but streaming data is going to become the default way in which we operate with data. So in the next say three to five years, I would expect that data platforms and we're using the term data platforms to represent the evolution of databases and data lakes, that the data platforms will incorporate these streaming capabilities. We're going to process data as it streams into an organization and then it's going to roll off into historical database. So historical databases don't go away, but they become a thing of the past. They store the data that occurred previously. And as data is occurring, we're going to be processing it, we're going to be analyzing it, we're going to be acting on it. I mean we only ever ended up with historical databases because we were limited by the technology that was available to us. Data doesn't occur in patches. But we processed it in patches because that was the best we could do. And it wasn't bad and we've continued to improve and we've improved and we've improved. But streaming data today is still the exception. It's not the rule, right? There are projects within organizations that deal with streaming data. But it's not the default way in which we deal with data yet. And so that's my prediction is that this is going to change, we're going to have streaming data be the default way in which we deal with data and how you label it and what you call it. You know, maybe these databases and data platforms just evolved to be able to handle it. But we're going to deal with data in a different way. And our research shows that already, about half of the participants in our analytics and data benchmark research, are using streaming data. You know, another third are planning to use streaming technologies. So that gets us to about eight out of 10 organizations need to use this technology. And that doesn't mean they have to use it throughout the whole organization, but it's pretty widespread in its use today and has continued to grow. If you think about the consumerization of IT, we've all been conditioned to expect immediate access to information, immediate responsiveness. You know, we want to know if an item is on the shelf at our local retail store and we can go in and pick it up right now. You know, that's the world we live in and that's spilling over into the enterprise IT world We have to provide those same types of capabilities. So that's my prediction, historical databases become a thing of the past, streaming data becomes the default way in which we operate with data. >> All right thank you David. Well, so what say you, Carl, the guy who has followed historical databases for a long time? >> Well, one thing actually, every database is historical because as soon as you put data in it, it's now history. They'll no longer reflect the present state of things. But even if that history is only a millisecond old, it's still history. But I would say, I mean, I know you're trying to be a little bit provocative in saying this Dave 'cause you know, as well as I do that people still need to do their taxes, they still need to do accounting, they still need to run general ledger programs and things like that. That all involves historical data. That's not going to go away unless you want to go to jail. So you're going to have to deal with that. But as far as the leading edge functionality, I'm totally with you on that. And I'm just, you know, I'm just kind of wondering if this requires a change in the way that we perceive applications in order to truly be manifested and rethinking the way applications work. Saying that an application should respond instantly, as soon as the state of things changes. What do you say about that? >> I think that's true. I think we do have to think about things differently. It's not the way we designed systems in the past. We're seeing more and more systems designed that way. But again, it's not the default. And I agree 100% with you that we do need historical databases you know, that's clear. And even some of those historical databases will be used in conjunction with the streaming data, right? >> Absolutely. I mean, you know, let's take the data warehouse example where you're using the data warehouse as its context and the streaming data as the present and you're saying, here's the sequence of things that's happening right now. Have we seen that sequence before? And where? What does that pattern look like in past situations? And can we learn from that? >> So Tony Baer, I wonder if you could comment? I mean, when you think about, you know, real time inferencing at the edge, for instance, which is something that a lot of people talk about, a lot of what we're discussing here in this segment, it looks like it's got a great potential. What are your thoughts? >> Yeah, I mean, I think you nailed it right. You know, you hit it right on the head there. Which is that, what I'm seeing is that essentially. Then based on I'm going to split this one down the middle is that I don't see that basically streaming is the default. What I see is streaming and basically and transaction databases and analytics data, you know, data warehouses, data lakes whatever are converging. And what allows us technically to converge is cloud native architecture, where you can basically distribute things. So you can have a node here that's doing the real-time processing, that's also doing... And this is where it leads in or maybe doing some of that real time predictive analytics to take a look at, well look, we're looking at this customer journey what's happening with what the customer is doing right now and this is correlated with what other customers are doing. So the thing is that in the cloud, you can basically partition this and because of basically the speed of the infrastructure then you can basically bring these together and kind of orchestrate them sort of a loosely coupled manner. The other parts that the use cases are demanding, and this is part of it goes back to what Dave is saying. Is that, you know, when you look at Customer 360, when you look at let's say Smart Utility products, when you look at any type of operational problem, it has a real time component and it has an historical component. And having predictive and so like, you know, my sense here is that technically we can bring this together through the cloud. And I think the use case is that we can apply some real time sort of predictive analytics on these streams and feed this into the transactions so that when we make a decision in terms of what to do as a result of a transaction, we have this real-time input. >> Sanjeev, did you have a comment? >> Yeah, I was just going to say that to Dave's point, you know, we have to think of streaming very different because in the historical databases, we used to bring the data and store the data and then we used to run rules on top, aggregations and all. But in case of streaming, the mindset changes because the rules are normally the inference, all of that is fixed, but the data is constantly changing. So it's a completely reversed way of thinking and building applications on top of that. >> So Dave Menninger, there seem to be some disagreement about the default. What kind of timeframe are you thinking about? Is this end of decade it becomes the default? What would you pin? >> I think around, you know, between five to 10 years, I think this becomes the reality. >> I think its... >> It'll be more and more common between now and then, but it becomes the default. And I also want Sanjeev at some point, maybe in one of our subsequent conversations, we need to talk about governing streaming data. 'Cause that's a whole nother set of challenges. >> We've also talked about it rather in two dimensions, historical and streaming, and there's lots of low latency, micro batch, sub-second, that's not quite streaming, but in many cases its fast enough and we're seeing a lot of adoption of near real time, not quite real-time as good enough for many applications. (indistinct cross talk from panelists) >> Because nobody's really taking the hardware dimension (mumbles). >> That'll just happened, Carl. (panelists laughing) >> So near real time. But maybe before you lose the customer, however we define that, right? Okay, let's move on to Brad. Brad, you want to talk about automation, AI, the pipeline people feel like, hey, we can just automate everything. What's your prediction? >> Yeah I'm an AI aficionados so apologies in advance for that. But, you know, I think that we've been seeing automation play within AI for some time now. And it's helped us do a lot of things especially for practitioners that are building AI outcomes in the enterprise. It's helped them to fill skills gaps, it's helped them to speed development and it's helped them to actually make AI better. 'Cause it, you know, in some ways provide some swim lanes and for example, with technologies like AutoML can auto document and create that sort of transparency that we talked about a little bit earlier. But I think there's an interesting kind of conversion happening with this idea of automation. And that is that we've had the automation that started happening for practitioners, it's trying to move out side of the traditional bounds of things like I'm just trying to get my features, I'm just trying to pick the right algorithm, I'm just trying to build the right model and it's expanding across that full life cycle, building an AI outcome, to start at the very beginning of data and to then continue on to the end, which is this continuous delivery and continuous automation of that outcome to make sure it's right and it hasn't drifted and stuff like that. And because of that, because it's become kind of powerful, we're starting to actually see this weird thing happen where the practitioners are starting to converge with the users. And that is to say that, okay, if I'm in Tableau right now, I can stand up Salesforce Einstein Discovery, and it will automatically create a nice predictive algorithm for me given the data that I pull in. But what's starting to happen and we're seeing this from the companies that create business software, so Salesforce, Oracle, SAP, and others is that they're starting to actually use these same ideals and a lot of deep learning (chuckles) to basically stand up these out of the box flip-a-switch, and you've got an AI outcome at the ready for business users. And I am very much, you know, I think that's the way that it's going to go and what it means is that AI is slowly disappearing. And I don't think that's a bad thing. I think if anything, what we're going to see in 2022 and maybe into 2023 is this sort of rush to put this idea of disappearing AI into practice and have as many of these solutions in the enterprise as possible. You can see, like for example, SAP is going to roll out this quarter, this thing called adaptive recommendation services, which basically is a cold start AI outcome that can work across a whole bunch of different vertical markets and use cases. It's just a recommendation engine for whatever you needed to do in the line of business. So basically, you're an SAP user, you look up to turn on your software one day, you're a sales professional let's say, and suddenly you have a recommendation for customer churn. Boom! It's going, that's great. Well, I don't know, I think that's terrifying. In some ways I think it is the future that AI is going to disappear like that, but I'm absolutely terrified of it because I think that what it really does is it calls attention to a lot of the issues that we already see around AI, specific to this idea of what we like to call at Omdia, responsible AI. Which is, you know, how do you build an AI outcome that is free of bias, that is inclusive, that is fair, that is safe, that is secure, that its audible, et cetera, et cetera, et cetera, et cetera. I'd take a lot of work to do. And so if you imagine a customer that's just a Salesforce customer let's say, and they're turning on Einstein Discovery within their sales software, you need some guidance to make sure that when you flip that switch, that the outcome you're going to get is correct. And that's going to take some work. And so, I think we're going to see this move, let's roll this out and suddenly there's going to be a lot of problems, a lot of pushback that we're going to see. And some of that's going to come from GDPR and others that Sanjeev was mentioning earlier. A lot of it is going to come from internal CSR requirements within companies that are saying, "Hey, hey, whoa, hold up, we can't do this all at once. "Let's take the slow route, "let's make AI automated in a smart way." And that's going to take time. >> Yeah, so a couple of predictions there that I heard. AI simply disappear, it becomes invisible. Maybe if I can restate that. And then if I understand it correctly, Brad you're saying there's a backlash in the near term. You'd be able to say, oh, slow down. Let's automate what we can. Those attributes that you talked about are non trivial to achieve, is that why you're a bit of a skeptic? >> Yeah. I think that we don't have any sort of standards that companies can look to and understand. And we certainly, within these companies, especially those that haven't already stood up an internal data science team, they don't have the knowledge to understand when they flip that switch for an automated AI outcome that it's going to do what they think it's going to do. And so we need some sort of standard methodology and practice, best practices that every company that's going to consume this invisible AI can make use of them. And one of the things that you know, is sort of started that Google kicked off a few years back that's picking up some momentum and the companies I just mentioned are starting to use it is this idea of model cards where at least you have some transparency about what these things are doing. You know, so like for the SAP example, we know, for example, if it's convolutional neural network with a long, short term memory model that it's using, we know that it only works on Roman English and therefore me as a consumer can say, "Oh, well I know that I need to do this internationally. "So I should not just turn this on today." >> Thank you. Carl could you add anything, any context here? >> Yeah, we've talked about some of the things Brad mentioned here at IDC and our future of intelligence group regarding in particular, the moral and legal implications of having a fully automated, you know, AI driven system. Because we already know, and we've seen that AI systems are biased by the data that they get, right? So if they get data that pushes them in a certain direction, I think there was a story last week about an HR system that was recommending promotions for White people over Black people, because in the past, you know, White people were promoted and more productive than Black people, but it had no context as to why which is, you know, because they were being historically discriminated, Black people were being historically discriminated against, but the system doesn't know that. So, you know, you have to be aware of that. And I think that at the very least, there should be controls when a decision has either a moral or legal implication. When you really need a human judgment, it could lay out the options for you. But a person actually needs to authorize that action. And I also think that we always will have to be vigilant regarding the kind of data we use to train our systems to make sure that it doesn't introduce unintended biases. In some extent, they always will. So we'll always be chasing after them. But that's (indistinct). >> Absolutely Carl, yeah. I think that what you have to bear in mind as a consumer of AI is that it is a reflection of us and we are a very flawed species. And so if you look at all of the really fantastic, magical looking supermodels we see like GPT-3 and four, that's coming out, they're xenophobic and hateful because the people that the data that's built upon them and the algorithms and the people that build them are us. So AI is a reflection of us. We need to keep that in mind. >> Yeah, where the AI is biased 'cause humans are biased. All right, great. All right let's move on. Doug you mentioned mentioned, you know, lot of people that said that data lake, that term is not going to live on but here's to be, have some lakes here. You want to talk about lake house, bring it on. >> Yes, I do. My prediction is that lake house and this idea of a combined data warehouse and data lake platform is going to emerge as the dominant data management offering. I say offering that doesn't mean it's going to be the dominant thing that organizations have out there, but it's going to be the pro dominant vendor offering in 2022. Now heading into 2021, we already had Cloudera, Databricks, Microsoft, Snowflake as proponents, in 2021, SAP, Oracle, and several of all of these fabric virtualization/mesh vendors joined the bandwagon. The promise is that you have one platform that manages your structured, unstructured and semi-structured information. And it addresses both the BI analytics needs and the data science needs. The real promise there is simplicity and lower cost. But I think end users have to answer a few questions. The first is, does your organization really have a center of data gravity or is the data highly distributed? Multiple data warehouses, multiple data lakes, on premises, cloud. If it's very distributed and you'd have difficulty consolidating and that's not really a goal for you, then maybe that single platform is unrealistic and not likely to add value to you. You know, also the fabric and virtualization vendors, the mesh idea, that's where if you have this highly distributed situation, that might be a better path forward. The second question, if you are looking at one of these lake house offerings, you are looking at consolidating, simplifying, bringing together to a single platform. You have to make sure that it meets both the warehouse need and the data lake need. So you have vendors like Databricks, Microsoft with Azure Synapse. New really to the data warehouse space and they're having to prove that these data warehouse capabilities on their platforms can meet the scaling requirements, can meet the user and query concurrency requirements. Meet those tight SLS. And then on the other hand, you have the Oracle, SAP, Snowflake, the data warehouse folks coming into the data science world, and they have to prove that they can manage the unstructured information and meet the needs of the data scientists. I'm seeing a lot of the lake house offerings from the warehouse crowd, managing that unstructured information in columns and rows. And some of these vendors, Snowflake a particular is really relying on partners for the data science needs. So you really got to look at a lake house offering and make sure that it meets both the warehouse and the data lake requirement. >> Thank you Doug. Well Tony, if those two worlds are going to come together, as Doug was saying, the analytics and the data science world, does it need to be some kind of semantic layer in between? I don't know. Where are you in on this topic? >> (chuckles) Oh, didn't we talk about data fabrics before? Common metadata layer (chuckles). Actually, I'm almost tempted to say let's declare victory and go home. And that this has actually been going on for a while. I actually agree with, you know, much of what Doug is saying there. Which is that, I mean I remember as far back as I think it was like 2014, I was doing a study. I was still at Ovum, (indistinct) Omdia, looking at all these specialized databases that were coming up and seeing that, you know, there's overlap at the edges. But yet, there was still going to be a reason at the time that you would have, let's say a document database for JSON, you'd have a relational database for transactions and for data warehouse and you had basically something at that time that resembles a dupe for what we consider your data life. Fast forward and the thing is what I was seeing at the time is that you were saying they sort of blending at the edges. That was saying like about five to six years ago. And the lake house is essentially on the current manifestation of that idea. There is a dichotomy in terms of, you know, it's the old argument, do we centralize this all you know in a single place or do we virtualize? And I think it's always going to be a union yeah and there's never going to be a single silver bullet. I do see that there are also going to be questions and these are points that Doug raised. That you know, what do you need for your performance there, or for your free performance characteristics? Do you need for instance high concurrency? You need the ability to do some very sophisticated joins, or is your requirement more to be able to distribute and distribute our processing is, you know, as far as possible to get, you know, to essentially do a kind of a brute force approach. All these approaches are valid based on the use case. I just see that essentially that the lake house is the culmination of it's nothing. It's a relatively new term introduced by Databricks a couple of years ago. This is the culmination of basically what's been a long time trend. And what we see in the cloud is that as we start seeing data warehouses as a check box items say, "Hey, we can basically source data in cloud storage, in S3, "Azure Blob Store, you know, whatever, "as long as it's in certain formats, "like, you know parquet or CSP or something like that." I see that as becoming kind of a checkbox item. So to that extent, I think that the lake house, depending on how you define is already reality. And in some cases, maybe new terminology, but not a whole heck of a lot new under the sun. >> Yeah. And Dave Menninger, I mean a lot of these, thank you Tony, but a lot of this is going to come down to, you know, vendor marketing, right? Some people just kind of co-op the term, we talked about you know, data mesh washing, what are your thoughts on this? (laughing) >> Yeah, so I used the term data platform earlier. And part of the reason I use that term is that it's more vendor neutral. We've tried to sort of stay out of the vendor terminology patenting world, right? Whether the term lake houses, what sticks or not, the concept is certainly going to stick. And we have some data to back it up. About a quarter of organizations that are using data lakes today, already incorporate data warehouse functionality into it. So they consider their data lake house and data warehouse one in the same, about a quarter of organizations, a little less, but about a quarter of organizations feed the data lake from the data warehouse and about a quarter of organizations feed the data warehouse from the data lake. So it's pretty obvious that three quarters of organizations need to bring this stuff together, right? The need is there, the need is apparent. The technology is going to continue to converge. I like to talk about it, you know, you've got data lakes over here at one end, and I'm not going to talk about why people thought data lakes were a bad idea because they thought you just throw stuff in a server and you ignore it, right? That's not what a data lake is. So you've got data lake people over here and you've got database people over here, data warehouse people over here, database vendors are adding data lake capabilities and data lake vendors are adding data warehouse capabilities. So it's obvious that they're going to meet in the middle. I mean, I think it's like Tony says, I think we should declare victory and go home. >> As hell. So just a follow-up on that, so are you saying the specialized lake and the specialized warehouse, do they go away? I mean, Tony data mesh practitioners would say or advocates would say, well, they could all live. It's just a node on the mesh. But based on what Dave just said, are we gona see those all morphed together? >> Well, number one, as I was saying before, there's always going to be this sort of, you know, centrifugal force or this tug of war between do we centralize the data, do we virtualize? And the fact is I don't think that there's ever going to be any single answer. I think in terms of data mesh, data mesh has nothing to do with how you're physically implement the data. You could have a data mesh basically on a data warehouse. It's just that, you know, the difference being is that if we use the same physical data store, but everybody's logically you know, basically governing it differently, you know? Data mesh in space, it's not a technology, it's processes, it's governance process. So essentially, you know, I basically see that, you know, as I was saying before that this is basically the culmination of a long time trend we're essentially seeing a lot of blurring, but there are going to be cases where, for instance, if I need, let's say like, Upserve, I need like high concurrency or something like that. There are certain things that I'm not going to be able to get efficiently get out of a data lake. And, you know, I'm doing a system where I'm just doing really brute forcing very fast file scanning and that type of thing. So I think there always will be some delineations, but I would agree with Dave and with Doug, that we are seeing basically a confluence of requirements that we need to essentially have basically either the element, you know, the ability of a data lake and the data warehouse, these need to come together, so I think. >> I think what we're likely to see is organizations look for a converge platform that can handle both sides for their center of data gravity, the mesh and the fabric virtualization vendors, they're all on board with the idea of this converged platform and they're saying, "Hey, we'll handle all the edge cases "of the stuff that isn't in that center of data gravity "but that is off distributed in a cloud "or at a remote location." So you can have that single platform for the center of your data and then bring in virtualization, mesh, what have you, for reaching out to the distributed data. >> As Dave basically said, people are happy when they virtualized data. >> I think we have at this point, but to Dave Menninger's point, they are converging, Snowflake has introduced support for unstructured data. So obviously literally splitting here. Now what Databricks is saying is that "aha, but it's easy to go from data lake to data warehouse "than it is from databases to data lake." So I think we're getting into semantics, but we're already seeing these two converge. >> So take somebody like AWS has got what? 15 data stores. Are they're going to 15 converge data stores? This is going to be interesting to watch. All right, guys, I'm going to go down and list do like a one, I'm going to one word each and you guys, each of the analyst, if you would just add a very brief sort of course correction for me. So Sanjeev, I mean, governance is going to to be... Maybe it's the dog that wags the tail now. I mean, it's coming to the fore, all this ransomware stuff, which you really didn't talk much about security, but what's the one word in your prediction that you would leave us with on governance? >> It's going to be mainstream. >> Mainstream. Okay. Tony Baer, mesh washing is what I wrote down. That's what we're going to see in 2022, a little reality check, you want to add to that? >> Reality check, 'cause I hope that no vendor jumps the shark and close they're offering a data niche product. >> Yeah, let's hope that doesn't happen. If they do, we're going to call them out. Carl, I mean, graph databases, thank you for sharing some high growth metrics. I know it's early days, but magic is what I took away from that, so magic database. >> Yeah, I would actually, I've said this to people too. I kind of look at it as a Swiss Army knife of data because you can pretty much do anything you want with it. That doesn't mean you should. I mean, there's definitely the case that if you're managing things that are in fixed schematic relationship, probably a relation database is a better choice. There are times when the document database is a better choice. It can handle those things, but maybe not. It may not be the best choice for that use case. But for a great many, especially with the new emerging use cases I listed, it's the best choice. >> Thank you. And Dave Menninger, thank you by the way, for bringing the data in, I like how you supported all your comments with some data points. But streaming data becomes the sort of default paradigm, if you will, what would you add? >> Yeah, I would say think fast, right? That's the world we live in, you got to think fast. >> Think fast, love it. And Brad Shimmin, love it. I mean, on the one hand I was saying, okay, great. I'm afraid I might get disrupted by one of these internet giants who are AI experts. I'm going to be able to buy instead of build AI. But then again, you know, I've got some real issues. There's a potential backlash there. So give us your bumper sticker. >> I'm would say, going with Dave, think fast and also think slow to talk about the book that everyone talks about. I would say really that this is all about trust, trust in the idea of automation and a transparent and visible AI across the enterprise. And verify, verify before you do anything. >> And then Doug Henschen, I mean, I think the trend is your friend here on this prediction with lake house is really becoming dominant. I liked the way you set up that notion of, you know, the data warehouse folks coming at it from the analytics perspective and then you get the data science worlds coming together. I still feel as though there's this piece in the middle that we're missing, but your, your final thoughts will give you the (indistinct). >> I think the idea of consolidation and simplification always prevails. That's why the appeal of a single platform is going to be there. We've already seen that with, you know, DoOP platforms and moving toward cloud, moving toward object storage and object storage, becoming really the common storage point for whether it's a lake or a warehouse. And that second point, I think ESG mandates are going to come in alongside GDPR and things like that to up the ante for good governance. >> Yeah, thank you for calling that out. Okay folks, hey that's all the time that we have here, your experience and depth of understanding on these key issues on data and data management really on point and they were on display today. I want to thank you for your contributions. Really appreciate your time. >> Enjoyed it. >> Thank you. >> Thanks for having me. >> In addition to this video, we're going to be making available transcripts of the discussion. We're going to do clips of this as well we're going to put them out on social media. I'll write this up and publish the discussion on wikibon.com and siliconangle.com. No doubt, several of the analysts on the panel will take the opportunity to publish written content, social commentary or both. I want to thank the power panelists and thanks for watching this special CUBE presentation. This is Dave Vellante, be well and we'll see you next time. (bright music)

(gentle music) >> Okay, thank you Adam. Thank you everyone for joining us on the main stage here, folks watching, appreciate it. I'm John Furrier, Dave Vellante co-hosts of theCube. We're here in the main stage to talk to the two main players of Totogi, Danielle Royston, CEO as of today, the big news. Congratulations. >> Danielle: Yeah. Thank you. >> And Robin Langdon the CTO, Totogi. >> Robin: Thanks. So big news, CEO news today and $100 million investment. Every wants to know where's all the action? Why is this so popular right now? (Danielle chuckles) What's going on? Give us the quick update. >> Yeah, I met the Totogi guys and they have this great product I was really excited about. They're focused purely on telco software and bringing, coupling that with the Public Cloud, which is everything that I talk about, what I've been about for so long. And I really wanted to give them enough funding so they could focus on building great products. A lot of times, telcos, startups, you know they try to get a quick win. They kind of chase the big guys and I really wanted to make sure they were focused on building a great product. #2, I really wanted to show the industry, they had the funding they needed to be a real player. This wasn't like $5 million or a couple million dollars, so that was really important. And then #3, I want to make sure that we could hire great talent and you need money for compensation. And so $100 million it is. >> $100 million is a lot of fresh fat financing as they say. I got to ask you, what's different? Because I've been researching on the refactoring aspect of with the Cloud, obviously public cloud with AWS, a big deal. What's different about the charging aspect of this? >> Yeah I mean, charging hasn't been exciting, maybe ever. I mean, it's kind of like this really sort of sleepy area, but I think what the Totogi guys are doing is they're really coupling the idea of charging and network data to bring hyper-personalization to subscribers. And I think that's where it changes from being a charging engine to become an engagement engine. Telcos know more about us than Google, which is kind of crazy to think about it. They know when we wake up, they know what apps we use. If we call or text, if we game or stream and it's time to start using that data to drive a better experience to us. And I think to Totogi is enabling that. I'm super excited to do that. >> So Robin, I wonder if you could talk about that a little bit. I mean, maybe we get into the plumbing and I don't want to go too deep in it, but I think it's important because we've seen this movie before where people take their on-prem stacks, they wrap it in containers and they shove it into the Public Cloud and they say, "Hey, we're cloud too." If reading a press release, you guys are taking advantage of things like Amazon Nitro of course, but also Graviton and Graviton2 and eventually 3, which is the underlying capabilities, give you a cloud native advantage. Can you explain that a little bit? >> Yeah, absolutely. I mean, we wanted to build this in the Cloud using all of those great cloud innovations. So Graviton2, DynamoDB and using their infrastructure, just allowing us to be able to scale out. These all available to us to use and essentially free for us to use. And it's great, so as you say, we're not shoehorning something in that's decade's old technology, wrapping it in some kind of container and pushing it in. Which is just then, you just can't use any of those great innovations. >> And you've selected DynamoDB as the database. Okay, that's fine. We don't have to get so much into why, but maybe you could explain the advantage because I saw some benchmark numbers which were, like an order of magnitude greater than the competition, like share with us, why? How you were able to get there? And maybe share those numbers. >> Yeah, no, we do. So we just launched our benchmark. So, a million transactions per second. So we just blew away everyone else out there. And that's really because we could take advantage of all that great AWS technology in there and the database side we're using DynamoDB, where we had a huge debate about using what kind of database to go and use? There's a lot of people out there probably get very religious about the kind of database technology that you should be using. And whether it should it be SQL in-memory object database type technology, but really a single table design, gives you that true scalability. You can just horizontally scale that easily, across the whole planet. >> You know, Danielle. Again, I said that we've seen this movie before. There are a lot of parallels in telco with the enterprise. And if you look at enterprise SAS pricing, a lot of it is very static, kind of lock you in, per seat pricing, kind of an old model. And you're seeing a lot of the modern SAS companies who are emerging with a consumption pricing models. How are you guys thinking about pricing? >> Yeah, I don't know of any other company in telco that's starting to price by usage. And that is a very standard offering with the cloud providers, right? Google we know, Amazon, all those guys have a price by the API, price by the transaction. So we're really excited to offer that to telcos. They've been asking for it for awhile, right? Pay for what you need, when you need it, by the use. And so we're really excited to offer that, but I think what's really cool is the idea of a free tier, right? And so I think it's smaller telcos have a trade-off to make, whether, am I going to buy the best technology and pay through the nose and maybe at an unaffordable level, or do I compromise and buy something more affordable, but not as great. And what's so great about Totogi, it's the same product just priced for what you need. And so I think a CSP it'll, below 250,000 subscribers should be able to use the Totogi absolutely for free. And that is, and it's the same product that the big guy would get. So it's not a junior version or scaled back. And so I think that's really exciting. I think we're the only ones that do it. So here we go. >> Love the freemium model. So Robin, maybe you could explain why that's so much, so important in the charging space, because you've got a lot of different options that you want to configure for the consumer. >> Yeah. >> Maybe you could talk about sort of how the old world does that, the old guard and how long it takes and how you're approaching this. >> Yeah so it's, I mean traditionally, charging design, there's as you say, there's lots of different pricing leavers you want to be able to move and change to charge different people. And these systems, even if they say they're configurable, if they normally turn into an IT project where it takes weeks, months, even years to build out the system, you know, marketing can't just go in there and configure the dials and push out your new plans and tariffs. They have to go and create a requirement specification. They hand it down to IT. Those guys go and create a big change project. And by the time they're finished, the market's moved on. They're on to their next plan, their next tariff to go and build. So we wanted to create something that was truly configurable from a marketing standpoint. You know, user-friendly, they can go in there, configure it and be live in minutes, not even days or weeks. >> No, IT necessary. >> Robin: No IT necessary. >> So you know, I've been thinking about, John and I talk about this all the time, It's that there's a data play here. And what I think you're doing is actually building a data product. I think there's a new metric emerging in the industry, which is how long does it take me to go from idea to monetization with a data product. And that's what this is. This is a data product >> Yeah. >> for your customers. >> Absolutely, what Robin was talking about is totally the way the industry works. It's weeks before you have an idea and get it out to the market. And like Robin was mentioning, the market's changed by the time you get it out there, the data's stale. And so we researched every single plan in the world from every single CSP. There is about 30,000 plans in the world, right? The bigger you are, the more plans you have. On average, a tier one telco has 40 to 50 plans. And so how many offers, I mean think about, that's how many phones to buy, plans to buy. And so we're like, let's get some insight on the plans. Let's drive it into a standardization, right? Let's make them, which ones work, which ones don't. And that's, I think you're right. I think it's a data play and putting the power back into the marketer's hands and not with IT. >> So there's a lot of data on-prem. Explain why I can't do this with my on-prem data. >> Oh, well today that, I mean, sorry if you want to jump in. Feel free to jump in, right. But today, the products are designed in a way where they're, perpetually licensed, by the subscriber, rigid systems, not API based. I mean, there might be an API, but you got to pay through the nose to use it or you got to use the provider's people to code against it. They're inflexible. They were written when voice was the primary revenue driver, not data, right? And so they've been shoehorned, right? Like Robin was saying, shoehorned to be able to move into the world that we are now. I mean, when the iPhone came about that introduced apps and data went through the roof and the systems were written for voice, not written for data. >> And that's a good point, if you think about the telco industry, it seems like it could be a glacier that just needs to just break and just like, just get modern because we all have phones. We have apps. We can delete them. And the billing plans, like either nonexistent or it has to be all free. >> Well I mean, I'll ask you. Do you know what your billing plan is? Do you know how much data you use on a monthly basis? No one knows. >> I have no clue. >> A lot. >> No one. And so what you do is you buy unlimited. >> Dave: Right. >> You overpay. And so what we're seeing in the plans is that if you actually knew how much you used, you would be able to maybe pay less, which I remember the telcos are not excited to hear that message, but it's a win for the subscriber. And if you could >> I mean it's only >> accurately predict that. >> get lower and lower. I having a conversation last night at dinner with industry analysts, we're talking about a vehicle e-commerce, commerce in your car as you're driving. You can get that kind of with a 5G. The trend is transactions everywhere, ad-hoc, ephemeral... >> Yeah. >> The new apps are going to demand this kind of subscriber billing. >> Yeah >> Do people get this? Are you guys the only ones kind of like on this? >> No I think people have been talking about it for years. I think there's vendors out there that have been trying to offer this idea of like, build your own plan and all that other stuff but I think it's more than just minutes, text and data. It's starting to really understand what subscribers are using, right? Are you a football fan? Are you a golf fan? Are you a shopper? Are you a concert goer? And couple that with how you use your phone and putting out offers that are really exciting to subscribers so that we love our telco. Like we should be loving our telco. And I don't... I don't know that people talk >> They saved us >> about loving their telco. >> from the pandemic >> They saved us during the pandemic. The internet didn't crash, we got our zoom meetings. We got everything going on. What's the technical issue on solving these problems? Is it just legacy? Is it just mindset? Robin, what's your take on that? >> I'll keep talking as long as Robin will let me. (Daniel laughing) >> So the big technical issues, you're trying to build in this flexibility so that you can have, we don't know what people are going to configure in the future. It's minutes and text messages are given away for free. They're unlimited. Data is where it's at, about charging for apps and about using all that data in the network the telcos have, which is extremely valuable and there's a wealth of information in there that can be used to be monetized and push that out. And they need a charging system on top that can manage that and we have the flexibility that you don't have to go off and then start creating programs and IT projects that are going to do that. >> Well it's funny Danielle, you say that the telcos might not like that, right? 'Cause you might pay less. But in fact, that is the kind of on-prem mindset because when you have a fixed resource, you say, okay, don't use too much because we have to buy more. Or you overbuy to your point. The cloud mindset is, I'll try it. I'll try some more, I'll try some more. I'm aligning it with business value. Oh, I'm making money. Oh, great. I'm going to keep buying more. And it's very clear. It's transparent where the business value is. So my question is when you think about your charging engine and all this data conversation, is there more than just a charging engine in this platform? >> Well, I think just going back to what Robin was talking about. I think what Totogi is doing differently is by building it on the Public Cloud gives you virtually unlimited resources, right? In a couple of different directions, certainly hardware and capacity and scalability and all those other things, right? But also as Amazon is putting out more and more product, when you build it in this new way, you can take advantage of these new services very, very easily. And that is a different mindset. It's a different way to deploy applications. And I think that's what makes Totogi really different. You couldn't build Totogi on-premise because you need the infinite scalability. You need the machine learning, you need the AI of Amazon, which they have been investing in for decades, if they now charge you by the API call. And you get to use it like you were saying. Just give it a try, don't like it, stop. And it's just a completely different way of thinking, yeah. >> If I have to ask you a question about the Public Cloud, because the theme here in Cloud City is the Public Cloud is driving innovation, which is also includes disruption. And the new brands are coming in, old brands are either reinventing themselves or falling away. What is the Public Cloud truly enabling? Is it the scale? Is it the data? Is it the refactoring capability? What is the real driver of the Public Cloud innovation? >> I think the insight that CSPs are going to have is what Jamie Dimon had in banking. Like I think he was pretty famously saying, "I'm never going to use the Public Cloud. Our data is too precious, you know, regulations and all that stuff." But I think the insight they're going to have, and I hopefully, I do a keynote and I mentioned this, which is feature velocity. The ability to put out features in a day or two. Our feature velocity in telco is months. Months, months. >> Seriously? >> Yeah, sometimes years. It's just so slow between big iterations of new capability and to be able to put out new features in minutes or days and be able to outmaneuver your competition is unheard of. So the CSPs that starts to get this, it's going to be a real big get, and then they're going to start to.. (Danielle makes swishing sound) >> We just interviewed (Dave speaking indistinctly) a venture capitalist, Dave and I last month. And he's a big investor in Snowflake, on the big deals. He said that the new strategy that's working is you go to be agile with feature acceleration. We just talked about this at lunch and you get data. And you can dismantle the bad features quickly and double down >> Yup. >> on the winners. >> Ones that are working. So what used to be feature creep now is a benefit if you play it right? >> Danielle: It's feature experimentation. >> That's essentially what you- >> It's experimentation, right? And you're like, that one worked, this one didn't, kill that one, double down on this one, go faster and faster and so feature experimentation, which you can't do in telco, because every time we ask for a feature from your current vendor, it's hundreds of thousands, if not millions of dollars. So you don't experiment. And so yeah- >> You can make features disposable. >> Correct. And I think that we just discovered that on this stage just now. (group chuckling) >> Hey look at this. Digital revolution, DR. Telco DR. >> Yeah. >> Great to have you guys. >> This is super awesome. Thanks so much. >> You guys are amazing. Congratulations. And we're looking forward to the more innovation stories again, get out there, get the momentum. Great stuff. >> Danielle: It's going to be great. >> And awesome. >> Feature experimentation. >> Yeah. >> Hashtag. >> And Dave and I are going to head back over to our Cube set here, here on the main stage. We'll toss it back to the Adam in the studio. Adam, back to you and take it from here.

>>from around the globe. It's the Cube with digital coverage of AWS reinvent 2020 sponsored by Intel, AWS and our community partners. >>Hey, welcome back already, Jeffrey here with the Cube coming to you from Palo Alto studios today for our ongoing coverage of aws reinvent 2020. It's a digital event like everything else in 2020. We're excited for our next segment, so let's jump into it. We're joined in our next segment by Andrew Rafa. He is the principal and zero trust offering lead at the Light and Touche LLP. Andrew, great to see you. >>Thanks for having me. >>Absolutely. And joining him is Robbie Deval. He is the AWS cyber risk lead for Deloitte and Touche LLP. Robbie, Good to see you as well. >>Hey, Jeff, good to see you as well. >>Absolutely. So let's jump into it. You guys are all about zero trust and I know a little bit about zero trust I've been going to are safe for a number of years and I think one of the people that you like to quote analysts chase Cunningham from Forrester, who's been doing a lot of work around zero trust. But for folks that aren't really familiar with it. Andrew, why don't you give us kind of the 101? About zero trust. What is it? What's it all about? And why is it important? >>Sure thing. So is your trust is, um, it's a conceptual framework that helps organizations deal with kind of the ubiquitous nature of modern enterprise environments. Um, and then its course. Your trust commits to a risk based approach to enforcing the concept of least privileged across five key pillars those being users, workloads, data networks and devices. And the reason we're seeing is your trust really come to the forefront is because modern enterprise environments have shifted dramatically right. There is no longer a defined, clearly defined perimeter where everything on the outside is inherently considered, considered untrusted, and everything on the inside could be considered inherently trusted. There's a couple what I call macro level drivers that are, you know, changing the need for organizations to think about securing their enterprises in a more modern way. Um, the first macro level driver is really the evolving business models. So as organizations are pushing to the cloud, um, maybe expanding into into what they were considered high risk geography is dealing with M and A transactions and and further relying on 3rd and 4th parties to maintain some of their critical business operations. Um, the data and the assets by which the organization, um transact are no longer within the walls of the data center. Right? So, again, the perimeter is very much dissolved. The second, you know, macro level driver is really the shifting and evolving workforce. Um, especially given the pandemic and the need for organizations to support almost an entirely remote workforce nowadays, um, organizations, they're trying to think about how they revamp their traditional VPN technologies in order to provide connectivity to their employees into other third parties that need to get access to, uh, the enterprise. So how do we do so in a secure, scalable and reliable way and then the last kind of macro level driver is really the complexity of the I t landscape. So, you know, in legacy environment organizations on Lee had to support managed devices, and today you're seeing the proliferation of unmanaged devices, whether it be you know, B y o d devices, um, Internet of things, devices or other smart connected devices. So organizations are now, you know, have the need to provide connectivity to some of these other types of devices. But how do you do so in a way that, you know limits the risk of the expanding threat surface that you might be exposing your organization to by supporting from these connected devices? So those are some three kind of macro level drivers that are really, you know, constituting the need to think about security in a different >>way. Right? Well, I love I downloaded. You guys have, ah zero trust point of view document that that I downloaded. And I like the way that you you put real specificity around those five pillars again users, workloads, data networks and devices. And as you said, you have to take this kind of approach that it's kind of on a need to know basis. The less, you know, at kind of the minimum they need to know. But then, to do that across all of those five pillars, how hard is that to put in place? I mean, there's a There's a lot of pieces of this puzzle. Um, and I'm sure you know, we talk all the time about baking security and throughout the entire stack. How hard is it to go into a large enterprise and get them started or get them down the road on this zero trust journey? >>Yeah. So you mentioned the five pillars. And one thing that we do in our framework because we put data at the center of our framework and we do that on purpose because at the end of the day, you know, data is the center of all things. It's important for an organization to understand. You know what data it has, what the criticality of that data is, how that data should be classified and the governance around who and what should access it from a no users workloads, uh, networks and devices perspective. Um, I think one misconception is that if an organization wants to go down the path of zero trust, there's a misconception that they have to rip out and replace everything that they have today. Um, it's likely that most organizations are already doing something that fundamentally aligned to the concept of these privilege as it relates to zero trust. So it's important to kind of step back, you know, set a vision and strategy as faras What it is you're trying to protect, why you're trying to protect it. And what capability do you have in place today and take more of an incremental and iterative approach towards adoption, starting with some of your kind of lower risk use cases or lower risk parts of your environment and then implementing lessons learned along the way along the journey? Um, before enforcing, you know more of those robust controls around your critical assets or your crown jewels, if you >>will. Right? So, Robbie, I want to follow up with you, you know? And you just talked about a lot of the kind of macro trends that are driving this and clearly covert and work from anywhere is a big one. But one of the ones that you didn't mention that's coming right around the pike is five g and I o t. Right, so five g and and I o. T. We're going to see, you know, the scale and the volume and the mass of machine generated data, which is really what five g is all about, grow again exponentially. We've seen enough curves up into the right on the data growth, but we've barely scratched the surface and what's coming on? Five G and I o t. How does that work into your plans? And how should people be thinking about security around this kind of new paradigm? >>Yeah, I think that's a great question, Jeff. And as you said, you know, I UT continues to accelerate, especially with the recent investments and five G that you know pushing, pushing more and more industries and companies to adopt a coyote. Deloitte has been and, you know, helping our customers leverage a combination of these technologies cloud, Iot, TML and AI to solve their problems in the industry. For instance, uh, we've been helping restaurants automate their operations. Uh, we've helped automate some of the food safety audit processes they have, especially given the code situation that's been helping them a lot. We are currently working with companies to connect smart, wearable devices that that send the patient vital information back to the cloud. And once it's in the cloud, it goes through further processing upstream through applications and data. Let's etcetera. The way we've been implementing these solutions is largely leveraging a lot of the native services that AWS provides, like device manager that helps you onboard hundreds of devices and group them into different categories. Uh, we leveraged device Defender. That's a monitoring service for making sure that the devices are adhering to a particular security baseline. We also have implemented AWS green grass on the edge, where the device actually resides. Eso that it acts as a central gateway and a secure gateway so that all the devices are able to connect to this gateway and then ultimately connect to the cloud. One common problem we run into is ah, lot of the legacy i o t devices. They tend to communicate using insecure protocols and in clear text eso we actually had to leverage AWS lambda Function on the edge to convert these legacy protocols. Think of very secure and Q t t protocol that ultimately, you know, sense data encrypted to the cloud eso the key thing to recognize. And then the transformational shift here is, um, Cloud has the ability today to impact security off the device and the edge from the cloud using cloud native services, and that continues to grow. And that's one of the key reasons we're seeing accelerated growth and adoption of Iot devices on did you brought up a point about five G and and that's really interesting. And a recent set of investments that eight of us, for example, has been making. And they launched their AWS Waveland zones that allows you to deploy compute and storage infrastructure at the five G edge. So millions of devices they can connect securely to the computer infrastructure without ever having to leave the five g network Our go over the Internet insecurely talking to the cloud infrastructure. Uh, that allows us to actually enable our customers to process large volumes of data in a short, near real time. And also it increases the security of the architectures. Andi, I think truly, uh, this this five g combination with I o t and cloudy, I m l the are the technologies of the future that are collectively pushing us towards a a future where we're gonna Seymour smart cities that come into play driverless connected cars, etcetera. >>That's great. Now I wanna impact that a little bit more because we are here in aws re invent and I was just looking up. We had Glenn Goran 2015, introducing a W S s I O T Cloud. And it was a funny little demo. They had a little greenhouse, and you could turn on the water and open up the windows. But it's but it's a huge suite of services that you guys have at your disposal. Leveraging aws. I wonder, I guess, Andrew, if you could speak a little bit more suite of tools that you can now bring to bear when you're helping your customers go to the zero trust journey. >>Yeah, sure thing. So, um, obviously there's a significant partnership in place, and, uh, we work together, uh, pretty tremendously in the market, one of the service are one of solution offering that we've built out which we dub Delight Fortress, um is a is a concept that plays very nicely into our zero trust framework. More along the kind of horizontal components of our framework, which is really the fabric that ties it all together. Um s o the two horizontal than our framework around telemetry and analytics. A swell the automation orchestration. If I peel back the automation orchestration capability just a little bit, um, we we built this avoid fortress capability in order for organizations to kind of streamline um, some of the vulnerability management aspect of the enterprise. And so we're able through integration through AWS, Lambda and other functions, um, quickly identify cloud configuration issues and drift eso that, um, organizations cannot only, uh, quickly identify some of those issues that open up risk to the enterprise, but also in real time. Um, take some action to close down those vulnerabilities and ultimately re mediate them. Right? So it's way for, um, to have, um or kind of proactive approach to security rather than a reactive approach. Everyone knows that cloud configuration issues are likely the number one kind of threat factor for Attackers. And so we're able to not only help organizations identify those, but then closed them down in real time. >>Yeah, it's interesting because we hear that all the time. If there's a breach and if if they w s involved often it's a it's a configuration. You know, somebody left the door open basically, and and it really drives something you were talking about. Ravi is the increasing important of automation, um, and and using big data. And you talked about this kind of horizontal tele metrics and analytics because without automation, these systems are just getting too big and and crazy for people Thio manage by themselves. But more importantly, it's kind of a signal to noise issue when you just have so much traffic, right? You really need help surfacing. That signals you said so that your pro actively going after the things that matter and not being just drowned in the things that don't matter. Ravi, you're shaking your head up and down. I think you probably agree with this point. >>Yeah, yeah, Jeff and definitely agree with you. And what you're saying is truly automation is a way off dealing with problems at scale. When when you have hundreds of accounts and that spans across, you know, multiple cloud service providers, it truly becomes a challenge to establish a particular security baseline and continue to adhere to it. And you wanna have some automation capabilities in place to be able to react, you know, and respond to it in real time versus it goes down to a ticketing system and some person is having to do you know, some triaging and then somebody else is bringing in this, you know, solution that they implement. And eventually, by the time you're systems could be compromised. So ah, good way of doing this and is leveraging automation and orchestration is just a capability that enhances your operational efficiency by streamlining summed Emmanuel in repetitive tasks, there's numerous examples off what automation and orchestration could do, but from a security context. Some of the key examples are automated security operations, automated identity provisioning, automated incident response, etcetera. One particular use case that Deloitte identified and built a solution around is the identification and also the automated remediation of Cloud security. Miss Consideration. This is a common occurrence and use case we see across all our customers. So the way in the context of a double as the way we did this is we built a event driven architectures that's leveraging eight of us contribute config service that monitors the baselines of these different services. Azzan. When it detects address from the baseline, it fires often alert. That's picked up by the Cloudwatch event service that's ultimately feeding it upstream into our workflow that leverages event bridge service. From there, the workflow goes into our policy engine, which is a database that has a collection off hundreds of rules that we put together uh, compliance activities. It also matched maps back to, ah, large set of controls frameworks so that this is applicable to any industry and customer, and then, based on the violation that has occurred, are based on the mis configuration and the service. The appropriate lambda function is deployed and that Lambda is actually, uh, performing the corrective actions or the remediation actions while, you know, it might seem like a lot. But all this is happening in near real time because it is leveraging native services. And some of the key benefits that our customers see is truly the ease of implementation because it's all native services on either worse and then it can scale and, uh, cover any additional eight of those accounts as the organization continues to scale on. One key benefit is we also provide a dashboard that provides visibility into one of the top violations that are occurring in your ecosystem. How many times a particular lambda function was set off to go correct that situation. Ultimately, that that kind of view is informing. Thea Outfront processes off developing secure infrastructure as code and then also, you know, correcting the security guard rails that that might have drifted over time. Eso That's how we've been helping our customers and this particular solution that we developed. It's called the Lloyd Fortress, and it provides coverage across all the major cloud service providers. >>Yeah, that's a great summary. And I'm sure you have huge demand for that because he's mis configuration things. We hear about him all the time and I want to give you the last word for we sign off. You know, it's easy to sit on the side of the desk and say, Yeah, we got a big security and everything and you got to be thinking about security from from the time you're in, in development all the way through, obviously deployment and production and all the minutes I wonder if you could share. You know, you're on that side of the glass and you're out there doing this every day. Just a couple of you know, kind of high level thoughts about how people need to make sure they're thinking about security not only in 2020 but but really looking down the like another road. >>Yeah, yeah, sure thing. So, you know, first and foremost, it's important to align. Uh, any transformation initiative, including your trust to business objectives. Right? Don't Don't let this come off as another I t. Security project, right? Make sure that, um, you're aligning to business priorities, whether it be, you know, pushing to the cloud, uh, for scalability and efficiency, whether it's digital transformation initiative, whether it be a new consumer identity, Uh uh, an authorization, um, capability of china built. Make sure that you're aligning to those business objectives and baking in and aligning to those guiding principles of zero trust from the start. Right, Because that will ultimately help drive consensus across the various stakeholder groups within the organization. Uh, and build trust, if you will, in the zero trust journey. Um, one other thing I would say is focus on the fundamentals. Very often, organizations struggle with some. You know what we call general cyber hygiene capabilities. That being, you know, I t asset management and data classifications, data governance. Um, to really fully appreciate the benefits of zero trust. It's important to kind of get some of those table six, right? Right. So you have to understand, you know what assets you have, what the criticality of those assets are? What business processes air driven by those assets. Um, what your data criticality is how it should be classified intact throughout the ecosystem so that you could really enforce, you know, tag based policy, uh, decisions within, within the control stack. Right. And then finally, in order to really push the needle on automation orchestration, make sure that you're using technology that integrate with each other, right? So taken a p I driven approach so that you have the ability to integrate some of these heterogeneous, um, security controls and drive some level of automation and orchestration in order to enhance your your efficiency along the journey. Right. So those were just some kind of lessons learned about some of the things that we would, uh, you know, tell our clients to keep in mind as they go down the adoption journey. >>That's a great That's a great summary s So we're gonna have to leave it there. But Andrew Robbie, thank you very much for sharing your insight and and again, you know, supporting this This move to zero trust because that's really the way it's got to be as we continue to go forward. So thanks again and enjoy the rest of your reinvent. >>Yeah, absolutely. Thanks for your time. >>All right. He's Andrew. He's Robbie. I'm Jeff. You're watching the Cube from AWS reinvent 2020. Thanks for watching. See you next time.

>> Hi and welcome to Data Cloud catalyst Women in Tech Round Table Panel discussion. I am so excited to have three fantastic female executives with me today, who have been driving transformations through data throughout their entire career. With me today is Lisa Davis, SVP and CIO OF Blue shield of California. We also have Nishita Henry who is the Chief Innovation Officer at Deloitte and Teresa Briggs who is on a variety of board of directors including our very own Snowflake. Welcome ladies. >> Thank you. >> So I am just going to dive right in, you all have really amazing careers and resumes behind you, am really curious throughout your career, how have you seen the use of data evolve throughout your career and Lisa am going to start with you. >> Thank you, having been in technology my entire career, technology and data has really evolved from being the province of a few in an organization to frankly being critical to everyone's business outcomes. Now every business leader really needs to embrace data analytics and technology. We've been talking about digital transformation, probably the last five, seven years, we've all talked about, disrupt or be disrupted, At the core of that digital transformation is the use of data. Data and analytics that we derive insights from and actually improve our decision making by driving a differentiated experience and capability into market. So data has involved as being I would say almost tactical, in some sense over my technology career to really being a strategic asset of what we leverage personally in our own careers, but also what we must leverage as companies to drive a differentiated capability to experience and remain relative in the market today. >> Nishita curious your take on, how you have seen data evolve? >> Yeah, I agree with Lisa, it has definitely become a the lifeblood of every business, right? It used to be that there were a few companies in the business of technology, every business is now a technology business. Every business is a data business, it is the way that they go to market, shape the market and serve their clients. Whether you're in construction, whether you're in retail, whether you're in healthcare doesn't matter, right? Data is necessary for every business to survive and thrive. And I remember at the beginning of my career, data was always important, but it was about storing data, it was about giving people individual reports, it was about supplying that data to one person or one business unit in silos. And it then evolved right over the course of time into integrating data into saying, alright, how does one piece of data correlate to the other and how can I get insights out of that data? Now, its gone to the point of how do I use that data to predict the future? How do I use that data to automate the future? How do I use that data not just for humans to make decisions, but for other machines to make decisions, right? Which is a big leap and a big change in how we use data, how we analyze data and how we use it for insights and involving our businesses. >> Yeah its really changed so tremendously just in the past five years, its amazing. So Teresa we've talked a lot about the Data Cloud, where do you think we are heading with that and also how can future leaders really guide their careers in data especially in those jobs where we don't traditionally think of them in the data science space? Teresa your thoughts on that. >> Yeah, well since I'm on the Snowflake Board, I'll talk a little bit about the Snowflake Data Cloud, we're getting your company's data out of the silos that exist all over your organization. We're bringing third party data in to combine with your own data and we're wrapping a governance structure around it and feeding it out to your employees so they can get their jobs done, as simple as that. I think we've all seen the pandemic accelerate the digitization of our work. And if you ever doubted that the future of work is here, it is here and companies are scrambling to catch up by providing the right amount of data, collaboration tools, workflow tools for their workers to get their jobs done. Now, it used to be as prior people have mentioned that in order to work with data you had to be a data scientist, but I was an auditor back in the day we used to work on 16 column spreadsheets. And now if you're an accounting major coming out of college joining an auditing firm, you have to be tech and data savvy because you're going to be extracting, manipulating, analyzing and auditing data, that massive amounts of data that sit in your clients IT systems. I'm on the board of Warby Parker, and you might think that their most valuable asset is their amazing frame collection, but it's actually their data, their 360 degree view of the customer. And so if you're a merchant, or you're in strategy, or marketing or talent or the Co-CEO, you're using data every day in your work. And so I think it's going to become a ubiquitous skill that any anyone who's a knowledge worker has to be able to work with data. >> Yeah I think its just going to be organic to every role going forward in the industry. So, Lisa curious about your thoughts about Data Cloud, the future of it and how people can really leverage it in their jobs for future leaders. >> Yeah, absolutely most enterprises today are, I would say, hybrid multicloud enterprises. What does that mean? That means that we have data sitting on-prem, we have data sitting in public clouds through software as a service applications. We have a data everywhere. Most enterprises have data everywhere, certainly those that have owned infrastructure or weren't born on the web. One of the areas that I love that Data Cloud is addressing is area around data portability and mobility. Because I have data sitting in various locations through my enterprise, how do I aggregate that data to really drive meaningful insights out of that data to drive better business outcomes? And at Blue Shield of California, one of our key initiatives is what we call an Experienced Cube. What does that mean? That means how do I drive transparency of data between providers, members and payers? So that not only do I reduce overhead on providers and provide them a better experience, our hospital systems are doctors, but ultimately, how do we have the member have it their power of their fingertips the value of their data holistically, so that we're making better decisions about their health care. One of the things Teresa was talking about, was the use of this data and I would drive to data democratization. We got to put the power of data into the hands of everyone, not just data scientists, yes we need those data scientists to help us build AI models to really drive and tackle these tough old, tougher challenges and business problems that we may have in our environments. But everybody in the company both on the IT side, both on the business side, really need to understand of how do we become a data insights driven enterprise, put the power of the data into everyone's hands so that we can accelerate capabilities, right? And leverage that data to ultimately drive better business results. So as a leader, as a technology leader, part of our responsibility, our leadership is to help our companies do that. And that's really one of the exciting things that I'm doing in my role now at Blue Shield of California. >> Yeah its really, really exciting time. I want to shift gears a little bit and focus on women in Tech. So I think in the past five to ten years there has been a lot of headway in this space but the truth is women are still under represented in the tech space. So what can we do to attract more women into technology quite honestly. So Nishita curious what your thoughts are on that? >> Great question and I am so passionate about this for a lot of reasons, not the least of which is I have two daughters of my own and I know how important it is for women and young girls to actually start early in their love for technology and data and all things digital, right? So I think it's one very important to start early started early education, building confidence of young girls that they can do this, showing them role models. We at Deloitte just partnered with LV Engineer to actually make comic books centered around young girls and boys in the early elementary age to talk about how heroes in tech solve everyday problems. And so really helping to get people's minds around tech is not just in the back office coding on a computer, tech is about solving problems together that help us as citizens, as customers, right? And as humanity, so I think that's important. I also think we have to expand that definition of tech, as we just said it's not just about right, database design, It's not just about Java and Python coding, it's about design, it's about the human machine interfaces, it's about how do you use it to solve real problems and getting people to think in that kind of mindset makes it more attractive and exciting. And lastly, I'd say look we have a absolute imperative to get a diverse population of people, not just women, but minorities, those with other types of backgrounds, disabilities, et cetera involved because this data is being used to drive decision making in all involved, right, and how that data makes decisions, it can lead to unnatural biases that no one intended but can happen just 'cause we haven't involved a diverse enough group of people around it. >> Absolutely, lisa curious about your thoughts on this. >> I agree with everything Nishita said, I've been passionate about this area, I think it starts with first we need more role models, we need more role models as women in these leadership roles throughout various sectors. And it really is it starts with us and helping to pull other women forward. So I think certainly it's part of my responsibility, I think all of us as female executives that if you have a seat at the table to leverage that seat at the table to drive change, to bring more women forward more diversity forward into the boardroom and into our executive suites. I also want to touch on a point Nishita made about women we're the largest consumer group in the company yet we're consumers but we're not builders. This is why it's so important that we start changing that perception of what tech is and I agree that it starts with our young girls, we know the data shows that we lose our like young girls by middle school, very heavy peer pressure, it's not so cool to be smart, or do robotics, or be good at math and science, we start losing our girls in middle school. So they're not prepared when they go to high school, and they're not taking those classes in order to major in these STEM fields in college. So we have to start the pipeline early with our girls. And then I also think it's a measure of what your boards are doing, what is the executive leadership in your goals around diversity and inclusion? How do we invite more diverse population to the decision making table? So it's really a combination of efforts. One of the things that certainly is concerning to me is during this pandemic, I think we're losing one in four women in the workforce now because of all the demands that our families are having to navigate through this pandemic. The last statistic I saw in the last four months is we've lost 850,000 women in the workforce. This pipeline is critical to making that change in these leadership positions. >> Yeah its really a critical time and now we are coming to the end of this conversation I want to ask you Teresa what would be a call to action to everyone listening both men and women since its to be solved by everyone to address the gender gap in the industry? >> I'd encourage each of you to become an active sponsor. Research shows that women and minorities are less likely to be sponsored than white men. Sponsorship is a much more active form than mentorship. Sponsorship involves helping someone identify career opportunities and actively advocating for them and those roles opening your network, giving very candid feedback. And we need men to participate too, there are not enough women in tech to pull forward and sponsor the high potential women that are in our pipelines. And so we need you to be part of the solution. >> Nishita real quickly what would be your call to action to everyone? >> I'd say look around your teams, see who's on them and make deliberate decisions about diversifying those teams, as positions open up, make sure that you have a diverse set of candidates, make sure that there are women that are part to that team and make sure that you are actually hiring and putting people into positions based on potential not just experience. >> And real quickly Lisa, we'll close it out with you what would your call to action be? >> Wow, it's hard to what Nishita and what Tricia shared I think we're very powerful actions. I think it starts with us. Taking action at our own table, making sure you're driving diverse panels and hiring setting goals for the company, having your board engaged and holding us accountable and driving to those goals will help us all see a better outcome with more women at the executive table and diverse populations. >> Great advice and great action for all of us to take. Thank you all so much for spending time with me today and talking about this really important issue, I really appreciate it. Stay with us.

>> Hi, and welcome to Data Cloud Catalyst, Women in Tech Round Table Panel Discussion. I am so excited to have three fantastic female executives with me today who have been driving transformation through data throughout their entire career. With me today is Lisa Davis, SVP and CIO of Blue Shield of California. We also have Nishita Henry, who is the Chief Innovation Officer at Deloitte and Theresa Briggs, who is on a variety of board of directors, including our own very own Snowflake. Welcome, ladies. >> Thank you. >> Thank you. >> So I'm just going to dive right in. You all have really amazing careers and resumes behind you. I'm really curious, throughout your career, how have you seen the use of data evolve throughout your career? And, Lisa, I'm going to start with you. >> Thank you. Having been in technology my entire career, technology and data has really evolved from being the province of a few in an organization to frankly being critical to everyone's business outcomes. Now every business leader really needs to embrace data analytics and technology. We've been talking about digital transformation probably the last five, seven years, we've all talked about disrupt or be disrupted. At the core of that digital transformation is the use of data. Data and analytics that we derive insights from and actually improve our decision making by driving a differentiated experience and capability into market. So data has involved as being, I would say, almost tactical in some sense over my technology career, to really being a strategic asset of what we leverage personally in our own careers, but also what we must leverage as companies to drive a differentiated capability to experience and remain relative in the market today. >> Nishita, curious your take on how you've seen data evolve? >> Yeah, I agree with Lisa. It has definitely become the lifeblood of every business, right? It used to be that there were a few companies in the business of technology, every business is now a technology business. Every business is a data business. It is the way that they go to market, shape the market and serve their clients. Whether you're in construction, whether you're in retail, whether you're in healthcare it doesn't matter, right? Data is necessary for every business to survive and thrive. And I remember at the beginning of my career, data was always important but it was about storing data. It was about giving people individual reports, it was about supplying that data to one person or one business unit in silos. And it then evolved right over the course of time into integrating data and to saying, all right, how does one piece of data correlate to the other and how can I get insights out of that data? Now, let's go on to the point of how do I use that data to predict the future? How do I use that data to automate the future? How do I use that data not just for humans to make decisions, but for other machines to make decisions, right? Which is a big leap. And a big change in how we use data, how we analyze data and how we use it for insights in evolving our businesses. >> Yeah, it's really changed so tremendously just in the past five years. It's amazing. So Teresa, we've talked a lot about the Data Cloud, where do you think we're heading with that? And also, how can future leaders really guide their careers in data, especially in those jobs where we don't traditionally think of them in the data science space? Curious your thoughts on that? >> Yeah, well, since I'm on the Snowflake board, I'll talk a little bit about the Snowflake Data Cloud. Now we're getting your company's data out of the silos that exists all over your organization, we're bringing third party data in to combine with your own data, and we're wrapping a governance structure around it and feeding it out to your employees so that they can get their jobs done. And is as simple as that. I think we've all seen the pandemic accelerate the digitization of our work. And if you ever doubted the future of work is here, it is here. And companies are scrambling to catch up by providing the right amount of data, collaboration tools, workflow tools for their workers to get their jobs done. Now, it used to be as prior people have mentioned that in order to work with data you had to be a data scientist. But I was an auditor back in the day and we used to work on 16 columns spreadsheet. And now if you're an accounting major coming out of college joining an auditing firm, you have to be tech and data savvy because you're going to be extracting, manipulating, analyzing and auditing data, that massive amounts of data that sit in your client's IT systems. I'm on the board of Warby Parker, and you might think that their most valuable asset is their amazing frame collection, but it's actually their data, their 360 degree view of the customer. And so if you're a merchant or you're in strategy, or marketing or talent or the co-CEO, you're using data every day in your work. And so I think it's going to become a ubiquitous skill that anyone who's a knowledge worker has to be able to work with data. >> Yeah, I think it's just going to be organic to every role going forward in the industry. So Lisa, curious about your thoughts about Data Cloud, the future of it, and how people can really leverage it in their jobs from future leaders? >> Yeah, absolutely. Most enterprises today are, I would say, hybrid multi cloud enterprises. What does that mean? That means that we have data sitting on prem, we have data sitting in public clouds through software as a service applications, we have a data everywhere, most enterprises have data everywhere. Certainly those that have owned infrastructure or weren't born on the web. One of the areas that I love that Data Cloud is addressing is the area around data portability and mobility. Because I have data sitting in various locations through my enterprise, how do I aggregate that data to really drive meaningful insights out of that data to drive better business outcomes? And at Blue Shield of California, one of our key initiatives is what we call an experienced cube. What does that mean? It means how do I drive transparency of data between providers, members and payers? So that not only do I reduce overhead on providers and provide them a better experience, or hospital systems or doctors, but ultimately, how do we have the member have it their power of their fingertips the value of their data holistically, so that we're making better decisions about their health care? One of the things Teresa was talking about was the use of this data, and I would drive to data democratization. We got to put the power of data into the hands of everyone, not just data scientists. Yes, we need those data scientists to help us build AI models to really drive and tackle these tougher challenges and business problems that we may have in our environments. But everybody in the company, both on the IT side, both on the business side, really need to understand of how do we become a data insights driven enterprise. Put the power of the data into everyone's hands so that we can accelerate capabilities, right? And leverage that data to ultimately drive better business results. So as a leader, as a technology leader, part of our responsibility, our leadership is to help our companies do that. And that's really one of the exciting things that I'm doing in my role now at Blue Shield of California. >> Yeah, it's really, really exciting time. I want to shift gears a little bit and focus on women in tech. So I think in the past five to 10 years, there has been a lot of headway in this space. But the truth is women are still underrepresented in the tech space. So what can we do to attract more women into technology quite honestly. So Nishita, curious, what your thoughts are on that? >> Great question. And I am so passionate about this for a lot of reasons, not the least of which is I have two daughters of my own. And I know how important it is for women and young girls to actually start early in their love for technology, and data and all things digital, right? So I think it's one very important to start early, start an early education, building confidence of young girls that they can do this, showing them role models. We at Deloitte just partnered with Ella the Engineer to actually make comic books centered around young girls and boys in the early elementary age to talk about how heroes and tech solve everyday problems. And so really helping to get people's minds around tech is not just in the back office coding on a computer, tech is about solving problems together that help us as citizens, as customers, right? And as humanity. So I think that's important. I also think we have to expand that definition of tech, as we just said. It's not just about, right? Database design. It's not just about Java and Python coding, it's about design. It's about the human machine interfaces. It's about how do you use it to solve real problems and getting people to think in that kind of mindset makes it more attractive and exciting. And lastly, I'd say look, we have absolute imperative to get a diverse population of people, not just women, but minorities, those with other types of backgrounds, disabilities, etc involved. Because this data is being used to drive decision making, and if we are not all involved, right? In how that data makes decisions, it can lead to unnatural biases that no one intended but can happen just 'cause we haven't involved a diverse enough group of people around it. >> Absolutely. Lisa, curious about your thoughts on this. >> I agree with everything Nishita said. I've been passionate about this area, I think it starts with first we need more role models. We need more role models as women in these leadership roles throughout various sectors. And it really is it starts with us and helping to pull other women forward. So I think certainly, it's part of my responsibility, I think all of us as female executives that if you have a seat at the table to leverage that seat at the table to drive change, to bring more women forward, more diversity forward into the boardroom and into our executive suites. I also want to touch on a point Nishita made about women, we're the largest consumer group in the company yet we're consumers, but we're not builders. This is why it's so important that we start changing that perception of what tech is. And I agree that it starts with our young girls. We know the data shows that we lose our young girls by middle school. Very heavy peer pressure, it's not so cool to be smart, or do robotics, or be good at math and science. We start losing our girls in middle school. So they're not prepared when they go to high school and they're not taking those classes in order to major in the STEM fields in college. So we have to start the pipeline early with our girls. And then I also think it's a measure of what your boards are doing. What is the executive leadership and your goals around diversity and inclusion? How do we invite more diverse population to the decision making table? So it's really a combination of efforts. One of the things that certainly is concerning to me is during this pandemic, I think we're losing one in four women in the workforce now, because of all the demands that our families are having to navigate through this pandemic. The last statistic I saw in the last four months is we've lost 850,000 women in the workforce. This pipeline is critical to making that change in these leadership positions. >> Yeah, it's really a critical time. And now we're coming to the end of this conversation, I want to ask you Teresa, what would be a call to action to everyone listening, both men and women since its needs to be solved by everyone, to address the gender gap in the industry? >> I'd encourage each of you to become an active sponsor. Research shows that women and minorities are less likely to be sponsored than white men. Sponsorship is a much more active form than mentorship. Sponsorship involves helping someone identify career opportunities and actively advocating for them in those roles, opening your network, giving very candid feedback. And we need men to participate too. There are not enough women in tech to pull forward and sponsor the high potential women that are in our pipelines. And so we need you to be part of the solution. >> Nishita real quickly, what would be your call to action to everyone? >> I'd say look around your teams, see who's on them and make deliberate decisions about diversifying those teams. As positions open up, make sure that you have a diverse set of candidates, and make sure that there are women that are part of that team. And make sure that you are actually hiring and putting people into positions based on potential not just experience. >> And real quickly Lisa, will close it out with you, what would your call to action be? >> Well, it's hard to... What Nishita and what Teresa shared I think were very powerful actions. I think it starts with us. Taking action at our own table, making sure you're driving diverse panels and hiring, setting goals for the company. Having your board engaged and holding us accountable and driving to those goals, will help us all see a better outcome but with more women at the executive table and diverse populations. >> Great advice and great action for all of us to take. Thank you all so much for spending time with me today and talking about this really important issue. I really appreciate it. Stay with us.

>>from around the globe. It's the Cube covering HP. Discover Virtual experience Brought to you by HP. >>Welcome to the Cube's coverage of HP Discover 2020 Virtual Experience. I'm Lisa Martin, and I'm pleased to be joined by two guests from HP longtime partner Tech Data. We have calling Blair the vice president of sales and marketing of I. O. T. And Data Solutions and David Smith, H P E Pre Sales Field Solutions are common. And David, Welcome to the Cube. Thanks, Lisa. Great to see. So let's start with the calling. HP and Technical have been partners for over 40 years, but tell our audience a little bit about tech data before we get into the specifics of what you're doing and some of the cool I o. T. Stuff with HP. I >>think that the Tech data is a Fortune 100 distributor. We continued to evolved to be a solutions aggregator in these next generation technology businesses. As you've mentioned, we've been serving the I T distribution markets globally for for 40 plus years, and we're now moving into next generation technologies like Wild Analytics, I O. T and Security bubble Lifecycle Management services. But to be able todo position ourselves with our customer base and the needs of their clients have. So I'm excited to be here today to talk a little bit about what we're doing in I, O. T. And Analytics with David on the HPC side >>and in addition to the 40 plus years of partnership calling that you mentioned that Detected and HP have you've got over 200 plus hp. Resource is David, you're one of those guys in the field. Talk to us about some of the things that you're working on with Channel Partners Table David to enable them, especially during such crazy times of living and now >>absolutely, absolutely so. What we can do is we can provide strong sales and technical enablement if your team, for example, wants to better understand how to position HP portfolio if they require assistance and architect ing a secure performance i o t. Solution. We can help ensure that you're technical team is fully capable of having that conversation, and it's one that they're able to have of confidence, weaken validate the proposed HP solutions with the customers, technical requirements and proposed use case. We can even exist on a customer calls, if it would, would benefit our partner to kind of extend out to that. We also have a a a deep technical bench that Colin can speak to in the OT space toe lean on as well. For so solution is that kind of span into the space beyond where HP typically operates, which would be edge, compute computing and network. Sic security. >>Excellent call and tell me a little bit about Tech Data's investments in I o. T. When did this start? What are you guys doing today? >>Sure, we started in the cloud space. First tackle this opportunity in data center modernization and hybrid cloud. That was about seven years ago. Shortly thereafter we started investing very materially in the security cyber security space. And then we follow that with Data Analytics and then the Internet of things. Now we've been in those spaces with our long term partners for some time. But now that we're seeing this movement to the intelligent edge and a real focus on business outcomes and specialization, we've kind of tracked with the market, and we feel like we've invested a little bit ahead of where the channel is in terms of supporting our ecosystem of partners in this space. >>So the intelligent edge has been growing for quite some time. Poland in the very unique times that we're living in in 2020 how are you seeing that intelligent edge expand even more? And what are some of the pressing opportunities that tech data and HPC i O T solutions together can address? >>So a couple. So the first is a Xai mentioned earlier just data center modernization. And so, in the middle of code 19 and perhaps postcode 19 we're going to see a lot of clients that are really focused on monetizing the things that they've got. But doing so to drive business outcomes. We believe that increasingly, the predominance of use cases and compute and analytics is going to move to the edge. And HP has got a great portfolio for not just on premise high performance computing but also hybrid cloud computing. And then when we get into the edge with edge line and networking with Aruba and devices that need to be a digitized and sense arised, it's a really great partnership. And then what we're able to do also, Lisa, is we've been investing in vertical markets since 2000 and seven, and I've been a long the ride with that team, most all of that way. So we've got deep specialization and healthcare and industrial manufacturing, retail and then public sector. And then the last thing we've kind of turned on here recently just last month is a strategic partnership in the smarter cities space. So we're able to leverage a lot of those vertical market capabilities. Couple that with our HP organization and really drive specialized repeatable solutions in these vertical markets, where we believe increasingly, customers are going to be more interested in a repeatable solutions that can drive quick proof of value proof of concepts with minimal viable what kinds of products. And that's that's kind of the apartment today with RHB Organization and the HP Corporation >>David. Let's double click into some of those of vertical markets that Colin mentioned some of the things that pop into minor healthcare manufacturing. As we know, supply chains have been very challenged during covered. Give us an insight into what you're hearing from channel partners now virtually, but what are some of the things that are pressing importance? >>So from a pressing and important to Collins exact point, and your exact point as well is really it's all about the edge computing space now from a product perspective Azaz Colin had mentioned earlier. HP has their edge line converged systems, which is kind of taking the functionality of OT and edge T Excuse me of OT and I t and combine it into a single edge processing compute solution. You kind of couple that with the ability to configure components such as Tesla GP, use in specific excellent offerings to offer an aid and things like realtime, video processing and analytics. Uh, and a perfect example of this is, ah so for dissing and covert space. If if I need to be able to analyze a group of people to ensure they're staying as far apart as possible or, you know within self distant guidelines, that is where kind of the real time that's like an aspect of things can be taken advantage of same things with with the leveraging cameras where you could actually take temperature detection as as well, so it's really kind of best to think of Edge Lines Solutions is data center computing at the edge kind of transition into the Aruba space. Uh Rubio says offerings aid in the island Security is such a clear pass device inside, which allows for device discovery of network and monitoring of wired and wireless devices. There's also Aruba asset tracking and real time location of solutions, and that's particularly important in the healthcare space as well. If I have a lot of high value assets, things like wheelchairs, things like ventilation devices, where these things low located within my facilities and how can I keep keep track of them? They also, and by that I mean HP. They also kind of leveraging expanse ecosystem of partners. As an example, they leverage thing works allow their i o t solutions as well, when you kind of tying it all together with HP Point. Next to the end, customers provided with comprehensive loyalty solution. >>So, Colin, how ready? Our channel partners and the end user customers to rapidly pivot and start either deploying more technologies at the edge to be able to deliver some of the capabilities that David talked about in terms of analytics and sensors for social distancing. How ready are the channel partners and customers to be able to understand, adopt and execute this technology. >>So I think on the understanding side, I think the partners are there. We've been talking about digital transformation in the channel for a couple of years now, and I think what's happened through the 19 Pandemic is that it's been a real spotlight on the need for those business outcomes to to solve for very specific problems. And that's one of the values that we serve in the channel. So we've got a solution offering that we call our solution factory. And what we do really says is we leverage a process to look outside the industry. At Gartner, Magic Quadrant Solutions forced a Wave G two crowd. You know, top leaders, visionaries and understand What are those solutions that are in demand in these vertical markets that we talked about? And then we do a lot of work with David and his team internally in the HP organization to be able to do that and then build out that reference architectures so that we know that there's a solution that drives a bill of materials and a reference architecture that's going to work that clients are going to need and then we can do it quickly. You know, Tech data. Everything's about being bold, acting now getting scale. And we've got a large ecosystem partners that already have great relationships. So we pride ourselves on being able to identify what are those solutions that we can take to our partners that they can quickly take to their end users where you know we've We've kind of developed out what we think the 70 or 80% of that solution is going to look like. And then we drive point next and other services capabilities to be able to complete that last mile, if you will, of some of the customization. So we're helping them. For those who aren't ready, we're helping them. For those who already have very specific use cases and a practice that they drive with repeatable solutions were coming alongside them and understanding. What can we do? Using a practice builder approach, which is our consultative approach to understand where our partners are going in the market, who their clients are, what skill sets do they have? What supplier affinities do they want to drive? What brand marketing or demand generation support do they need? And that's where we can take some of these solutions, bring them to bear and engage in that consultative engagement to accelerate being ready as, as you rightly say, >>so tech. It has a lot of partners. You in general. You also have a lot of partners in the i o T space calling What? How do you from a marketing hat perspective? How do you describe the differentiation that Tech data and HP ease Iot solutions delivered to the channel to the end user? >>A couple of different things? I think that's that's differentiation. And that's one of the things that we strive for in the channel is to be specialized and to be competitively differentiated. And so the first part, I say to all of my team, Lisa, is you know, whether it's our solution consultants or our technical consultants, our solutions to the developers or the software development team that works my organization. Our goal is to be specialized in such a way that we're having relevant value added conversations not only our channel partners, but also end users of our partners want to bring us into those conversations, and many do. The next is really education and enablement as you would expect. And so there's a lot of things that are specialized in our technical. We drive education certification programs, roadshows, seminars, one of the things that we're seeing a lot of interest now. Lisa is for a digital marketing, and we're driving. Some really need offerings around digital marketing platforms that not only educate our partners but also allow our partners to bring their end users and tour some of this some of these technologies. So whether it's at our Clearwater office, where we've got an I. O T. Solution center, that we we take our partners and their clients through or we're using our facilities Teoh to do executive briefings and ideation as a service that, you know, kind of understanding the art of the possible. With both our resellers and their clients work, we're using our solution. Our solution catalogs that we've built an interactive pdf that allows our partners to understand over 50 solutions that we've got and then be able to identify. Where would they like to bring in David and his team and then my consultants to do that, that deep planning on business development, uh, that we talked about a little bit earlier. >>So the engagement right now is maybe even more important than it has been in a while because it's all hands off and virtual David. Talk to me about some of the engagement and the enablement piece that call and talked about. How are you able to really keep a channel partner and their end user customers engaged and interested in what you're able to deliver through this from New Virtual World? >>That's a great, great question. And we work in conjunction with our marketing teams to make sure that as new technologies and quite in I O. T space as well as within the HP East base as well that that our channel partners are educated and aware that these solutions exist. I know for a fact that for the majority of them you kind of get this consistent bombardment of new technology. But being able to actually have someone go out and explain it and then being able to correspondingly position it's use case and it's functionality and why it would provide value for your end customer is one of the benefits of tech data ads to kind of build upon that previous statement. The fact that We have such a huge portfolio of partners, so you kind of have HP and the edge compute space. But we have so many different partners in the OT space where it's really just a phone call, an email, a Skype message, a way to have that conversation around interoperability and then provide those responses back to our partners. >>Excellent. One more question before we go. Colin for you, A lot of partners. Why HP fry Mt. >>So a couple of reasons? One of the one of the biggest reasons as HP is just a great partner. And so when you look at evaluating I. O. T solutions that tend to be pretty comprehensive in many cases, Lisa it takes 10 or 12 partners to complete a really i o t solution and address that use case that that's in the field. And so when you have a partner like HP who's investing in these programs, investing in demand generation, investing in the spectrum of technology, whether it's hybrid Cloud Data Center, compute storage or your edge devices and Iot gateways, then to be able to contextualize those into what we call market ready solutions in each one of these vertical markets where there's references and there's use cases. And there were coupling education that specific rest of solutions. You know HP can do all of those things, and that's very important. Because in this new world, no one can go it alone anymore. It takes it takes partnerships, and we're all better together. And HP really does embrace that philosophy. And they've been a great partner for us in the Iot space. >>Excellent. Well, Colin and David, thank you so much for joining me today on the Cube Tech data. H p e i o t better together. Thank you so much. It's been a pleasure talking with you. >>Thank you. >>Thank you. Lisa. >>And four Collet and David. I am Lisa Martin. You're watching the Cube's virtual coverage of HP Discover 2020. Thanks for watching. Yeah, yeah, yeah, yeah.

>> Narrator: From theCUBE Studios and Palo Alto in Boston. Connecting with thought leaders all around the world. This is a CUBE conversation. >> Hey, welcome back everybody. Jeff Frick here with theCUBE. We're at our Palo Alto studio as kind of our on going leadership coverage of what's happening with the COVID crisis, and really looking out into our community to find experts who can provide tips and tricks, and some guidance as everyone is kind of charting these uncharted waters if you will. And we've got a great cube alarm in our database. He's a fantastic resourcer. We're excited to get him on. Share the information with you. We'd like to welcome once again, Darren Murph. He is the Head of Remote for GitLab. Darren, great to see you. >> Absolutely, great to be here. Thanks for having me. >> Absolutely, so thank you and. First off, we had you on earlier this year, back when things were normal, in kind of a regular review. Who knew that you would be at the center of the work-from-home universe just a few short months later. I mean, you've been doing this for ever. So it's kind of a wile old veteran of the work-from-home, or not even from home, just work from some place else. What are some top level things that you can share for people that have never experienced this before? >> Yeah, on the working front. If you're one of the people that are working from home, I think there's a couple of things you can do to help acclimate, make your world a little bit better. The first is to try to create some sort of separation between your work life and your personal life. Now if you have a home big enough that you can dedicate a workspace to being your office, that's going to help a lot. Help from a focus standpoint and just. You don't want those lines between work and life to blur too much, that's where isolation kicks in. That's where burnout kicks in. You want to do whatever you can to avoid. You got to remember, when you're not physically walking out of a office and disconnecting from work. You have to replicate that and recreate that. I actually recommend for people that used to have a commute and now they don't. I would actually black something in your calendar, whether that's cooking, cleaning, spending time with your family. Resting more, anything so that you ramp into your day very deliberately and ramp out of very deliberately. Now on the team leading front. I'm going to say it may feel a little counter intuitive, but the further your team is from you, the more distributed they are. The more you really need to let go and allow them to have mechanism for feeding back to you. Managers job in a remote setting switches from just being a pure director, you're actually being an unblocker. A really active listener. And for people who have gotten to a certain point in their career through command and control, this is going to feel very strange, jarring and counter intuitive, but we've seen it time and time again. You need to trust that your workers are in a new environment. You have to give them a mechanism feeding back to you to help them unblock whatever it is. >> You know that's funny, we had someone on as part of this the other day, talking about leaders need to change their objectives that they're managing to, from kind of activity based, to deliverals based. And it actually floored me that someone is still writing in a blog in 2020, that people have to change their management deliverables from activity to deliverables. And it was so funny, you know, you had Martin Mikos on, we had him on too. My favorite comment was, "It's so easy to fake it in the office and look busy, "but when you're at home all you have is your deliverable." so it really, it seems like there's kind of a forcing function to get people to pay attention to the things they should be managing to anyway. >> You said it, forcing functions. I talk about this all the time, but there are so many forcing functions in remote that help you do remote well. But not only just do remote well, just run your business well. Even if you plan on going back to office. On some level there's a lot of things you can do now to help pave the infrastructure to creating a better and more effective team. And as a manager, if you have it in a writing down. The metrics or expectations for your direct reports in the office, now's the time to do it. Subjectivity is allowed to flourish in the office. You can praise or promote people just kind of how much you like them or how easy they are to work with. That really has nothing to do with metrics and results. I've often been asked, "How do you know if "someone's been working remotely?" And my response is, how do you know if they were working in the office. If you can't clearly answer that in the office, then you're not going to be able to answer it remotely. So frankly, in these times a lot of the burden falls more on the manager to actually take a hard look at what they're clarifying to their team. And if the metrics aren't laid out. It's on the manager to lay that out. It's not the responsibility of the direct report to figure out how to prove their worth. The manager has to be very articulant about what that value looks like. >> Right, and not only do they have to be articulate about what the deliverables are and what their expectations are, but. You guys have a remote play book GitLab has published, which is terrific. People should go online, it's 38 pages of dense, dense, dense material. It's a terrific resource, it's a open source, you got to love the open source, eat those. But one of the slides that jumped out to me, and it's consistent with a lot of these conversations that we're having, is that your frequency of communications when people are not in the same room together. Has to go up dramatically, which is a little counter intuitive, but what I found even more interesting was the variety of types of communication. Not just you kind of standard meeting, or you standard status on a project, or maybe a little bit of a look forward to some strategic stuff. But you outlined a whole variety of types of communication. Objectives or methods, or feel if you will, to help people stay connected and to help kind keep this team building going forward. >> So here's the thing about communication. You've got to be intentional about it in a remote setting. And in fact, you need to have more intentionality across the board in a remote setting. And communications is just a very obvious. So for a lot of companies, they leave a lot of things to spontaneity Inter-personal relationships and communications are two of the biggest ones. Where you may not actually lay out a plan for how work is communicated about, or what opportunities you give people to chat about their weekends, or sports, or anything like that. You just kind of put them in the same building and then people just kind of figure it out. In a remote setting that's unwise. You're going to get a lot of chaos and disfunction when people don't know how to communicate and on what channel. So at GitLab we're very prescriptive that work communication happens in a GitLab issue or a merge request. And then informal communication happens through Zoom calls or Slack. We actually expired our Slack messages after 90 days, specifically to force people not to do work in Slack. We want the work to begin where it needs to end up, and in that case it's a very, it's a tool, GitLab, that's built for asynchronous communication. We want to continue to encourage that bias towards asynchronous communication. So yeah, we write down everything about how we want people to communicate and through what channels. And that may sound like a lot of rules, but actually it's very much appreciated by our global team. We have over 1200 people, in more that 65 countries. And they all just need to know where communication is going to happen. And our team is really cohesive and on the same page because we're articulant about that. >> So I want to double down on that. On 'A secret is peace', 'cause you brought this up, or you and Stu brought it up in your conversation with Stu, and Stu raised an interesting point, right. Unfortunately in the day of email and connected phones, and this and that, there has grown an expectation that used to be business, okay was, "I'll get back to you within 24 hours "if you leave me a voicemail." And lord knows what it was when we were still typing letters and memos, and sticking stuff in the yellow envelop with the string, right, as multiple days. But somehow that all got changed to, "I need to hear back form you now." And often it feels like, if your trying to have just some uninterrupted work time, to get something done. It's like, why is your lack of planning suddenly my emergency. And you talked about, you can't operate that on a global, asynchronous team because everyone's in different timezones. And just by rule, there are going to be a lot of people that are not awake when you need the answer to that question. But that you've developed a culture that that's okay, and that that is kind of the flow and the pacing which A, forces people to ask in advance, not immediately when you need it. But also gives people unfettered time to actually plan to do work versus plan to answer communications. I wonder if you can dig into how did that evolve and how do you enforce that when somebody comes in from the outside world. >> The real key to that is something that might not be immediately apparent to everyone. Which is, at GitLab we try to shift as much burden as we possibly can humans to documentation. And this even starts at onboarding, where to get onboarded at GitLab, you get an onboarding issue within GitLab, with over 200 check boxes of things to read and knowledge assessments to take. And humans are a part of it, but very minimal compared to what most companies would do. And the thing that you just outlined was, we're talking about asking questions. Or tapping someone on the shoulder to fill in a knowledge gap. But at GitLab we want to write everything down in a very formalized structured way. We try to work handbook first. So we need to document all of our processes, protocols and solutions. Basically everything that we've ever seen or done, needs to be documented in the handbook. So it's not that GitLab team members just magically need less information, it's just that instead of having to ask someone on our team, we go ask the handbook. We go consult the documentation. And the more rich that your documentation is, the less you have to bother other people, and the less you need to rely on synchronicity. So for us it all starts with operating handbook first. That allows our humans to reserve their cycles for doing truly creative things, not just answering your question for the thousandth time. >> Right, another thing you covered, which I really enjoyed was getting senior executives to work from home for an extended period of time. Now obviously, before COVID that would probably be a lot harder to do. Well now COVID has forced that. And I think to your point about that is, it really forces the empathy for someone who had no interest in working from home. Didn't like to work from home. Loves going to the office, has their routine. Been doing it for decades, to kind of wake up to A, you need to have more empathy for what this is all about. And B, what's it all about by actually doing it. So I wonder, kind of your take in the movement to more of a work from anywhere future. Now that all the senior executives have been thrown into this work from home situation. >> Look Jeff, you never want to waste a crisis. We can't wish away the crisis that's in front of us, but we can choose how we respond to it. And this does present an opportunity to lay ground work, to lay infrastructure, to build a more remote organization. And I have absolutely advocated for companies to get their leadership teams out of the office for a meaningful amount of time. A month, ideally a quarter. So that they actually understand what the remote life is. They actually have some of those communication gaps and challenges so they can document what's happening. And then help fix it. But to your point, executives love going to the office because they're on a different playing field to begin with. They usually have an executive assistant. Things are just. There's less friction in general. So it behooves them to just kind of keep charging in that direction, but now what we have is a situation where all of those executives are remote. And I'm seeing a lot of them say, "You know what, I'm seeing the myths that I've perpetrated "break down in front of me." And this is even in the most suboptimal time ever to go remote. This isn't remote work, this is crisis induced work from home. We're all dealing with social isolation. Our parents are also doubling as homeschool teachers. We have a lot going on. And even on top of all of that, I'm amazed at how adaptable the human society has been. In just adjusting to this and figuring it out on the fly. And I think the companies that take this opportunity, to ask themselves the right question, and build this into their ongoing talent and operational strategy, will actually come out stronger on the other side. >> Yeah, as you said. This is as challenging as it's ever been. There was no planning ahead, you're spouse or significant other's also working from home. And has the same Zoom schedule as you do, for some strange reason, right. The kids are home as you said, and your homeschooling them. And they also have to get on Zoom to do their classes. So it's really suboptimal. But as you said, it's a forcing function and people are going to learn. One of the other things in your handbook is the kind of definitions. It's not just work from home or work at the office, but there's actually a continuum and a spectrum. And as people are doing this for weeks and months. And behaviors turn into habits. People are not going to want to go back to sitting on 101 for two hours every morning to go work on a laptop in the office. It just doesn't make sense. So as you kind of look forward. How do you see the evolution. How are people taking baby steps, if you will. To incorporate more of this learning as we go forward. And incorporate into more of their regular, everyday procedures. >> I'm really optimistic about the future because what I see happening here is people are unlocking their imaginations. So once they've kind of stabilized, they're starting to realize, "Hey, I'm getting a lot more time with my family. "I'm spending a lot less on gas. "I just feel better as a person because I don't show up "to work everyday with road rage. "So how can I keep this going." And I genuinely think what's going to happen in four or five months, we're going to have millions of people collectively look at each other and they say, "The boss just called me back into the office "but I just did my job from home. "Even in suboptimal conditions. "I saw my family more, I exercised more. "I had more time to cook and clean. "How about no, I'm not going to go back to the office "as my default location." And I think what's going to happen is the 80, 20 rule is going to flip. Right now people work from home only for a special occasion, like the cable company's coming or something like that. Going forward it think the offices are going to be the special occasion. You're only going to commute to the office, or fly to the office when you have a large contingent of people coming in and you need to wine and dine them, or something like that. And the second order of this is, people that are only living in expensive cities because of their location. When their lease comes up for renewal, they're going to cast a glance at places like Wyoming and Idaho, and Ohio. Maybe even Vietnam and Cambodia, or foreign places. Because now you have them thinking of, "What could life look like if I decouple geography at work. "I still want to work really hard "and contribute this knowledge. "But I can go to a place with better air quality, "better schools, better opportunity to actually "invest in a smaller community, "where I can see real impact." And I think that's just going to have massive, massive societal impacts. People are really taking this time to consider how tightly their identity has been woven into work. Now that they're home and they've become something more than just whatever the office life has defined them as. I think that's really healthy. I think a lot of people may have intertwined those two things too tightly in the past. And now it's a forcing function to really ask yourself, you aren't just your work, you're more than your work. And what can that look like when you can do that job from anywhere. >> Right, right. And as you said, there's so many kind of secondary benefits in terms of traffic and infrastructure, and the environment and all kinds of things. And the other thing I think that's interesting what you said, 80, 20 I think that was pretty generous. I wouldn't give it a 20 percent. But if people, even in this hybrid steps, do more once a week, twice a week. Once every two weeks, right. The impact on the infrastructure and peoples lives is going to be huge. But I wanted to drill on something as we go into kind of this hybrid mode at some point in time. And you talked about, and I thought it was fascinating, about the norms and really coming out from a work from home first, or a work from anywhere first. Your very good at specifying anywhere doesn't mean home. Could be the library, could be the coffee shop. Could be an office, could be a WeWork. Could be wherever. Because if you talked about the new norms and the one I thought was really interesting, which probably impacts a lot of teams, is when some of the team's in the office and some of the team isn't. The typical move, right, is to have everyone in the office go into the conference room. We sit around one big screen. So you get like five people sitting around one table and you got a bunch of heads on Zoom. And you said, "You know, no. "Let's all be remote. So if we just be happen to be sitting at our desk. If we happen to be in the office, that's okay. But really normalize. And like we saw the movement from Cloud got to Cloud-e to Cloud first, why not Cloud. And then you know, kind of mobile and does it work in a mobile. No, no, no it has to. It's mobile first. Really the shift to not, can it be done at home, but tell me why it shouldn't be done at home, a really different kind of opening position as to how people deploy resources and think about staffing and assigning teams. It's like turning the whole thing upside down. >> Completely upside down. I think remote first to your point, is going to be the default going forward. I think we're just one or two quarters away from major CEO's sitting on the hot seat on CNBC, when it's their turn for quarterly earnings. And they're going to have to justify why they're spending what they spend on real estate. Is if your spending a billion dollars a year on real estate, you could easily deploy that to more people, more R&D. Once that question is asked in mass, that is when you're going to see the next phase of this. Where you really have to justify, even from a cost stand point, why are you spending so much? Why are you tying so much of your business results to geography. The thing about remote first is that it's not a us versus them. A lot of what we've learned at GitLab, and how we operate so efficiently. They work really well for remote teams, and they are remote first. But they would work just as well in an office. We attach a Google doc agenda to every single business meeting that we have, so that there's always an artifact. There's always a documented thread on what happened in a meeting. Now this would work just as well in a co-located meeting. Who wouldn't want to have a meeting where it's not just in one ear and out the other. You're going to give the time to the meeting, you might as well get something out of it. And so a lot of these remote forced. Remote first forcing functions, they do help remote teams work well. But I think it's especially important for hybrid teams. Offices aren't going to vanish overnight. A lot of these companies are going to have some part of their company return to the office, when travel restrictions are lifted. It think the key here is that its going to be a lot more fluent. You're never going to know on a day to day basis, who is coming into the office, and who is not. So you need to optimize for everyone being out of the office. And if they just so happen to be there, they just so happen to be there. >> Right, right. So before we. I want to get into one little nitty gritty subject, in terms of investment into the home office. You know, we're doing 100% remote interviews now on theCUBE, we used to go to pretty much. Probably 80% of our business was at events, or at peoples offices, or facilities. Now it's all dial-in. You talked a lot about people need to flex a little bit on enabling people to invest in the little bits and pieces of infrastructure for their home office, that they just don't have the same set ups. You're talking about multiple monitors, a comfortable chair, a good light. That there's a few things you can invest in, not tremendous amounts of money. But a couple of hundred bucks here and there, to make a big difference on the home work environment. And how people should think about making that investment into a big monitor that they don't see. It's not sitting at the desk in the office. >> 100%, look if you're coming from a co-located space, you're probably sitting in a cube that costs five, 10, maybe 20 thousand dollars put together. You might not notice that, but it's not cheap to build cubicles in a high rise. And if you go to your home and you have nothing set up, I would say it's on the people group to think really hard about being more lax and more lenient about spending policy. People need multiple monitors. You need a decent webcam, you need a decent microphone. You need a chair that isn't going to kill your back. You want to help people create healthy ergonomics. Sustainable workspaces in their home. This is the kind of thing that will inevitably impact productivity. You force someone to just be hunched over on their couch, in front of a 13 inch laptop. I mean, what kind of productivity do you really expect from that. That's not a great long term solution. I think the people group actually has a higher burden to bare all the way around. You know when it comes to making sure teams feel like teams and they have the atmosphere to connect on a meaningful level. It comes down to the people group, to not letting that just go to spontaneity. You want to have a happy hour virtually, you're going to have to put a calendar invite on peoples calendar. You're going to create topical channels in Slack for people to talk about things other than work. Someone's going to have to do that. They don't just happen by default. So, from hardware all the way to communication. The people group really needs to use this opportunity to think about, "Okay, what can we unlock in this new world." >> Right, I'm glad you said the people group and not the resources group because they're not coal, or steel, or a factory. >> No, if anything COVID has humanized this in a way, and I think it's actually a really big silver lining, where we're all now peering into each others homes. And it is glaringly obvious, that we're all humans first, colleagues second. And of course that always been the case, but there's something about a sterile or co-located work environment. You check a piece of you at the door. And you just kind of get down to business. Why is that, we have technology at out fingertips. We can be humans with each other. And that going to actually encourage more empathy. As we've seen at GitLab, more empathy leads to better business results. It leads to more meaningful connections. I mean, I have people, friends, located all over the world that I feel like I have a closer bond with. A closer, more intimate connection with that a lot of people I've met in office. To some degree you don't know who they really are. You don't know what they really love and what makes them tick. >> Right, right. All right Darren, so before I let you go and again thank you for the time, the conversation. I'm sure everyone is calling you up and I just love the open source ETHOS and the sharing. It's made such a huge impact on the technology world and second order impacts that a lot of people take advantage. Again, give us the place that people can go for the playbook, so they can come and leverage some of the resources. And again, thank you guys for publishing 'em. >> Absolutely, so we're an open source. We try to open source all of our learnings on remote. So go to allremote.info that will redirect you right into the All Remote section of GitLab handbook. All of which is open source. Right at the top you can download the remote playbook, which is PDF that we talked about. Download that, it takes you through all of our best information on getting started and thriving as a remote team. Just under that there's a lot of comprehensive guides on how we think about everything. And how we operate synchronously. How we handle meeting, and even hiring and compensation. allremote.info and of course you're welcome to reach out to me on Twitter, I'm @darrenmurph. >> All right, well thanks a lot Darren. And I find it somewhat ironic that you have a jetliner over your shoulder. Waiting for the lockdown and the quarantine to end so you can get back on the airplane. And we're looking forward to that day. >> Can't wait man, I miss, I miss the airplanes. I told someone the other day, I never thought I'd say I miss having a middle seat at the very back of the airplane, with someone reclined into my nose. But honestly, I can't wait. Take me anywhere. >> I think you'll be fighting people for that seat in another month or so. All right, thanks a lot, Darren. >> Absolutely, take care all. >> All right, he's Darren. I'm Jeff, you're watching theCUBE, from our Palo Altos Studios. Thanks for watching, we'll see you next time. (upbeat music)

>> Jeff: Hello everybody and thank you for joining us today for the Virtual VERTICA BDC 2020. Today's breakout session has been titled, "VERTICA Database Designer Today and Tomorrow." I'm Jeff Healey, Product VERTICA Marketing, I'll be your host for this breakout session. Joining me today is Yuanzhe Bei, Senior Technical Manager from VERTICA Engineering. But before we begin, (clearing throat) I encourage you to submit questions or comments during the virtual session. You don't have to wait, just type your question or comment in the question box below the slides and click Submit. As always, there will be a Q&A session at the end of the presentation. We'll answer as many questions, as we're able to during that time, any questions we don't address, we'll do our best to answer them offline. Alternatively, visit VERTICA forums at forum.vertica.com to post your questions there after the session. Our engineering team is planning to join the forums, to keep the conversation going. Also, a reminder that you can maximize your screen by clicking the double arrow button at the lower right corner of the slides. And yes, this virtual session is being recorded and will be available to view on demand this week. We will send you a notification as soon as it's ready. Now let's get started. Over to you Yuanzhe. >> Yuanzhe: Thanks Jeff. Hi everyone, my name is Yuanzhe Bei, I'm a Senior Technical Manager at VERTICA Server RND Group. I run the query optimizer, catalog and the disaggregated engine team. Very glad to be here today, to talk about, the "VERTICA Database Designer Today and Tomorrow". This presentation will be organized as the following; I will first refresh some knowledge about, VERTICA fundamentals such as Tables and Projections, which will bring to the question, "What is Database Designer?" and "Why we need this tool?". Then I will take you through a deep dive, into a Database Designer or we call DBD, and see how DBD's internals works, after that I'll show you some exciting DBD improvements, we have planned for 10.0 release and lastly, I will share with you, some DBD future roadmap we planned next. As most of you should already know, VERTICA is built on a columnar architecture. That means, data is stored column wise. Here we can see a very simple example, of table with four columns, and the many of you may also know, table in VERTICA is a virtual concept. It's just a logical representation of data, which means user can write SQL query, to reference the table names and column, just like other relational database management system, but the actual physical storage of data, is called Projection. A Projection can reference a subset, or all of the columns all to its anchor table, and must be sorted by at least one column. Each table need at least one C for projection which reference all the columns to the table. If you load data to a table with no projection, and automated, auto production will be created, which will be arbitrarily assorted by, the first couple of columns in the table. As you can imagine, even though such other production, can be used to answer any query, the performance is not optimized in most cases. A common practice in VERTICA, is to create multiple projections, contain difference step of column, and sorted in different ways on the same table. When query is sent to the server, the optimizer will pick the projection, that can answer the query in the most efficient way. For example, here you can say, let's say you have a query, that select columns B, D, C and sorted by B and D, the third projection will be ideal, because the data is already sorted, so you can save the sorting costs while executing the query. Basically when you choose the design of the projection, you need to consider four things. First and foremost, of course the sort order. The data already sorted in the right way, can benefit quite a lot of the query actually, like Ordered by, Group By, Analytics, Merge, Join, Predicates and so on. The select column group is also important, because the projection must contain, all the columns referenced by your workflow query. Even missing one column in the projection, this projection cannot be used for a particular query. In addition, VERTICA is the distributed database, and allow projection to be segmented, based on the hash of a set of columns, which is beneficial if the segmentation merged, the join keys or group keys. And finally encoding of each per columns is also part of the design, because the data is sorted in different way, may completely change the optimal encoding for each column. This example only show the benefit of the first two, but you can imagine the rest too are also important. But even for that, it doesn't sound that hard, right? Well I hope you change your mind already when you see this, at least I do. These machine generated queries, really beats me. It will probably take an experienced DBA hours, to figure out which projection can be benefit these queries, not even mentioning there could be hundreds of such queries, in the regular work logs in the real world. So what can we do? That's why we need DBD. DBD is a tool integrated in the VERTICA server, that it can help DBA to perform an access, on their work log query, tabled schema and data, and then automatically figure out, the most optimized projection design for their workload. In addition, DBD also a sophisticated tool, that can take customize by a user, by sending a lot of parameters objectives and so on. And lastly, DBD has access to the optimizer, so DB knows what kind of attribute, the projection need to have, in order to have the optimizer to benefit from them. DBD has been there for years, and I'm sure there are plenty of materials available online, to show you how DBD can be used in different scenarios, whether to achieve the query optimize, or load optimize, whether it's the comprehensive design, or the incremental design, whether it's a dumping deployment script, and manual deployment later, or let the DBD do the order deployment for you, and the many other options. I'm not planning to talk about this today, instead, I will take the opportunity today, to open this black box DBD, and show you what exactly hide inside. DBD is a complex tool and I have tried my best to summarize the DBD design process into seven steps; Extract, Permute, Prune, Build, Score, Identify and Encode. What do they mean? Don't worry, I will show you step by step. The first step is Extract. Extract Interesting Columns. In this step, DBD pass the design queries, and figure out the operations that can be benefited, by the potential projection design, and extract the corresponding columns, as interesting columns. So Predicates, Group By, Order By, Joint Condition, and analytics are all interesting Column to the DBD. As you can see this three simple sample queries, DBD can extract the interest in column sets on the right. Some of these column sets are unordered. For example, the green one for Group By a1 and b1, the DBD extracts the interesting column set, and put them in the own orders set, because either data sorted by a1 first or b1 first, can benefit from this Group By operation. Some of the other sets are ordered, and the best example is here, order by clause a2 and b2, and obviously you cannot sort it by b2 and then a2. These interesting columns set will be used as if, to extend to actual projection sort order candidates. The next step is Permute, once DBD extract all the C's, it will enumerate sort order using C, and how does DBD do that? I'm starting with a very simple example. So here you can see DBD can enumerate two sort orders, by extending d1 with the unordered set a1, b1, and the derived at two sort order candidates, d1, a1, b1, and d1, b1, a1. This sort order can benefit queries with predicate on d1, and also benefit queries by Group By a1, b1, when a1, sorry when d1 is constant. So with the same idea, DBD will try to extend other States with each other, and populate more sort order permutations. You can imagine that how many of them, there could be many of them, these candidates, based on how many queries you have in the design and that can be handled of the sort order candidates. That comes to the third step, which is Pruning. This step is to limit the candidates sort order, so that the design won't be running forever. DBD uses very simple capping mechanism. It sorts all the, sort all the candidates, are ranked by length, and only a certain number of the sort order, with longest length, will be moved forward to the next step. And now we have all the sort orders candidate, that we want to try, but whether this sort order candidate, will be actually be benefit from the optimizer, DBD need to ask the optiizer. So this step before that happens, this step has to build those projection candidate, in the catalog. So this step will build, will generates the projection DBL's, surround the sort order, and create this projection in the catalog. These projections won't be loaded with real data, because that takes a lot of time, instead, DBD will copy over the statistic, on existing projections, to this projection candidates, so that the optimizer can use them. The next step is Score. Scoring with optimizer. Now projection candidates are built in the catalog. DBD can send a work log queries to optimizer, to generate a query plan. And then optimizer will return the query plan, DBD will go through the query plan, and investigate whether, there are certain benefits being achieved. The benefits list have been growing over time, when optimizer add more optimizations. Let's say in this case because the projection candidates, can be sorted by the b1 and a1, it is eligible for Group By Pipe benefit. Each benefit has a preset score. The overall benefit score of all design queries, will be aggregated and then recorded, for each projection candidate. We are almost there. Now we have all the total benefit score, for the projection candidates, we derived on the work log queries. Now the job is easy. You can just pick the sort order with the highest score as the winner. Here we have the winner d1, b1 and a1. Sometimes you need to find more winners, because the chosen winner may only benefit a subset, of the work log query you provided to the DBD. So in order to have the rest of the queries, to be also benefit, you need more projections. So in this case, DBD will go to the next iteration, and let's say in this case find to another winner, d1, c1, to benefit the work log queries, that cannot be benefit by d1, b1 and a1. The number of iterations and thus the winner outcome, DBD really depends on the design objective that uses that. It can be load optimized, which means that only one, super projection winner will be selected, or query optimized, where DBD try to create as many projections, to cover most of the work log queries, or somewhat balance an objective in the middle. The last step is to decide encoding, for each projection columns, for the projection winners. Because the data are sorted differently, the encoding benefits, can be very different from the existing projection. So choose the right projection encoding design, will save the disk footprint a significant factor. So it's worth the effort, to find out the best thing encoding. DBD picks the encoding, based on the actual sampling the data, and measure the storage footprint. For example, in this case, the projection winner has three columns, and say each column has a few encoding options. DBD will write the sample data in the way this projection is sorted, and then you can see with different encoding, the disk footprint is different. DBD will then compare the disk footprint of each, of different options for each column, and pick the best encoding options, based on the one that has the smallest storage footprint. Nothing magical here, but it just works pretty well. And basic that how DBD internal works, of course, I think we've heard it quite a lot. For example, I didn't mention how the DBD handles segmentation, but the idea is similar to analyze the sort order. But I hope this section gave you some basic idea, about DBD for today. So now let's talk about tomorrow. And here comes the exciting part. In version 10.0, we significantly improve the DBD in many ways. In this talk I will highlight four issues in old DBD and describe how the 10.0 version new DBD, will address those issues. The first issue is that a DBD API is too complex. In most situations, what user really want is very simple. My queries were slow yesterday, with the new or different projection can help speed it up? However, to answer a simple question like this using DBD, user will be very likely to have the documentation open on the side, because they have to go through it's whole complex flow, from creating a projection, run the design, get outputs and then create a design in the end. And that's not there yet, for each step, there are several functions user need to call in order. So adding these up, user need to write the quite long script with dozens of functions, it's just too complicated, and most of you may find it annoying. They either manually tune the projection to themselves, or simply live with the performance and come back, when it gets really slow again, and of course in most situations, they never come back to use the DBD. In 10.0 VERTICA support the new simplified API, to run DBD easily. There will be just one function designer_single_run and one argument, the interval that you think, your query was slow. In this case, user complained about it yesterday. So what does this user to need to do, is just specify one day, as argument and run it. The user don't need to provide anything else, because the DBD will look up his query or history, within that time window and automatically populate design, run design and export the projection design, and the clean up, no user intervention needed. No need to have the documentation on the side and carefully write a script, and a debug, just one function call. That's it. Very simple. So that must be pretty impressive, right? So now here comes to another issue. To fully utilize this single round function, users are encouraged to run DBD on the production cluster. However, in fact, VERTICA used to not recommend, to run a design on a production cluster. One of the reasons issue, is that DBD picks massive locks, both table locks and catalog locks, which will badly interfere the running workload, on a production cluster. As of 10.0, we eliminated all the table and ten catalog locks from DBD. Yes, we eliminate 100% of them, simple improvement, clear win. The third issue, which user may not be aware of, is that DBD writes intermediate result. into real VERTICA tables, the real DBD have to do that is, DBD is the background task. So the intermediate results, some user needs to monitor it, the progress of the DBD in concurrent session. For complex design, the intermediate result can be quite massive, and as a result, many lost files will be created, and written to the disk, and we should both stress, the catalog, and that the disk can slow down the design. For ER mode, it's even worse because, the table are shared on communal storage. So writing to the regular table, means that it has to upload the data, to the communal storage, which is even more expensive and disruptive. In 10.0, we significantly restructure the intermediate results buffer, and make this shared in memory data structure. Monitoring queries will go directly look up, in memory data structure, and go through the system table, and return the results. No Intermediate Results files will be written anymore. Another expensive lubidge of local disk for DBD is encoding design, as I mentioned earlier in the deep dive, to determine which encoding works the best for the new projection design, there's no magic way, but the DBD need to actually write down, the sample data to the disk, using the different encoding options, and to find out which ones have the smallest footprint, or pick it as the best choice. These written sample data will be useless after this, and it will be wiped out right away, and you can imagine this is a huge waste of the system resource. In 10.0 we improve this process. So instead of writing, the different encoded data on the disk, and then read the file size, DBD aggregate the data block size on-the-fly. The data block will not be written to the disk, so the overall encoding and design is more efficient and non-disruptive. Of course, this is just about the start. The reason why we put a significant amount of the resource on the improving the DBD in 10.0, is because the VERTICA DBD, as essential component of the out of box performance design campaign. To simply illustrate the timeline, we are now on the second step, where we significantly reduced, the running overhead of the DBD, so that user will no longer fear, to run DBD on their production cluster. Please be noted that as of 10.0, we haven't really started changing, how DBD design algorithm works, so that what we have discussed in the deep dive today, still holds. For the next phase of DBD, we will briefly make the design process smarter, and this will include better enumeration mechanism, so that the pruning is more intelligence rather than brutal, then that will result in better design quality, and also faster design. The longer term is to make DBD to achieve the automation. What entail automation and what I really mean is that, instead of having user to decide when to use DBD, until their query is slow, VERTICA have to know, detect this event, and have have DBD run automatically for users, and suggest the better projections design, if the existing projection is not good enough. Of course, there will be a lot of work that need to be done, before we can actually fully achieve the automation. But we are working on that. At the end of day, what the user really wants, is the fast database, right? And thank you for listening to my presentation. so I hope you find it useful. Now let's get ready for the Q&A.

hello everybody and thank you for joining us today for the virtual verdict of BBC 2020 today's breakout session is entitled the shortest path to Vertica best practices for data warehouse migration ETL I'm Jeff Healey I'll leave verdict and marketing I'll be your host for this breakout session joining me today are Marco guesser and Mauricio lychee vertical product engineer is joining us from yume region but before we begin I encourage you to submit questions or comments or in the virtual session don't have to wait just type question in a comment in the question box below the slides that click Submit as always there will be a Q&A session the end of the presentation will answer as many questions were able to during that time any questions we don't address we'll do our best to answer them offline alternatively visit Vertica forums that formed at vertical comm to post your questions there after the session our engineering team is planning to join the forums to keep the conversation going also reminder that you can maximize your screen by clicking the double arrow button and lower right corner of the sides and yes this virtual session is being recorded be available to view on demand this week send you a notification as soon as it's ready now let's get started over to you mark marco andretti oh hello everybody this is Marco speaking a sales engineer from Amir said I'll just get going ah this is the agenda part one will be done by me part two will be done by Mauricio the agenda is as you can see big bang or piece by piece and the migration of the DTL migration of the physical data model migration of et I saw VTL + bi functionality what to do with store procedures what to do with any possible existing user defined functions and migration of the data doctor will be by Maurice it you want to talk about emeritus Rider yeah hello everybody my name is Mauricio Felicia and I'm a birth record pre-sales like Marco I'm going to talk about how to optimize that were always using some specific vertical techniques like table flattening live aggregated projections so let me start with be a quick overview of the data browser migration process we are going to talk about today and normally we often suggest to start migrating the current that allows the older disease with limited or minimal changes in the overall architecture and yeah clearly we will have to port the DDL or to redirect the data access tool and we will platform but we should minimizing the initial phase the amount of changes in order to go go live as soon as possible this is something that we also suggest in the second phase we can start optimizing Bill arouse and which again with no or minimal changes in the architecture as such and during this optimization phase we can create for example dog projections or for some specific query or optimize encoding or change some of the visual spools this is something that we normally do if and when needed and finally and again if and when needed we go through the architectural design for these operations using full vertical techniques in order to take advantage of all the features we have in vertical and this is normally an iterative approach so we go back to name some of the specific feature before moving back to the architecture and science we are going through this process in the next few slides ok instead in order to encourage everyone to keep using their common sense when migrating to a new database management system people are you often afraid of it it's just often useful to use the analogy of how smooth in your old home you might have developed solutions for your everyday life that make perfect sense there for example if your old cent burner dog can't walk anymore you might be using a fork lifter to heap in through your window in the old home well in the new home consider the elevator and don't complain that the window is too small to fit the dog through this is very much in the same way as Narita but starting to make the transition gentle again I love to remain in my analogy with the house move picture your new house as your new holiday home begin to install everything you miss and everything you like from your old home once you have everything you need in your new house you can shut down themselves the old one so move each by feet and go for quick wins to make your audience happy you do bigbang only if they are going to retire the platform you are sitting on where you're really on a sinking ship otherwise again identify quick wings implement published and quickly in Vertica reap the benefits enjoy the applause use the gained reputation for further funding and if you find that nobody's using the old platform anymore you can shut it down if you really have to migrate you can still go to really go to big battle in one go only if you absolutely have to otherwise migrate by subject area use the group all similar clear divisions right having said that ah you start off by migrating objects objects in the database that's one of the very first steps it consists of migrating verbs the places where you can put the other objects into that is owners locations which is usually schemers then what do you have that you extract tables news then you convert the object definition deploy them to Vertica and think that you shouldn't do it manually never type what you can generate ultimate whatever you can use it enrolls usually there is a system tables in the old database that contains all the roads you can export those to a file reformat them and then you have a create role and create user scripts that you can apply to Vertica if LDAP Active Directory was used for the authentication the old database vertical supports anything within the l dubs standard catalogued schemas should be relatively straightforward with maybe sometimes the difference Vertica does not restrict you by defining a schema as a collection of all objects owned by a user but it supports it emulates it for old times sake Vertica does not need the catalog or if you absolutely need the catalog from the old tools that you use it it usually said it is always set to the name of the database in case of vertical having had now the schemas the catalogs the users and roles in place move the take the definition language of Jesus thought if you are allowed to it's best to use a tool that translates to date types in the PTL generated you might see as a mention of old idea to listen by memory to by the way several times in this presentation we are very happy to have it it actually can export the old database table definition because they got it works with the odbc it gets what the old database ODBC driver translates to ODBC and then it has internal translation tables to several target schema to several target DBMS flavors the most important which is obviously vertical if they force you to use something else there are always tubes like sequel plots in Oracle the show table command in Tara data etc H each DBMS should have a set of tools to extract the object definitions to be deployed in the other instance of the same DBMS ah if I talk about youth views usually a very new definition also in the old database catalog one thing that you might you you use special a bit of special care synonyms is something that were to get emulated different ways depending on the specific needs I said I stop you on the view or table to be referred to or something that is really neat but other databases don't have the search path in particular that works that works very much like the path environment variable in Windows or Linux where you specify in a table an object name without the schema name and then it searched it first in the first entry of the search path then in a second then in third which makes synonym hugely completely unneeded when you generate uvl we remained in the analogy of moving house dust and clean your stuff before placing it in the new house if you see a table like the one here at the bottom this is usually corpse of a bad migration in the past already an ID is usually an integer and not an almost floating-point data type a first name hardly ever has 256 characters and that if it's called higher DT it's not necessarily needed to store the second when somebody was hired so take good care in using while you are moving dust off your stuff and use better data types the same applies especially could string how many bytes does a string container contains for eurozone's it's not for it's actually 12 euros in utf-8 in the way that Vertica encodes strings and ASCII characters one died but the Euro sign thinks three that means that you have to very often you have when you have a single byte character set up a source you have to pay attention oversize it first because otherwise it gets rejected or truncated and then you you will have to very carefully check what their best science is the best promising is the most promising approach is to initially dimension strings in multiples of very initial length and again ODP with the command you see there would be - I you 2 comma 4 will double the lengths of what otherwise will single byte character and multiply that for the length of characters that are wide characters in traditional databases and then load the representative sample of your cells data and profile using the tools that we personally use to find the actually longest datatype and then make them shorter notice you might be talking about the issues of having too long and too big data types on projection design are we live and die with our projects you might know remember the rules on how default projects has come to exist the way that we do initially would be just like for the profiling load a representative sample of the data collector representative set of already known queries from the Vertica database designer and you don't have to decide immediately you can always amend things and otherwise follow the laws of physics avoid moving data back and forth across nodes avoid heavy iOS if you can design your your projections initially by hand encoding matters you know that the database designer is a very tight fisted thing it would optimize to use as little space as possible you will have to think of the fact that if you compress very well you might end up using more time in reading it this is the testimony to run once using several encoding types and you see that they are l e is the wrong length encoded if sorted is not even visible while the others are considerably slower you can get those nights and look it in look at them in detail I will go in detail you now hear about it VI migrations move usually you can expect 80% of everything to work to be able to live to be lifted and shifted you don't need most of the pre aggregated tables because we have live like regain projections many BI tools have specialized query objects for the dimensions and the facts and we have the possibility to use flatten tables that are going to be talked about later you might have to ride those by hand you will be able to switch off casting because vertical speeds of everything with laps Lyle aggregate projections and you have worked with molap cubes before you very probably won't meet them at all ETL tools what you will have to do is if you do it row by row in the old database consider changing everything to very big transactions and if you use in search statements with parameter markers consider writing to make pipes and using verticals copy command mouse inserts yeah copy c'mon that's what I have here ask you custom functionality you can see on this slide the verticals the biggest number of functions in the database we compare them regularly by far compared to any other database you might find that many of them that you have written won't be needed on the new database so look at the vertical catalog instead of trying to look to migrate a function that you don't need stored procedures are very often used in the old database to overcome their shortcomings that Vertica doesn't have very rarely you will have to actually write a procedure that involves a loop but it's really in our experience very very rarely usually you can just switch to standard scripting and this is basically repeating what Mauricio said in the interest of time I will skip this look at this one here the most of the database data warehouse migration talks should be automatic you can use you can automate GDL migration using ODB which is crucial data profiling it's not crucial but game-changing the encoding is the same thing you can automate at you using our database designer the physical data model optimization in general is game-changing you have the database designer use the provisioning use the old platforms tools to generate the SQL you have no objects without their onus is crucial and asking functions and procedures they are only crucial if they depict the company's intellectual property otherwise you can almost always replace them with something else that's it from me for now Thank You Marco Thank You Marco so we will now point our presentation talking about some of the Vertica that overall the presentation techniques that we can implement in order to improve the general efficiency of the dot arouse and let me start with a few simple messages well the first one is that you are supposed to optimize only if and when this is needed in most of the cases just a little shift from the old that allows to birth will provide you exhaust the person as if you were looking for or even better so in this case probably is not really needed to to optimize anything in case you want optimize or you need to optimize then keep in mind some of the vertical peculiarities for example implement delete and updates in the vertical way use live aggregate projections in order to avoid or better in order to limit the goodbye executions at one time used for flattening in order to avoid or limit joint and and then you can also implement invert have some specific birth extensions life for example time series analysis or machine learning on top of your data we will now start by reviewing the first of these ballots optimize if and when needed well if this is okay I mean if you get when you migrate from the old data where else to birth without any optimization if the first four month level is okay then probably you only took my jacketing but this is not the case one very easier to dispute in session technique that you can ask is to ask basket cells to optimize the physical data model using the birth ticket of a designer how well DB deal which is the vertical database designer has several interfaces here I'm going to use what we call the DB DB programmatic API so basically sequel functions and using other databases you might need to hire experts looking at your data your data browser your table definition creating indexes or whatever in vertical all you need is to run something like these are simple as six single sequel statement to get a very well optimized physical base model you see that we start creating a new design then we had to be redesigned tables and queries the queries that we want to optimize we set our target in this case we are tuning the physical data model in order to maximize query performances this is why we are using my design query and in our statement another possible journal tip would be to tune in order to reduce storage or a mix between during storage and cheering queries and finally we asked Vertica to produce and deploy these optimized design in a matter of literally it's a matter of minutes and in a few minutes what you can get is a fully optimized fiscal data model okay this is something very very easy to implement keep in mind some of the vertical peculiarities Vaska is very well tuned for load and query operations aunt Berta bright rose container to biscuits hi the Pharos container is a group of files we will never ever change the content of this file the fact that the Rose containers files are never modified is one of the political peculiarities and these approach led us to use minimal locks we can add multiple load operations in parallel against the very same table assuming we don't have a primary or unique constraint on the target table in parallel as a sage because they will end up in two different growth containers salad in read committed requires in not rocket fuel and can run concurrently with insert selected because the Select will work on a snapshot of the catalog when the transaction start this is what we call snapshot isolation the kappa recovery because we never change our rows files are very simple and robust so we have a huge amount of bandages due to the fact that we never change the content of B rows files contain indiarose containers but on the other side believes and updates require a little attention so what about delete first when you believe in the ethica you basically create a new object able it back so it appeared a bit later in the Rose or in memory and this vector will point to the data being deleted so that when the feed is executed Vertica will just ignore the rules listed in B delete records and it's not just about the leak and updating vertical consists of two operations delete and insert merge consists of either insert or update which interim is made of the little insert so basically if we tuned how the delete work we will also have tune the update in the merge so what should we do in order to optimize delete well remember what we said that every time we please actually we create a new object a delete vector so avoid committing believe and update too often we reduce work the work for the merge out for the removal method out activities that are run afterwards and be sure that all the interested projections will contain the column views in the dedicate this will let workers directly after access the projection without having to go through the super projection in order to create the vector and the delete will be much much faster and finally another very interesting optimization technique is trying to segregate the update and delete operation from Pyrenean third workload in order to reduce lock contention beliefs something we are going to discuss and these contain using partition partition operation this is exactly what I want to talk about now here you have a typical that arouse architecture so we have data arriving in a landing zone where the data is loaded that is from the data sources then we have a transformation a year writing into a staging area that in turn will feed the partitions block of data in the green data structure we have at the end those green data structure we have at the end are the ones used by the data access tools when they run their queries sometimes we might need to change old data for example because we have late records or maybe because we want to fix some errors that have been originated in the facilities so what we do in this case is we just copied back the partition we want to change or we want to adjust from the green interior a the end to the stage in the area we have a very fast operation which is Tokyo Station then we run our updates or our adjustment procedure or whatever we need in order to fix the errors in the data in the staging area and at the very same time people continues to you with green data structures that are at the end so we will never have contention between the two operations when we updating the staging area is completed what we have to do is just to run a swap partition between tables in order to swap the data that we just finished to adjust in be staging zone to the query area that is the green one at the end this swap partition is very fast is an atomic operation and basically what will happens is just that well exchange the pointer to the data this is a very very effective techniques and lot of customer useless so why flops on table and live aggregate for injections well basically we use slot in table and live aggregate objection to minimize or avoid joint this is what flatten table are used for or goodbye and this is what live aggregate projections are used for now compared to traditional data warehouses better can store and process and aggregate and join order of magnitudes more data that is a true columnar database joint and goodbye normally are not a problem at all they run faster than any traditional data browse that page there are still scenarios were deficits are so big and we are talking about petabytes of data and so quickly going that would mean be something in order to boost drop by and join performances and this is why you can't reduce live aggregate projections to perform aggregations hard loading time and limit the need for global appear on time and flux and tables to combine information from different entity uploading time and again avoid running joint has query undefined okay so live aggregate projections at this point in time we can use live aggregate projections using for built in aggregate functions which are some min Max and count okay let's see how this works suppose that you have a normal table in this case we have a table unit sold with three columns PIB their time and quantity which has been segmented in a given way and on top of this base table we call it uncle table we create a projection you see that we create the projection using the salad that will aggregate the data we get the PID we get the date portion of the time and we get the sum of quantity from from the base table grouping on the first two columns so PID and the date portion of day time okay what happens in this case when we load data into the base table all we have to do with load data into the base table when we load data into the base table we will feel of course big injections that assuming we are running with k61 we will have to projection to projections and we will know the data in those two projection with all the detail in data we are going to load into the table so PAB playtime and quantity but at the very same time at the very same time and without having to do nothing any any particular operation or without having to run any any ETL procedure we will also get automatically in the live aggregate projection for the data pre aggregated with be a big day portion of day time and the sum of quantity into the table name total quantity you see is something that we get for free without having to run any specific procedure and this is very very efficient so the key concept is that during the loading operation from VDL point of view is executed again the base table we do not explicitly aggregate data or we don't have any any plc do the aggregation is automatic and we'll bring the pizza to be live aggregate projection every time we go into the base table you see the two selection that we have we have on in this line on the left side and you see that those two selects will produce exactly the same result so running select PA did they trying some quantity from the base table or running the select star from the live aggregate projection will result exactly in the same data you know this is of course very useful but is much more useful result that if we and we can observe this if we run an explained if we run the select against the base table asking for this group data what happens behind the scene is that basically vertical itself that is a live aggregate projection with the data that has been already aggregating loading phase and rewrite your query using polite aggregate projection this happens automatically you see this is a query that ran a group by against unit sold and vertical decided to rewrite this clearly as something that has to be collected against the light aggregates projection because if I decrease this will save a huge amount of time and effort during the ETL cycle okay and is not just limited to be information you want to aggregate for example another query like select count this thing you might note that can't be seen better basically our goodbyes will also take advantage of the live aggregate injection and again this is something that happens automatically you don't have to do anything to get this okay one thing that we have to keep very very clear in mind Brassica what what we store in the live aggregate for injection are basically partially aggregated beta so in this example we have two inserts okay you see that we have the first insert that is entered in four volts and the second insert which is inserting five rules well in for each of these insert we will have a partial aggregation you will never know that after the first insert you will have a second one so better will calculate the aggregation of the data every time irin be insert it is a key concept and be also means that you can imagine lies the effectiveness of bees technique by inserting large chunk of data ok if you insert data row by row this technique live aggregate rejection is not very useful because for every goal that you insert you will have an aggregation so basically they'll live aggregate injection will end up containing the same number of rows that you have in the base table but if you everytime insert a large chunk of data the number of the aggregations that you will have in the library get from structure is much less than B base data so this is this is a key concept you can see how these works by counting the number of rows that you have in alive aggregate injection you see that if you run the select count star from the solved live aggregate rejection the query on the left side you will get four rules but actually if you explain this query you will see that he was reading six rows so this was because every of those two inserts that we're actively interested a few rows in three rows in India in the live aggregate projection so this is a key concept live aggregate projection keep partially aggregated data this final aggregation will always happen at runtime okay another which is very similar to be live aggregate projection or what we call top K projection we actually do not aggregate anything in the top case injection we just keep the last or limit the amount of rows that we collect using the limit over partition by all the by clothes and this again in this case we create on top of the base stable to top gay projection want to keep the last quantity that has been sold and the other one to keep the max quantity in both cases is just a matter of ordering the data in the first case using the B time column in the second page using quantity in both cases we fill projection with just the last roof and again this is something that we do when we insert data into the base table and this is something that happens automatically okay if we now run after the insert our select against either the max quantity okay or be lost wanted it okay we will get the very last you see that we have much less rows in the top k projections okay we told at the beginning that basically we can use for built-in function you might remember me max sum and count what if I want to create my own specific aggregation on top of the lid and customer sum up because our customers have very specific needs in terms of live aggregate projections well in this case you can code your own live aggregate production user-defined functions so you can create the user-defined transport function to implement any sort of complex aggregation while loading data basically after you implemented miss VPS you can deploy using a be pre pass approach that basically means the data is aggregated as loading time during the data ingestion or the batch approach that means that the data is when that woman is running on top which things to remember on live a granade projections they are limited to be built in function again some max min and count but you can call your own you DTF so you can do whatever you want they can reference only one table and for bass cab version before 9.3 it was impossible to update or delete on the uncle table this limit has been removed in 9.3 so you now can update and delete data from the uncle table okay live aggregate projection will follow the segmentation of the group by expression and in some cases the best optimizer can decide to pick the live aggregates objection or not depending on if depending on the fact that the aggregation is a consistent or not remember that if we insert and commit every single role to be uncoachable then we will end up with a live aggregate indirection that contains exactly the same number of rows in this case living block or using the base table it would be the same okay so this is one of the two fantastic techniques that we can implement in Burtka this live aggregate projection is basically to avoid or limit goodbyes the other which we are going to talk about is cutting table and be reused in order to avoid the means for joins remember that K is very fast running joints but when we scale up to petabytes of beta we need to boost and this is what we have in order to have is problem fixed regardless the amount of data we are dealing with so how what about suction table let me start with normalized schemas everybody knows what is a normalized scheme under is no but related stuff in this slide the main scope of an normalized schema is to reduce data redundancies so and the fact that we reduce data analysis is a good thing because we will obtain fast and more brides we will have to write into a database small chunks of data into the right table the problem with these normalized schemas is that when you run your queries you have to put together the information that arrives from different table and be required to run joint again jointly that again normally is very good to run joint but sometimes the amount of data makes not easy to deal with joints and joints sometimes are not easy to tune what happens in in the normal let's say traditional data browser is that we D normalize the schemas normally either manually or using an ETL so basically we have on one side in this light on the left side the normalized schemas where we can get very fast right on the other side on the left we have the wider table where we run all the three joints and pre aggregation in order to prepare the data for the queries and so we will have fast bribes on the left fast reads on the Left sorry fast bra on the right and fast read on the left side of these slides the probability in the middle because we will push all the complexity in the middle in the ETL that will have to transform be normalized schema into the water table and the way we normally implement these either manually using procedures that we call the door using ETL this is what happens in traditional data warehouse is that we will have to coach in ETL layer in order to round the insert select that will feed from the normalized schema and right into the widest table at the end the one that is used by the data access tools we we are going to to view store to run our theories so this approach is costly because of course someone will have to code this ETL and is slow because someone will have to execute those batches normally overnight after loading the data and maybe someone will have to check the following morning that everything was ok with the batch and is resource intensive of course and is also human being intensive because of the people that will have to code and check the results it ever thrown because it can fail and introduce a latency because there is a get in the time axis between the time t0 when you load the data into be normalized schema and the time t1 when we get the data finally ready to be to be queried so what would be inverter to facilitate this process is to create this flatten table with the flattened T work first you avoid data redundancy because you don't need the wide table on the normalized schema on the left side second is fully automatic you don't have to do anything you just have to insert the data into the water table and the ETL that you have coded is transformed into an insert select by vatika automatically you don't have to do anything it's robust and this Latin c0 is a single fast as soon as you load the data into the water table you will get all the joints executed for you so let's have a look on how it works in this case we have the table we are going to flatten and basically we have to focus on two different clauses the first one is you see that there is one table here I mentioned value 1 which can be defined as default and then the Select or set using okay the difference between the fold and set using is when the data is populated if we use default data is populated as soon as we know the data into the base table if we use set using Google Earth to refresh but everything is there I mean you don't need them ETL you don't need to code any transformation because everything is in the table definition itself and it's for free and of course is in latency zero so as soon as you load the other columns you will have the dimension value valued as well okay let's see an example here suppose here we have a dimension table customer dimension that is on the left side and we have a fact table on on the right you see that the fact table uses columns like o underscore name or Oh the score city which are basically the result of the salad on top of the customer dimension so Beezus were the join is executed as soon as a remote data into the fact table directly into the fact table without of course loading data that arise from the dimension all the data from the dimension will be populated automatically so let's have an example here suppose that we are running this insert as you can see we are running be inserted directly into the fact table and we are loading o ID customer ID and total we are not loading made a major name no city those name and city will be automatically populated by Vertica for you because of the definition of the flood table okay you see behave well all you need in order to have your widest tables built for you your flattened table and this means that at runtime you won't need any join between base fuck table and the customer dimension that we have used in order to calculate name and city because the data is already there this was using default the other option was is using set using the concept is absolutely the same you see that in this case on the on the right side we have we have basically replaced this all on the school name default with all underscore name set using and same is true for city the concept that I said is the same but in this case which we set using then we will have to refresh you see that we have to run these select trash columns and then the name of the table in this case all columns will be fresh or you can specify only certain columns and this will bring the values for name and city reading from the customer dimension so this technique this technique is extremely useful the difference between default and said choosing just to summarize the most important differences remember you just have to remember that default will relate your target when you load set using when you refresh end and in some cases you might need to use them both so in some cases you might want to use both default end set using in this example here we'll see that we define the underscore name using both default and securing and this means that we love the data populated either when we load the data into the base table or when we run the Refresh this is summary of the technique that we can implement in birth in order to make our and other browsers even more efficient and well basically this is the end of our presentation thank you for listening and now we are ready for the Q&A session you

>> Paige: Hello, everybody and thank you for joining us today on the Virtual Vertica BDC 2020. Today's breakout session is entitled A Technical Overview of the Vertica Architecture. I'm Paige Roberts, Open Source Relations Manager at Vertica and I'll be your host for this webinar. Now joining me is Ryan Role-kuh? Did I say that right? (laughs) He's a Vertica Senior Software Engineer. >> Ryan: So it's Roelke. (laughs) >> Paige: Roelke, okay, I got it, all right. Ryan Roelke. And before we begin, I want to be sure and encourage you guys to submit your questions or your comments during the virtual session while Ryan is talking as you think of them as you go along. You don't have to wait to the end, just type in your question or your comment in the question box below the slides and click submit. There'll be a Q and A at the end of the presentation and we'll answer as many questions as we're able to during that time. Any questions that we don't address, we'll do our best to get back to you offline. Now, alternatively, you can visit the Vertica forums to post your question there after the session as well. Our engineering team is planning to join the forums to keep the conversation going, so you can have a chat afterwards with the engineer, just like any other conference. Now also, you can maximize your screen by clicking the double arrow button in the lower right corner of the slides and before you ask, yes, this virtual session is being recorded and it will be available to view on demand this week. We'll send you a notification as soon as it's ready. Now, let's get started. Over to you, Ryan. >> Ryan: Thanks, Paige. Good afternoon, everybody. My name is Ryan and I'm a Senior Software Engineer on Vertica's Development Team. I primarily work on improving Vertica's query execution engine, so usually in the space of making things faster. Today, I'm here to talk about something that's more general than that, so we're going to go through a technical overview of the Vertica architecture. So the intent of this talk, essentially, is to just explain some of the basic aspects of how Vertica works and what makes it such a great database software and to explain what makes a query execute so fast in Vertica, we'll provide some background to explain why other databases don't keep up. And we'll use that as a starting point to discuss an academic database that paved the way for Vertica. And then we'll explain how Vertica design builds upon that academic database to be the great software that it is today. I want to start by sharing somebody's approximation of an internet minute at some point in 2019. All of the data on this slide is generated by thousands or even millions of users and that's a huge amount of activity. Most of the applications depicted here are backed by one or more databases. Most of this activity will eventually result in changes to those databases. For the most part, we can categorize the way these databases are used into one of two paradigms. First up, we have online transaction processing or OLTP. OLTP workloads usually operate on single entries in a database, so an update to a retail inventory or a change in a bank account balance are both great examples of OLTP operations. Updates to these data sets must be visible immediately and there could be many transactions occurring concurrently from many different users. OLTP queries are usually key value queries. The key uniquely identifies the single entry in a database for reading or writing. Early databases and applications were probably designed for OLTP workloads. This example on the slide is typical of an OLTP workload. We have a table, accounts, such as for a bank, which tracks information for each of the bank's clients. An update query, like the one depicted here, might be run whenever a user deposits $10 into their bank account. Our second category is online analytical processing or OLAP which is more about using your data for decision making. If you have a hardware device which periodically records how it's doing, you could analyze trends of all your devices over time to observe what data patterns are likely to lead to failure or if you're Google, you might log user search activity to identify which links helped your users find the answer. Analytical processing has always been around but with the advent of the internet, it happened at scales that were unimaginable, even just 20 years ago. This SQL example is something you might see in an OLAP workload. We have a table, searches, logging user activity. We will eventually see one row in this table for each query submitted by users. If we want to find out what time of day our users are most active, then we could write a query like this one on the slide which counts the number of unique users running searches for each hour of the day. So now let's rewind to 2005. We don't have a picture of an internet minute in 2005, we don't have the data for that. We also don't have the data for a lot of other things. The term Big Data is not quite yet on anyone's radar and The Cloud is also not quite there or it's just starting to be. So if you have a database serving your application, it's probably optimized for OLTP workloads. OLAP workloads just aren't mainstream yet and database engineers probably don't have them in mind. So let's innovate. It's still 2005 and we want to try something new with our database. Let's take a look at what happens when we do run an analytic workload in 2005. Let's use as a motivating example a table of stock prices over time. In our table, the symbol column identifies the stock that was traded, the price column identifies the new price and the timestamp column indicates when the price changed. We have several other columns which, we should know that they're there, but we're not going to use them in any example queries. This table is designed for analytic queries. We're probably not going to make any updates or look at individual rows since we're logging historical data and want to analyze changes in stock price over time. Our database system is built to serve OLTP use cases, so it's probably going to store the table on disk in a single file like this one. Notice that each row contains all of the columns of our data in row major order. There's probably an index somewhere in the memory of the system which will help us to point lookups. Maybe our system expects that we will use the stock symbol and the trade time as lookup keys. So an index will provide quick lookups for those columns to the position of the whole row in the file. If we did have an update to a single row, then this representation would work great. We would seek to the row that we're interested in, finding it would probably be very fast using the in-memory index. And then we would update the file in place with our new value. On the other hand, if we ran an analytic query like we want to, the data access pattern is very different. The index is not helpful because we're looking up a whole range of rows, not just a single row. As a result, the only way to find the rows that we actually need for this query is to scan the entire file. We're going to end up scanning a lot of data that we don't need and that won't just be the rows that we don't need, there's many other columns in this table. Many information about who made the transaction, and we'll also be scanning through those columns for every single row in this table. That could be a very serious problem once we consider the scale of this file. Stocks change a lot, we probably have thousands or millions or maybe even billions of rows that are going to be stored in this file and we're going to scan all of these extra columns for every single row. If we tried out our stocks use case behind the desk for the Fortune 500 company, then we're probably going to be pretty disappointed. Our queries will eventually finish, but it might take so long that we don't even care about the answer anymore by the time that they do. Our database is not built for the task we want to use it for. Around the same time, a team of researchers in the North East have become aware of this problem and they decided to dedicate their time and research to it. These researchers weren't just anybody. The fruits of their labor, which we now like to call the C-Store Paper, was published by eventual Turing Award winner, Mike Stonebraker, along with several other researchers from elite universities. This paper presents the design of a read-optimized relational DBMS that contrasts sharply with most current systems, which are write-optimized. That sounds exactly like what we want for our stocks use case. Reasoning about what makes our queries executions so slow brought our researchers to the Memory Hierarchy, which essentially is a visualization of the relative speeds of different parts of a computer. At the top of the hierarchy, we have the fastest data units, which are, of course, also the most expensive to produce. As we move down the hierarchy, components get slower but also much cheaper and thus you can have more of them. Our OLTP databases data is stored in a file on the hard disk. We scanned the entirety of this file, even though we didn't need most of the data and now it turns out, that is just about the slowest thing that our query could possibly be doing by over two orders of magnitude. It should be clear, based on that, that the best thing we can do to optimize our query's execution is to avoid reading unnecessary data from the disk and that's what the C-Store researchers decided to look at. The key innovation of the C-Store paper does exactly that. Instead of storing data in a row major order, in a large file on disk, they transposed the data and stored each column in its own file. Now, if we run the same select query, we read only the relevant columns. The unnamed columns don't factor into the table scan at all since we don't even open the files. Zooming out to an internet scale sized data set, we can appreciate the savings here a lot more. But we still have to read a lot of data that we don't need to answer this particular query. Remember, we had two predicates, one on the symbol column and one on the timestamp column. Our query is only interested in AAPL stock, but we're still reading rows for all of the other stocks. So what can we do to optimize our disk read even more? Let's first partition our data set into different files based on the timestamp date. This means that we will keep separate files for each date. When we query the stocks table, the database knows all of the files we have to open. If we have a simple predicate on the timestamp column, as our sample query does, then the database can use it to figure out which files we don't have to look at at all. So now all of our disk reads that we have to do to answer our query will produce rows that pass the timestamp predicate. This eliminates a lot of wasteful disk reads. But not all of them. We do have another predicate on the symbol column where symbol equals AAPL. We'd like to avoid disk reads of rows that don't satisfy that predicate either. And we can avoid those disk reads by clustering all the rows that match the symbol predicate together. If all of the AAPL rows are adjacent, then as soon as we see something different, we can stop reading the file. We won't see any more rows that can pass the predicate. Then we can use the positions of the rows we did find to identify which pieces of the other columns we need to read. One technique that we can use to cluster the rows is sorting. So we'll use the symbol column as a sort key for all of the columns. And that way we can reconstruct a whole row by seeking to the same row position in each file. It turns out, having sorted all of the rows, we can do a bit more. We don't have any more wasted disk reads but we can still be more efficient with how we're using the disk. We've clustered all of the rows with the same symbol together so we don't really need to bother repeating the symbol so many times in the same file. Let's just write the value once and say how many rows we have. This one length encoding technique can compress large numbers of rows into a small amount of space. In this example, we do de-duplicate just a few rows but you can imagine de-duplicating many thousands of rows instead. This encoding is great for reducing the amounts of disk we need to read at query time, but it also has the additional benefit of reducing the total size of our stored data. Now our query requires substantially fewer disk reads than it did when we started. Let's recap what the C-Store paper did to achieve that. First, we transposed our data to store each column in its own file. Now, queries only have to read the columns used in the query. Second, we partitioned the data into multiple file sets so that all rows in a file have the same value for the partition column. Now, a predicate on the partition column can skip non-matching file sets entirely. Third, we selected a column of our data to use as a sort key. Now rows with the same value for that column are clustered together, which allows our query to stop reading data once it finds non-matching rows. Finally, sorting the data this way enables high compression ratios, using one length encoding which minimizes the size of the data stored on the disk. The C-Store system combined each of these innovative ideas to produce an academically significant result. And if you used it behind the desk of a Fortune 500 company in 2005, you probably would've been pretty pleased. But it's not 2005 anymore and the requirements of a modern database system are much stricter. So let's take a look at how C-Store fairs in 2020. First of all, we have designed the storage layer of our database to optimize a single query in a single application. Our design optimizes the heck out of that query and probably some similar ones but if we want to do anything else with our data, we might be in a bit of trouble. What if we just decide we want to ask a different question? For example, in our stock example, what if we want to plot all the trade made by a single user over a large window of time? How do our optimizations for the previous query measure up here? Well, our data's partitioned on the trade date, that could still be useful, depending on our new query. If we want to look at a trader's activity over a long period of time, we would have to open a lot of files. But if we're still interested in just a day's worth of data, then this optimization is still an optimization. Within each file, our data is ordered on the stock symbol. That's probably not too useful anymore, the rows for a single trader aren't going to be clustered together so we will have to scan all of the rows in order to figure out which ones match. You could imagine a worse design but as it becomes crucial to optimize this new type of query, then we might have to go as far as reconfiguring the whole database. The next problem of one of scale. One server is probably not good enough to serve a database in 2020. C-Store, as described, runs on a single server and stores lots of files. What if the data overwhelms this small system? We could imagine exhausting the file system's inodes limit with lots of small files due to our partitioning scheme. Or we could imagine something simpler, just filling up the disk with huge volumes of data. But there's an even simpler problem than that. What if something goes wrong and C-Store crashes? Then our data is no longer available to us until the single server is brought back up. A third concern, another one of scalability, is that one deployment does not really suit all possible things and use cases we could imagine. We haven't really said anything about being flexible. A contemporary database system has to integrate with many other applications, which might themselves have pretty restricted deployment options. Or the demands imposed by our workloads have changed and the setup you had before doesn't suit what you need now. C-Store doesn't do anything to address these concerns. What the C-Store paper did do was lead very quickly to the founding of Vertica. Vertica's architecture and design are essentially all about bringing the C-Store designs into an enterprise software system. The C-Store paper was just an academic exercise so it didn't really need to address any of the hard problems that we just talked about. But Vertica, the first commercial database built upon the ideas of the C-Store paper would definitely have to. This brings us back to the present to look at how an analytic query runs in 2020 on the Vertica Analytic Database. Vertica takes the key idea from the paper, can we significantly improve query performance by changing the way our data is stored and give its users the tools to customize their storage layer in order to heavily optimize really important or commonly wrong queries. On top of that, Vertica is a distributed system which allows it to scale up to internet-sized data sets, as well as have better reliability and uptime. We'll now take a brief look at what Vertica does to address the three inadequacies of the C-Store system that we mentioned. To avoid locking into a single database design, Vertica provides tools for the database user to customize the way their data is stored. To address the shortcomings of a single node system, Vertica coordinates processing among multiple nodes. To acknowledge the large variety of desirable deployments, Vertica does not require any specialized hardware and has many features which smoothly integrate it with a Cloud computing environment. First, we'll look at the database design problem. We're a SQL database, so our users are writing SQL and describing their data in SQL way, the Create Table statement. Create Table is a logical description of what your data looks like but it doesn't specify the way that it has to be stored, For a single Create Table, we could imagine a lot of different storage layouts. Vertica adds some extensions to SQL so that users can go even further than Create Table and describe the way that they want the data to be stored. Using terminology from the C-Store paper, we provide the Create Projection statement. Create Projection specifies how table data should be laid out, including column encoding and sort order. A table can have multiple projections, each of which could be ordered on different columns. When you query a table, Vertica will answer the query using the projection which it determines to be the best match. Referring back to our stock example, here's a sample Create Table and Create Projection statement. Let's focus on our heavily optimized example query, which had predicates on the stock symbol and date. We specify that the table data is to be partitioned by date. The Create Projection Statement here is excellent for this query. We specify using the order by clause that the data should be ordered according to our predicates. We'll use the timestamp as a secondary sort key. Each projection stores a copy of the table data. If you don't expect to need a particular column in a projection, then you can leave it out. Our average price query didn't care about who did the trading, so maybe our projection design for this query can leave the trader column out entirely. If the question we want to ask ever does change, maybe we already have a suitable projection, but if we don't, then we can create another one. This example shows another projection which would be much better at identifying trends of traders, rather than identifying trends for a particular stock. Next, let's take a look at our second problem, that one, or excuse me, so how should you decide what design is best for your queries? Well, you could spend a lot of time figuring it out on your own, or you could use Vertica's Database Designer tool which will help you by automatically analyzing your queries and spitting out a design which it thinks is going to work really well. If you want to learn more about the Database Designer Tool, then you should attend the session Vertica Database Designer- Today and Tomorrow which will tell you a lot about what the Database Designer does and some recent improvements that we have made. Okay, now we'll move to our next problem. (laughs) The challenge that one server does not fit all. In 2020, we have several orders of magnitude more data than we had in 2005. And you need a lot more hardware to crunch it. It's not tractable to keep multiple petabytes of data in a system with a single server. So Vertica doesn't try. Vertica is a distributed system so will deploy multiple severs which work together to maintain such a high data volume. In a traditional Vertica deployment, each node keeps some of the data in its own locally-attached storage. Data is replicated so that there is a redundant copy somewhere else in the system. If any one node goes down, then the data that it served is still available on a different node. We'll also have it so that in the system, there's no special node with extra duties. All nodes are created equal. This ensures that there is no single point of failure. Rather than replicate all of your data, Vertica divvies it up amongst all of the nodes in your system. We call this segmentation. The way data is segmented is another parameter of storage customization and it can definitely have an impact upon query performance. A common way to segment data is by using a hash expression, which essentially randomizes the node that a row of data belongs to. But with a guarantee that the same data will always end up in the same place. Describing the way data is segmented is another part of the Create Projection Statement, as seen in this example. Here we segment on the hash of the symbol column so all rows with the same symbol will end up on the same node. For each row that we load into the system, we'll apply our segmentation expression. The result determines which segment the row belongs to and then we'll send the row to each node which holds the copy of that segment. In this example, our projection is marked KSAFE 1, so we will keep one redundant copy of each segment. When we load a row, we might find that its segment had copied on Node One and Node Three, so we'll send a copy of the row to each of those nodes. If Node One is temporarily disconnected from the network, then Node Three can serve the other copy of the segment so that the whole system remains available. The last challenge we brought up from the C-Store design was that one deployment does not fit all. Vertica's cluster design neatly addressed many of our concerns here. Our use of segmentation to distribute data means that a Vertica system can scale to any size of deployment. And since we lack any special hardware or nodes with special purposes, Vertica servers can run anywhere, on premise or in the Cloud. But let's suppose you need to scale out your cluster to rise to the demands of a higher workload. Suppose you want to add another node. This changes the division of the segmentation space. We'll have to re-segment every row in the database to find its new home and then we'll have to move around any data that belongs to a different segment. This is a very expensive operation, not something you want to be doing all that often. Traditional Vertica doesn't solve that problem especially well, but Vertica Eon Mode definitely does. Vertica's Eon Mode is a large set of features which are designed with a Cloud computing environment in mind. One feature of this design is elastic throughput scaling, which is the idea that you can smoothly change your cluster size without having to pay the expenses of shuffling your entire database. Vertica Eon Mode had an entire session dedicated to it this morning. I won't say any more about it here, but maybe you already attended that session or if you haven't, then I definitely encourage you to listen to the recording. If you'd like to learn more about the Vertica architecture, then you'll find on this slide links to several of the academic conference publications. These four papers here, as well as Vertica Seven Years Later paper which describes some of the Vertica designs seven years after the founding and also a paper about the innovations of Eon Mode and of course, the Vertica documentation is an excellent resource for learning more about what's going on in a Vertica system. I hope you enjoyed learning about the Vertica architecture. I would be very happy to take all of your questions now. Thank you for attending this session.