(bright music) >> Hi, everybody, this is Dave Vellante, welcome back to Supercloud 2. Last August, at the first Supercloud event, we invited the broader community to help further define Supercloud, we assessed its viability, and identified the critical elements and deployment models of the concept. The objectives here at Supercloud too are, first of all, to continue to tighten and test the concept, the second is, we want to get real world input from practitioners on the problems that they're facing and the viability of Supercloud in terms of applying it to their business. So on the program, we got companies like Walmart, Sachs, Western Union, Ionis Pharmaceuticals, NASDAQ, and others. And the third thing that we want to do is we want to drill into the intersection of cloud and data to project what the future looks like in the context of Supercloud. So in this segment, we want to explore the concept of data architectures and what's going to be required for Supercloud. And I'm pleased to welcome one of our Supercloud sponsors, ChaosSearch, Ed Walsh is the CEO of the company, with Thomas Hazel, who's the Founder, CTO, and Chief Scientist. Guys, good to see you again, thanks for coming into our Marlborough studio. >> Always great. >> Great to be here. >> Okay, so there's a little debate, I'm going to put you right in the spot. (Ed chuckling) A little debate going on in the community started by Bob Muglia, a former CEO of Snowflake, and he was at Microsoft for a long time, and he looked at the Supercloud definition, said, "I think you need to tighten it up a little bit." So, here's what he came up with. He said, "A Supercloud is a platform that provides a programmatically consistent set of services hosted on heterogeneous cloud providers." So he's calling it a platform, not an architecture, which was kind of interesting. And so presumably the platform owner is going to be responsible for the architecture, but Dr. Nelu Mihai, who's a computer scientist behind the Cloud of Clouds Project, he chimed in and responded with the following. He said, "Cloud is a programming paradigm supporting the entire lifecycle of applications with data and logic natively distributed. Supercloud is an open architecture that integrates heterogeneous clouds in an agnostic manner." So, Ed, words matter. Is this an architecture or is it a platform? >> Put us on the spot. So, I'm sure you have concepts, I would say it's an architectural or design principle. Listen, I look at Supercloud as a mega trend, just like cloud, just like data analytics. And some companies are using the principle, design principles, to literally get dramatically ahead of everyone else. I mean, things you couldn't possibly do if you didn't use cloud principles, right? So I think it's a Supercloud effect, you're able to do things you're not able to. So I think it's more a design principle, but if you do it right, you get dramatic effect as far as customer value. >> So the conversation that we were having with Muglia, and Tristan Handy of dbt Labs, was, I'll set it up as the following, and, Thomas, would love to get your thoughts, if you have a CRM, think about applications today, it's all about forms and codifying business processes, you type a bunch of stuff into Salesforce, and all the salespeople do it, and this machine generates a forecast. What if you have this new type of data app that pulls data from the transaction system, the e-commerce, the supply chain, the partner ecosystem, et cetera, and then, without humans, actually comes up with a plan. That's their vision. And Muglia was saying, in order to do that, you need to rethink data architectures and database architectures specifically, you need to get down to the level of how the data is stored on the disc. What are your thoughts on that? Well, first of all, I'm going to cop out, I think it's actually both. I do think it's a design principle, I think it's not open technology, but open APIs, open access, and you can build a platform on that design principle architecture. Now, I'm a database person, I love solving the database problems. >> I'm waited for you to launch into this. >> Yeah, so I mean, you know, Snowflake is a database, right? It's a distributed database. And we wanted to crack those codes, because, multi-region, multi-cloud, customers wanted access to their data, and their data is in a variety of forms, all these services that you're talked about. And so what I saw as a core principle was cloud object storage, everyone streams their data to cloud object storage. From there we said, well, how about we rethink database architecture, rethink file format, so that we can take each one of these services and bring them together, whether distributively or centrally, such that customers can access and get answers, whether it's operational data, whether it's business data, AKA search, or SQL, complex distributed joins. But we had to rethink the architecture. I like to say we're not a first generation, or a second, we're a third generation distributed database on pure, pure cloud storage, no caching, no SSDs. Why? Because all that availability, the cost of time, is a struggle, and cloud object storage, we think, is the answer. >> So when you're saying no caching, so when I think about how companies are solving some, you know, pretty hairy problems, take MySQL Heatwave, everybody thought Oracle was going to just forget about MySQL, well, they come out with Heatwave. And the way they solve problems, and you see their benchmarks against Amazon, "Oh, we crush everybody," is they put it all in memory. So you said no caching? You're not getting performance through caching? How is that true, and how are you getting performance? >> Well, so five, six years ago, right? When you realize that cloud object storage is going to be everywhere, and it's going to be a core foundational, if you will, fabric, what would you do? Well, a lot of times the second generation say, "We'll take it out of cloud storage, put in SSDs or something, and put into cache." And that adds a lot of time, adds a lot of costs. But I said, what if, what if we could actually make the first read hot, the first read distributed joins and searching? And so what we went out to do was said, we can't cache, because that's adds time, that adds cost. We have to make cloud object storage high performance, like it feels like a caching SSD. That's where our patents are, that's where our technology is, and we've spent many years working towards this. So, to me, if you can crack that code, a lot of these issues we're talking about, multi-region, multicloud, different services, everybody wants to send their data to the data lake, but then they move it out, we said, "Keep it right there." >> You nailed it, the data gravity. So, Bob's right, the data's coming in, and you need to get the data from everywhere, but you need an environment that you can deal with all that different schema, all the different type of technology, but also at scale. Bob's right, you cannot use memory or SSDs to cache that, that doesn't scale, it doesn't scale cost effectively. But if you could, and what you did, is you made object storage, S3 first, but object storage, the only persistence by doing that. And then we get performance, we should talk about it, it's literally, you know, hundreds of terabytes of queries, and it's done in seconds, it's done without memory caching. We have concepts of caching, but the only caching, the only persistence, is actually when we're doing caching, we're just keeping another side-eye track of things on the S3 itself. So we're using, actually, the object storage to be a database, which is kind of where Bob was saying, we agree, but that's what you started at, people thought you were crazy. >> And maybe make it live. Don't think of it as archival or temporary space, make it live, real time streaming, operational data. What we do is make it smart, we see the data coming in, we uniquely index it such that you can get your use cases, that are search, observability, security, or backend operational. But we don't have to have this, I dunno, static, fixed, siloed type of architecture technologies that were traditionally built prior to Supercloud thinking. >> And you don't have to move everything, essentially, you can do it wherever the data lands, whatever cloud across the globe, you're able to bring it together, you get the cost effectiveness, because the only persistence is the cheapest storage persistent layer you can buy. But the key thing is you cracked the code. >> We had to crack the code, right? That was the key thing. >> That's where the plans are. >> And then once you do that, then everything else gets easier to scale, your architecture, across regions, across cloud. >> Now, it's a general purpose database, as Bob was saying, but we use that database to solve a particular issue, which is around operational data, right? So, we agree with Bob's. >> Interesting. So this brings me to this concept of data, Jimata Gan is one of our speakers, you know, we talk about data fabric, which is a NetApp, originally NetApp concept, Gartner's kind of co-opted it. But so, the basic concept is, data lives everywhere, whether it's an S3 bucket, or a SQL database, or a data lake, it's just a node on the data mesh. So in your view, how does this fit in with Supercloud? Ed, you've said that you've built, essentially, an enabler for that, for the data mesh, I think you're an enabler for the Supercloud-like principles. This is a big, chewy opportunity, and it requires, you know, a team approach. There's got to be an ecosystem, there's not going to be one Supercloud to rule them all, so where does the ecosystem fit into the discussion, and where do you fit into the ecosystem? >> Right, so we agree completely, there's not one Supercloud in effect, but we use Supercloud principles to build our platform, and then, you know, the ecosystem's going to be built on leveraging what everyone else's secret powers are, right? So our power, our superpower, based upon what we built is, we deal with, if you're having any scale, or cost effective scale issues, with data, machine generated data, like business observability or security data, we are your force multiplier, we will take that in singularly, just let it, simply put it in your object storage wherever it sits, and we give you uniformity access to that using OpenAPI access, SQL, or you know, Elasticsearch API. So, that's what we do, that's our superpower. So I'll play it into data mesh, that's a perfect, we are a node on a data mesh, but I'll play it in the soup about how, the ecosystem, we see it kind of playing, and we talked about it in just in the last couple days, how we see this kind of possibly. Short term, our superpowers, we deal with this data that's coming at these environments, people, customers, building out observability or security environments, or vendors that are selling their own Supercloud, I do observability, the Datadogs of the world, dot dot dot, the Splunks of the world, dot dot dot, and security. So what we do is we fit in naturally. What we do is a cost effective scale, just land it anywhere in the world, we deal with ingest, and it's a cost effective, an order of magnitude, or two or three order magnitudes more cost effective. Allows them, their customers are asking them to do the impossible, "Give me fast monitoring alerting. I want it snappy, but I want it to keep two years of data, (laughs) and I want it cost effective." It doesn't work. They're good at the fast monitoring alerting, we're good at the long-term retention. And yet there's some gray area between those two, but one to one is actually cheaper, so we would partner. So the first ecosystem plays, who wants to have the ability to, really, all the data's in those same environments, the security observability players, they can literally, just through API, drag our data into their point to grab. We can make it seamless for customers. Right now, we make it helpful to customers. Your Datadog, we make a button, easy go from Datadog to us for logs, save you money. Same thing with Grafana. But you can also look at ecosystem, those same vendors, it used to be a year ago it was, you know, its all about how can you grow, like it's growth at all costs, now it's about cogs. So literally we can go an environment, you supply what your customer wants, but we can help with cogs. And one-on one in a partnership is better than you trying to build on your own. >> Thomas, you were saying you make the first read fast, so you think about Snowflake. Everybody wants to talk about Snowflake and Databricks. So, Snowflake, great, but you got to get the data in there. All right, so that's, can you help with that problem? >> I mean we want simple in, right? And if you have to have structure in, you're not simple. So the idea that you have a simple in, data lake, schema read type philosophy, but schema right type performance. And so what I wanted to do, what we have done, is have that simple lake, and stream that data real time, and those access points of Search or SQL, to go after whatever business case you need, security observability, warehouse integration. But the key thing is, how do I make that click, click, click answer, and do it quickly? And so what we want to do is, that first read has to be fast. Why? 'Cause then you're going to do all this siloing, layers, complexity. If your first read's not fast, you're at a disadvantage, particularly in cost. And nobody says I want less data, but everyone has to, whether they say we're going to shorten the window, we're going to use AI to choose, but in a security moment, when you don't have that answer, you're in trouble. And that's why we are this service, this Supercloud service, if you will, providing access, well-known search, well-known SQL type access, that if you just have one access point, you're at a disadvantage. >> We actually talked about Snowflake and BigQuery, and a different platform, Data Bricks. That's kind of where we see the phase two of ecosystem. One is easy, the low-hanging fruit is observability and security firms. But the next one is, what we do, our super power is dealing with this messy data that schema is changing like night and day. Pipelines are tough, and it's changing all the time, but you want these things fast, and it's big data around the world. That's the next point, just use us alongside, or inside, one of their platforms, and now we get the best of both worlds. Our superpower is keeping this messy data as a streaming, okay, not a batch thing, allow you to do that. So, that's the second one. And then to be honest, the third one, which plays you to Supercloud, it also plays perfectly in the data mesh, is if you really go to the ultimate thing, what we have done is made object storage, S3, GCS, and blob storage, we made it a database. Put, get, complex query with big joins. You know, so back to your original thing, and Muglia teed it up perfectly, we've done that. Now imagine if that's an ecosystem, who would want that? If it's, again, it's uniform available across all the regions, across all the clouds, and it's right next to where you are building a service, or a client's trying, that's where the ecosystem, I think people are going to use Superclouds for their superpowers. We're really good at this, allows that short term. I think the Snowflakes and the Data Bricks are the medium term, you know? And then I think eventually gets to, hey, listen if you can make object storage fast, you can just go after it with simple SQL queries, or elastic. Who would want that? I think that's where people are going to leverage it. It's not going to be one Supercloud, and we leverage the super clouds. >> Our viewpoint is smart object storage can be programmable, and so we agree with Bob, but we're not saying do it here, do it here. This core, fundamental layer across regions, across clouds, that everyone has? Simple in. Right now, it's hard to get data in for access for analysis. So we said, simply, we'll automate the entire process, give you API access across regions, across clouds. And again, how do you do a distributed join that's fast? How do you do a distributed join that doesn't cost you an arm or a leg? And how do you do it at scale? And that's where we've been focused. >> So prior, the cloud object store was a niche. >> Yeah. >> S3 obviously changed that. How standard is, essentially, object store across the different cloud platforms? Is that a problem for you? Is that an easy thing to solve? >> Well, let's talk about it. I mean we've fundamentally, yeah we've extracted it, but fundamentally, cloud object storage, put, get, and list. That's why it's so scalable, 'cause it doesn't have all these other components. That complexity is where we have moved up, and provide direct analytical API access. So because of its simplicity, and costs, and security, and reliability, it can scale naturally. I mean, really, distributed object storage is easy, it's put-get anywhere, now what we've done is we put a layer of intelligence, you know, call it smart object storage, where access is simple. So whether it's multi-region, do a query across, or multicloud, do a query across, or hunting, searching. >> We've had clients doing Amazon and Google, we have some Azure, but we see Amazon and Google more, and it's a consistent service across all of them. Just literally put your data in the bucket of choice, or folder of choice, click a couple buttons, literally click that to say "that's hot," and after that, it's hot, you can see it. But we're not moving data, the data gravity issue, that's the other. That it's already natively flowing to these pools of object storage across different regions and clouds. We don't move it, we index it right there, we're spinning up stateless compute, back to the Supercloud concept. But now that allows us to do all these other things, right? >> And it's no longer just cheap and deep object storage. Right? >> Yeah, we make it the same, like you have an analytic platform regardless of where you're at, you don't have to worry about that. Yeah, we deal with that, we deal with a stateless compute coming up -- >> And make it programmable. Be able to say, "I want this bucket to provide these answers." Right, that's really the hope, the vision. And the complexity to build the entire stack, and then connect them together, we said, the fabric is cloud storage, we just provide the intelligence on top. >> Let's bring it back to the customers, and one of the things we're exploring in Supercloud too is, you know, is Supercloud a solution looking for a problem? Is a multicloud really a problem? I mean, you hear, you know, a lot of the vendor marketing says, "Oh, it's a disaster, because it's all different across the clouds." And I talked to a lot of customers even as part of Supercloud too, they're like, "Well, I solved that problem by just going mono cloud." Well, but then you're not able to take advantage of a lot of the capabilities and the primitives that, you know, like Google's data, or you like Microsoft's simplicity, their RPA, whatever it is. So what are customers telling you, what are their near term problems that they're trying to solve today, and how are they thinking about the future? >> Listen, it's a real problem. I think it started, I think this is a a mega trend, just like cloud. Just, cloud data, and I always add, analytics, are the mega trends. If you're looking at those, if you're not considering using the Supercloud principles, in other words, leveraging what I have, abstracting it out, and getting the most out of that, and then build value on top, I think you're not going to be able to keep up, In fact, no way you're going to keep up with this data volume. It's a geometric challenge, and you're trying to do linear things. So clients aren't necessarily asking, hey, for Supercloud, but they're really saying, I need to have a better mechanism to simplify this and get value across it, and how do you abstract that out to do that? And that's where they're obviously, our conversations are more amazed what we're able to do, and what they're able to do with our platform, because if you think of what we've done, the S3, or GCS, or object storage, is they can't imagine the ingest, they can't imagine how easy, time to glass, one minute, no matter where it lands in the world, querying this in seconds for hundreds of terabytes squared. People are amazed, but that's kind of, so they're not asking for that, but they are amazed. And then when you start talking on it, if you're an enterprise person, you're building a big cloud data platform, or doing data or analytics, if you're not trying to leverage the public clouds, and somehow leverage all of them, and then build on top, then I think you're missing it. So they might not be asking for it, but they're doing it. >> And they're looking for a lens, you mentioned all these different services, how do I bring those together quickly? You know, our viewpoint, our service, is I have all these streams of data, create a lens where they want to go after it via search, go after via SQL, bring them together instantly, no e-tailing out, no define this table, put into this database. We said, let's have a service that creates a lens across all these streams, and then make those connections. I want to take my CRM with my Google AdWords, and maybe my Salesforce, how do I do analysis? Maybe I want to hunt first, maybe I want to join, maybe I want to add another stream to it. And so our viewpoint is, it's so natural to get into these lake platforms and then provide lenses to get that access. >> And they don't want it separate, they don't want something different here, and different there. They want it basically -- >> So this is our industry, right? If something new comes out, remember virtualization came out, "Oh my God, this is so great, it's going to solve all these problems." And all of a sudden it just got to be this big, more complex thing. Same thing with cloud, you know? It started out with S3, and then EC2, and now hundreds and hundreds of different services. So, it's a complex matter for a lot of people, and this creates problems for customers, especially when you got divisions that are using different clouds, and you're saying that the solution, or a solution for the part of the problem, is to really allow the data to stay in place on S3, use that standard, super simple, but then give it what, Ed, you've called superpower a couple of times, to make it fast, make it inexpensive, and allow you to do that across clouds. >> Yeah, yeah. >> I'll give you guys the last word on that. >> No, listen, I think, we think Supercloud allows you to do a lot more. And for us, data, everyone says more data, more problems, more budget issue, everyone knows more data is better, and we show you how to do it cost effectively at scale. And we couldn't have done it without the design principles of we're leveraging the Supercloud to get capabilities, and because we use super, just the object storage, we're able to get these capabilities of ingest, scale, cost effectiveness, and then we built on top of this. In the end, a database is a data platform that allows you to go after everything distributed, and to get one platform for analytics, no matter where it lands, that's where we think the Supercloud concepts are perfect, that's where our clients are seeing it, and we're kind of excited about it. >> Yeah a third generation database, Supercloud database, however we want to phrase it, and make it simple, but provide the value, and make it instant. >> Guys, thanks so much for coming into the studio today, I really thank you for your support of theCUBE, and theCUBE community, it allows us to provide events like this and free content. I really appreciate it. >> Oh, thank you. >> Thank you. >> All right, this is Dave Vellante for John Furrier in theCUBE community, thanks for being with us today. You're watching Supercloud 2, keep it right there for more thought provoking discussions around the future of cloud and data. (bright music)

hi everybody welcome back to re invent 2022. this is thecube's exclusive coverage we're here at the satellite set it's up on the fifth floor of the Venetian Conference Center and this is part of the global startup program the AWS startup showcase series that we've been running all through last year and and into this year with AWS and featuring some of its its Global Partners Ed wallson series the CEO of chaos search many times Cube Alum and Kevin Miller there's also a cube Alum vice president GM of S3 at AWS guys good to see you again yeah great to see you Dave hi Kevin this is we call this our Super Bowl so this must be like your I don't know uh World Cup it's a pretty big event yeah it's the World Cup for sure yeah so a lot of S3 talk you know I mean that's what got us all started in 2006 so absolutely what's new in S3 yeah it's been a great show we've had a number of really interesting launches over the last few weeks and a few at the show as well so you know we've been really focused on helping customers that are running Mass scale data Lakes including you know whether it's structured or unstructured data we actually announced just a few just an hour ago I think it was a new capability to give customers cross-account access points for sharing data securely with other parts of the organization and that's something that we'd heard from customers is as they are growing and have more data sets and they're looking to to get more out of their data they are increasingly looking to enable multiple teams across their businesses to access those data sets securely and that's what we provide with cross-count access points we also launched yesterday our multi-region access point failover capabilities and so again this is where customers have data sets and they're using multiple regions for certain critical workloads they're now able to to use that to fail to control the failover between different regions in AWS and then one other launch I would just highlight is some improvements we made to storage lens which is our really a very novel and you need capability to help customers really understand what storage they have where who's accessing it when it's being accessed and we added a bunch of new metrics storage lens has been pretty exciting for a lot of customers in fact we looked at the data and saw that customers who have adopted storage lens typically within six months they saved more than six times what they had invested in turning storage lens on and certainly in this environment right now we have a lot of customers who are it's pretty top of mind they're looking for ways to optimize their their costs in the cloud and take some of those savings and be able to reinvest them in new innovation so pretty exciting with the storage lens launch I think what's interesting about S3 is that you know pre-cloud Object Store was this kind of a niche right and then of course you guys announced you know S3 in 2006 as I said and okay great you know cheap and deep storage simple get put now the conversations about how to enable value from from data absolutely analytics and it's just a whole new world and Ed you've talked many times I love the term yeah we built chaos search on the on the shoulders of giants right and so the under underlying that is S3 but the value that you can build on top of that has been key and I don't think we've talked about his shoulders and Giants but we've talked about how we literally you know we have a big Vision right so hard to kind of solve the challenge to analytics at scale we really focus on the you know the you know Big Data coming environment get analytics so we talk about the on the shoulders Giants obviously Isaac Newton's you know metaphor of I learned from everything before and we layer on top so really when you talk about all the things come from S3 like I just smile because like we picked it up naturally we went all in an S3 and this is where I think you're going Dave but everyone is so let's just cut the chase like so any of the data platforms you're using S3 is what you're building but we did it a little bit differently so at first people using a cold storage like you said and then they ETL it up into a different platforms for analytics of different sorts now people are using it closer they're doing caching layers and cashing out and they're that's where but that's where the attributes of a scale or reliability are what we did is we actually make S3 a database so literally we have no persistence outside that three and that kind of comes in so it's working really well with clients because most of the thing is we pick up all these attributes of scale reliability and it shows up in the clients environments and so when you launch all these new scalable things we just see it like our clients constantly comment like one of our biggest customers fintech in uh Europe they go to Black Friday again black Friday's not one days and they lose scale from what is it 58 terabytes a day and they're going up to 187 terabytes a day and we don't Flinch they say how do you do that well we built our platform on S3 as long as you can stream it to S3 so they're saying I can't overrun S3 and it's a natural play so it's it's really nice that but we take out those attributes but same thing that's why we're able to you know help clients get you know really you know Equifax is a good example maybe they're able to consolidate 12 their divisions on one platform we couldn't have done that without the scale and the performance of what you can get S3 but also they saved 90 I'm able to do that but that's really because the only persistence is S3 and what you guys are delivering but and then we really for focus on shoulders Giants we're doing on top of that innovating on top of your platforms and bringing that out so things like you know we have a unique data representation that makes it easy to ingest this data because it's kind of coming at you four v's of big data we allow you to do that make it performant on s3h so now you're doing hot analytics on S3 as if it's just a native database in memory but there's no memory SSC caching and then multi-model once you get it there don't move it leverage it in place so you know elasticsearch access you know Cabana grafana access or SQL access with your tools so we're seeing that constantly but we always talk about on the shoulders of giants but even this week I get comments from our customers like how did you do that and most of it is because we built on top of what you guys provided so it's really working out pretty well and you know we talk a lot about digital transformation of course we had the pleasure sitting down with Adam solipski prior John Furrier flew to Seattle sits down his annual one-on-one with the AWS CEO which is kind of cool yeah it was it's good it's like study for the test you know and uh and so but but one of the interesting things he said was you know we're one of our challenges going forward is is how do we go Beyond digital transformation into business transformation like okay well that's that's interesting I was talking to a customer today AWS customer and obviously others because they're 100 year old company and they're basically their business was they call them like the Uber for for servicing appliances when your Appliance breaks you got to get a person to serve it a service if it's out of warranty you know these guys do that so they got to basically have a you know a network of technicians yeah and they gotta deal with the customers no phone right so they had a completely you know that was a business transformation right they're becoming you know everybody says they're coming a software company but they're building it of course yeah right on the cloud so wonder if you guys could each talk about what's what you're seeing in terms of changing not only in the sort of I.T and the digital transformation but also the business transformation yeah I know I I 100 agree that I think business transformation is probably that one of the top themes I'm hearing from customers of all sizes right now even in this environment I think customers are looking for what can I do to drive top line or you know improve bottom line or just improve my customer experience and really you know sort of have that effect where I'm helping customers get more done and you know it is it is very tricky because to do that successfully the customers that are doing that successfully I think are really getting into the lines of businesses and figuring out you know it's probably a different skill set possibly a different culture different norms and practices and process and so it's it's a lot more than just a like you said a lot more than just the technology involved but when it you know we sort of liquidate it down into the data that's where absolutely we see that as a critical function for lines of businesses to become more comfortable first off knowing what data sets they have what data they they could access but possibly aren't today and then starting to tap into those data sources and then as as that progresses figuring out how to share and collaborate with data sets across a company to you know to correlate across those data sets and and drive more insights and then as all that's being done of course it's important to measure the results and be able to really see is this what what effect is this having and proving that effect and certainly I've seen plenty of customers be able to show you know this is a percentage increase in top or bottom line and uh so that pattern is playing out a lot and actually a lot of how we think about where we're going with S3 is related to how do we make it easier for customers to to do everything that I just described to have to understand what data they have to make it accessible and you know it's great to have such a great ecosystem of partners that are then building on top of that and innovating to help customers connect really directly with the businesses that they're running and driving those insights well and customers are hours today one of the things I loved that Adam said he said where Amazon is strategically very very patient but tactically we're really impatient and the customers out there like how are you going to help me increase Revenue how are you going to help me cut costs you know we were talking about how off off camera how you know software can actually help do that yeah it's deflationary I love the quote right so software's deflationary as costs come up how do you go drive it also free up the team and you nail it it's like okay everyone wants to save money but they're not putting off these projects in fact the digital transformation or the business it's actually moving forward but they're getting a little bit bigger but everyone's looking for creative ways to look at their architecture and it becomes larger larger we talked about a couple of those examples but like even like uh things like observability they want to give this tool set this data to all the developers all their sres same data to all the security team and then to do that they need to find a way an architect should do that scale and save money simultaneously so we see constantly people who are pairing us up with some of these larger firms like uh or like keep your data dog keep your Splunk use us to reduce the cost that one and one is actually cheaper than what you have but then they use it either to save money we're saving 50 to 80 hard dollars but more importantly to free up your team from the toil and then they they turn around and make that budget neutral and then allowed to get the same tools to more people across the org because they're sometimes constrained of getting the access to everyone explain that a little bit more let's say I got a Splunk or data dog I'm sifting through you know logs how exactly do you help so it's pretty simple I'll use dad dog example so let's say using data dog preservability so it's just your developers your sres managing environments all these platforms are really good at being a monitoring alerting type of tool what they're not necessarily great at is keeping the data for longer periods like the log data the bigger data that's where we're strong what you see is like a data dog let's say you're using it for a minister for to keep 30 days of logs which is not enough like let's say you're running environment you're finding that performance issue you kind of want to look to last quarter in last month in or maybe last Black Friday so 30 days is not enough but will charge you two eighty two dollars and eighty cents a gigabyte don't focus on just 280 and then if you just turn the knob and keep seven days but keep two years of data on us which is on S3 it goes down to 22 cents plus our list price of 80 cents goes to a dollar two compared to 280. so here's the thing what they're able to do is just turn a knob get more data we do an integration so you can go right from data dog or grafana directly into our platform so the user doesn't see it but they save money A lot of times they don't just save the money now they use that to go fund and get data dog to a lot more people make sense so it's a creativity they're looking at it and they're looking at tools we see the same thing with a grafana if you look at the whole grafana play which is hey you can't put it in one place but put Prometheus for metrics or traces we fit well with logs but they're using that to bring down their costs because a lot of this data just really bogs down these applications the alerting monitoring are good at small data they're not good at the big data which is what we're really good at and then the one and one is actually less than you paid for the one so it and it works pretty well so things are really unpredictable right now in the economy you know during the pandemic we've sort of lockdown and then the stock market went crazy we're like okay it's going to end it's going to end and then it looked like it was going to end and then it you know but last year it reinvented just just in that sweet spot before Omicron so we we tucked it in which which was awesome right it was a great great event we really really missed one physical reinvent you know which was very rare so that's cool but I've called it the slingshot economy it feels like you know you're driving down the highway and you got to hit the brakes and then all of a sudden you're going okay we're through it Oh no you're gonna hit the brakes again yeah so it's very very hard to predict and I was listening to jassy this morning he was talking about yeah consumers they're still spending but what they're doing is they're they're shopping for more features they might be you know buying a TV that's less expensive you know more value for the money so okay so hopefully the consumer spending will get us out of this but you don't really know you know and I don't yeah you know we don't seem to have the algorithms we've never been through something like this before so what are you guys seeing in terms of customer Behavior given that uncertainty well one thing I would highlight that I think particularly going back to what we were just talking about as far as business and digital transformation I think some customers are still appreciating the fact that where you know yesterday you may have had to to buy some Capital put out some capital and commit to something for a large upfront expenditure is that you know today the value of being able to experiment and scale up and then most importantly scale down and dynamically based on is the experiment working out am I seeing real value from it and doing that on a time scale of a day or a week or a few months that is so important right now because again it gets to I am looking for a ways to innovate and to drive Top Line growth but I I can't commit to a multi-year sort of uh set of costs to to do that so and I think plenty of customers are finding that even a few months of experimentation gives them some really valuable insight as far as is this going to be successful or not and so I think that again just of course with S3 and storage from day one we've been elastic pay for what you use if you're not using the storage you don't get charged for it and I think that particularly right now having the applications and the rest of the ecosystem around the storage and the data be able to scale up and scale down is is just ever more important and when people see that like typically they're looking to do more with it so if they find you usually find these little Department projects but they see a way to actually move faster and save money I think it is a mix of those two they're looking to expand it which can be a nightmare for sales Cycles because they take longer but people are looking well why don't you leverage this and go across division so we do see people trying to leverage it because they're still I don't think digital transformation is slowing down but a lot more to be honest a lot more approvals at this point for everything it is you know Adam and another great quote in his in his keynote he said if you want to save money the Cloud's a place to do it absolutely and I read an article recently and I was looking through and I said this is the first time you know AWS has ever seen a downturn because the cloud was too early back then I'm like you weren't paying attention in 2008 because that was the first major inflection point for cloud adoption where CFO said okay stop the capex we're going to Opex and you saw the cloud take off and then 2010 started this you know amazing cycle that we really haven't seen anything like it where they were doubling down in Investments and they were real hardcore investment it wasn't like 1998 99 was all just going out the door for no clear reason yeah so that Foundation is now in place and I think it makes a lot of sense and it could be here for for a while where people are saying Hey I want to optimize and I'm going to do that on the cloud yeah no I mean I've obviously I certainly agree with Adam's quote I think really that's been in aws's DNA from from day one right is that ability to scale costs with with the actual consumption and paying for what you use and I think that you know certainly moments like now are ones that can really motivate change in an organization in a way that might not have been as palatable when it just it didn't feel like it was as necessary yeah all right we got to go give you a last word uh I think it's been a great event I love all your announcements I think this is wonderful uh it's been a great show I love uh in fact how many people are here at reinvent north of 50 000. yeah I mean I feel like it was it's as big if not bigger than 2019. people have said ah 2019 was a record when you count out all the professors I don't know it feels it feels as big if not bigger so there's great energy yeah it's quite amazing and uh and we're thrilled to be part of it guys thanks for coming on thecube again really appreciate it face to face all right thank you for watching this is Dave vellante for the cube your leader in Enterprise and emerging Tech coverage we'll be right back foreign

(upbeat music) >> Welcome back to Boston, everybody. This is the birthplace of theCUBE. In 2010, May of 2010 at EMC World, right in this very venue, John Furrier called it the chowder and lobster post. I'm Dave Vellante. We're here at RE:INFORCE 2022, Ed Walsh, CEO of ChaosSearch. Doing a drive by Ed. Thanks so much for stopping in. You're going to help me wrap up in our final editorial segment. >> Looking forward to it. >> I really appreciate it. >> Thank you for including me. >> How about that? 2010. >> That's amazing. It was really in this-- >> Really in this building. Yeah, we had to sort of bury our way in, tunnel our way into the Blogger Lounge. We did four days. >> Weekends, yeah. >> It was epic. It was really epic. But I'm glad they're back in Boston. AWS was going to do June in Houston. >> Okay. >> Which would've been awful. >> Yeah, yeah. No, this is perfect. >> Yeah. Thank God they came back. You saw Boston in summer is great. I know it's been hot, And of course you and I are from this area. >> Yeah. >> So how you been? What's going on? I mean, it's a little crazy out there. The stock market's going crazy. >> Sure. >> Having the tech lash, what are you seeing? >> So it's an interesting time. So I ran a company in 2008. So we've been through this before. By the way, the world's not ending, we'll get through this. But it is an interesting conversation as an investor, but also even the customers. There's some hesitation but you have to basically have the right value prop, otherwise things are going to get sold. So we are seeing longer sales cycles. But it's nothing that you can't overcome. But it has to be something not nice to have, has to be a need to have. But I think we all get through it. And then there is some, on the VC side, it's now buckle down, let's figure out what to do which is always a challenge for startup plans. >> In pre 2000 you, maybe you weren't a CEO but you were definitely an executive. And so now it's different and a lot of younger people haven't seen this. You've got interest rates now rising. Okay, we've seen that before but it looks like you've got inflation, you got interest rates rising. >> Yep. >> The consumer spending patterns are changing. You had 6$, $7 gas at one point. So you have these weird crosscurrents, >> Yup. >> And people are thinking, "Okay post-September now, maybe because of the recession, the Fed won't have to keep raising interest rates and tightening. But I don't know what to root for. It's like half full, half empty. (Ed laughing) >> But we haven't been in an environment with high inflation. At least not in my career. >> Right. Right. >> I mean, I got into 92, like that was long gone, right?. >> Yeah. >> So it is a interesting regime change that we're going to have to deal with, but there's a lot of analogies between 2008 and now that you still have to work through too, right?. So, anyway, I don't think the world's ending. I do think you have to run a tight shop. So I think the grow all costs is gone. I do think discipline's back in which, for most of us, discipline never left, right?. So, to me that's the name of the game. >> What do you tell just generally, I mean you've been the CEO of a lot of private companies. And of course one of the things that you do to retain people and attract people is you give 'em stock and it's great and everybody's excited. >> Yeah. >> I'm sure they're excited cause you guys are a rocket ship. But so what's the message now that, Okay the market's down, valuations are down, the trees don't grow to the moon, we all know that. But what are you telling your people? What's their reaction? How do you keep 'em motivated? >> So like anything, you want over communicate during these times. So I actually over communicate, you get all these you know, the Sequoia decks, 2008 and the recent... >> (chuckles) Rest in peace good times, that one right? >> I literally share it. Why? It's like, Hey, this is what's going on in the real world. It's going to affect us. It has almost nothing to do with us specifically, but it will affect us. Now we can't not pay attention to it. It does change how you're going to raise money, so you got to make sure you have the right runway to be there. So it does change what you do, but I think you over communicate. So that's what I've been doing and I think it's more like a student of the game, so I try to share it, and I say some appreciate it others, I'm just saying, this is normal, we'll get through this and this is what happened in 2008 and trust me, once the market hits bottom, give it another month afterwards. Then everyone says, oh, the bottom's in and we're back to business. Valuations don't go immediately back up, but right now, no one knows where the bottom is and that's where kind of the world's ending type of things. >> Well, it's interesting because you talked about, I said rest in peace good times >> Yeah >> that was the Sequoia deck, and the message was tighten up. Okay, and I'm not saying you shouldn't tighten up now, but the difference is, there was this period of two years of easy money and even before that, it was pretty easy money. >> Yeah. >> And so companies are well capitalized, they have runway so it's like, okay, I was talking to Frank Slootman about this now of course there are public companies, like we're not taking the foot off the gas. We're inherently profitable, >> Yeah. >> we're growing like crazy, we're going for it. You know? So that's a little bit of a different dynamic. There's a lot of good runway out there, isn't there? >> But also you look at the different companies that were either born or were able to power through those environments are actually better off. You come out stronger in a more dominant position. So Frank, listen, if you see what Frank's done, it's been unbelievable to watch his career, right?. In fact, he was at Data Domain, I was Avamar so, but look at what he's done since, he's crushed it. Right? >> Yeah. >> So for him to say, Hey, I'm going to literally hit the gas and keep going. I think that's the right thing for Snowflake and a right thing for a lot of people. But for people in different roles, I literally say that you have to take it seriously. What you can't be is, well, Frank's in a different situation. What is it...? How many billion does he have in the bank? So it's... >> He's over a billion, you know, over a billion. Well, you're on your way Ed. >> No, no, no, it's good. (Dave chuckles) Okay, I want to ask you about this concept that we've sort of we coined this term called Supercloud. >> Sure. >> You could think of it as the next generation of multi-cloud. The basic premises that multi-cloud was largely a symptom of multi-vendor. Okay. I've done some M&A, I've got some Shadow IT, spinning up, you know, Shadow clouds, projects. But it really wasn't a strategy to have a continuum across clouds. And now we're starting to see ecosystems really build, you know, you've used the term before, standing on the shoulders of giants, you've used that a lot. >> Yep. >> And so we're seeing that. Jerry Chen wrote a seminal piece on Castles in The Cloud, so we coined this term SuperCloud to connote this abstraction layer that hides the underlying complexities and primitives of the individual clouds and then adds value on top of it and can adjudicate and manage, irrespective of physical location, Supercloud. >> Yeah. >> Okay. What do you think about that concept?. How does it maybe relate to some of the things that you're seeing in the industry? >> So, standing on shoulders of giants, right? So I always like to do hard tech either at big company, small companies. So we're probably your definition of a Supercloud. We had a big vision, how to literally solve the core challenge of analytics at scale. How are you going to do that? You're not going to build on your own. So literally we're leveraging the primitives, everything you can get out of the Amazon cloud, everything get out of Google cloud. In fact, we're even looking at what it can get out of this Snowflake cloud, and how do we abstract that out, add value to it? That's where all our patents are. But it becomes a simplified approach. The customers don't care. Well, they care where their data is. But they don't care how you got there, they just want to know the end result. So you simplify, but you gain the advantages. One thing's interesting is, in this particular company, ChaosSearch, people try to always say, at some point the sales cycle they say, no way, hold on, no way that can be fast no way, or whatever the different issue. And initially we used to try to explain our technology, and I would say 60% was explaining the public, cloud capabilities and then how we, harvest those I guess, make them better add value on top and what you're able to get is something you couldn't get from the public clouds themselves and then how we did that across public clouds and then extracted it. So if you think about that like, it's the Shoulders of giants. But what we now do, literally to avoid that conversation because it became a lengthy conversation. So, how do you have a platform for analytics that you can't possibly overwhelm for ingest. All your messy data, no pipelines. Well, you leverage things like S3 and EC2, and you do the different security things. You can go to environments say, you can't possibly overrun me, I could not say that. If I didn't literally build on the shoulders giants of all these public clouds. But the value. So if you're going to do hard tech as a startup, you're going to build, you're going to be the principles of Supercloud. Maybe they're not the same size of Supercloud just looking at Snowflake, but basically, you're going to leverage all that, you abstract it out and that's where you're able to have a lot of values at that. >> So let me ask you, so I don't know if there's a strict definition of Supercloud, We sort of put it out to the community and said, help us define it. So you got to span multiple clouds. It's not just running in each cloud. There's a metadata layer that kind of understands where you're pulling data from. Like you said you can pull data from Snowflake, it sounds like we're not running on Snowflake, correct? >> No, complimentary to them in their different customers. >> Yeah. Okay. >> They want to build on top of a data platform, data apps. >> Right. And of course they're going cross cloud. >> Right. >> Is there a PaaS layer in there? We've said there's probably a Super PaaS layer. You're probably not doing that, but you're allowing people to bring their own, bring your own PaaS sort of thing maybe. >> So we're a little bit different but basically we publish open APIs. We don't have a user interface. We say, keep the user interface. Again, we're solving the challenge of analytics at scale, we're not trying to retrain your analytics, either analysts or your DevOps or your SOV or your Secop team. They use the tools they already use. Elastic search APIs, SQL APIs. So really they program, they build applications on top of us, Equifax is a good example. Case said it coming out later on this week, after 18 months in production but, basically they're building, we provide the abstraction layer, the quote, I'm going to kill it, Jeff Tincher, who owns all of SREs worldwide, said to the effect of, Hey I'm able to rethink what I do for my data pipelines. But then he also talked about how, that he really doesn't have to worry about the data he puts in it. We deal with that. And he just has to, just query on the other side. That simplicity. We couldn't have done that without that. So anyway, what I like about the definition is, if you were going to do something harder in the world, why would you try to rebuild what Amazon, Google and Azure or Snowflake did? You're going to add things on top. We can still do intellectual property. We're still doing patents. So five grand patents all in this. But literally the abstraction layer is the simplification. The end users do not want to know that complexity, even though they ask the questions. >> And I think too, the other attribute is it's ecosystem enablement. Whereas I think, >> Absolutely >> in general, in the Multicloud 1.0 era, the ecosystem wasn't thinking about, okay, how do I build on top and abstract that. So maybe it is Multicloud 2.0, We chose to use Supercloud. So I'm wondering, we're at the security conference, >> RE: INFORCE is there a security Supercloud? Maybe Snyk has the developer Supercloud or maybe Okta has the identity Supercloud. I think CrowdStrike maybe not. Cause CrowdStrike competes with Microsoft. So maybe, because Microsoft, what's interesting, Merritt Bear was just saying, look, we don't show up in the spending data for security because we're not charging for most of our security. We're not trying to make a big business. So that's kind of interesting, but is there a potential for the security Supercloud? >> So, I think so. But also, I'll give you one thing I talked to, just today, at least three different conversations where everyone wants to log data. It's a little bit specific to us, but basically they want to do the security data lake. The idea of, and Snowflake talks about this too. But the idea of putting all the data in one repository and then how do you abstract out and get value from it? Maybe not the perfect, but it becomes simple to do but hard to get value out. So the different players are going to do that. That's what we do. We're able to, once you land it in your S3 or it doesn't matter, cloud of choice, simple storage, we allow you to get after that data, but we take the primitives and hide them from you. And all you do is query the data and we're spinning up stateless computer to go after it. So then if I look around the floor. There's going to be a bunch of these players. I don't think, why would someone in this floor try to recreate what Amazon or Google or Azure had. They're going to build on top of it. And now the key thing is, do you leave it in standard? And now we're open APIs. People are building on top of my open APIs or do you try to put 'em in a walled garden? And they're in, now your Supercloud. Our belief is, part of it is, it needs to be open access and let you go after it. >> Well. And build your applications on top of it openly. >> They come back to snowflake. That's what Snowflake's doing. And they're basically saying, Hey come into our proprietary environment. And the benefit is, and I think both can win. There's a big market. >> I agree. But I think the benefit of Snowflake's is, okay, we're going to have federated governance, we're going to have data sharing, you're going to have access to all the ecosystem players. >> Yep. >> And as everything's going to be controlled and you know what you're getting. The flip side of that is, Databricks is the other end >> Yeah. >> of that spectrum, which is no, no, you got to be open. >> Yeah. >> So what's going to happen, well what's happening clearly, is Snowflake's saying, okay we've got Snowpark. we're going to allow Python, we're going to have an Apache Iceberg. We're going to have open source tooling that you can access. By the way, it's not going to be as good as our waled garden where the flip side of that is you get Databricks coming at it from a data science and data engineering perspective. And there's a lot of gaps in between, aren't there? >> And I think they both win. Like for instance, so we didn't do Snowpark integration. But we work with people building data apps on top of Snowflake or data bricks. And what we do is, we can add value to that, or what we've done, again, using all the Supercloud stuff we're done. But we deal with the unstructured data, the four V's coming at you. You can't pipeline that to save. So we actually could be additive. As they're trying to do like a security data cloud inside of Snowflake or do the same thing in Databricks. That's where we can play. Now, we play with them at the application level that they get some data from them and some data for us. But I believe there's a partnership there that will do it inside their environment. To us they're just another large scaler environment that my customers want to get after data. And they want me to abstract it out and give value. >> So it's another repository to you. >> Yeah. >> Okay. So I think Snowflake recently added support for unstructured data. You chose not to do Snowpark because why? >> Well, so the way they're doing the unstructured data is not bad. It's JSON data. Basically, This is the dilemma. Everyone wants their application developers to be flexible, move fast, securely but just productivity. So you get, give 'em flexibility. The problem with that is analytics on the end want to be structured to be performant. And this is where Snowflake, they have to somehow get that raw data. And it's changing every day because you just let the developers do what they want now, in some structured base, but do what you need to do your business fast and securely. So it completely destroys. So they have large customers trying to do big integrations for this messy data. And it doesn't quite work, cause you literally just can't make the pipelines work. So that's where we're complimentary do it. So now, the particular integration wasn't, we need a little bit deeper integration to do that. So we're integrating, actually, at the data app layer. But we could, see us and I don't, listen. I think Snowflake's a good actor. They're trying to figure out what's best for the customers. And I think we just participate in that. >> Yeah. And I think they're trying to figure out >> Yeah. >> how to grow their ecosystem. Because they know they can't do it all, in fact, >> And we solve the key thing, they just can't do certain things. And we do that well. Yeah, I have SQL but that's where it ends. >> Yeah. >> I do the messy data and how to play with them. >> And when you talk to one of their founders, anyway, Benoit, he comes on the cube and he's like, we start with simple. >> Yeah. >> It reminds me of the guy's some Pure Storage, that guy Coz, he's always like, no, if it starts to get too complicated. So that's why they said all right, we're not going to start out trying to figure out how to do complex joins and workload management. And they turn that into a feature. So like you say, I think both can win. It's a big market. >> I think it's a good model. And I love to see Frank, you know, move. >> Yeah. I forgot So you AVMAR... >> In the day. >> You guys used to hate each other, right? >> No, no, no >> No. I mean, it's all good. >> But the thing is, look what he's done. Like I wouldn't bet against Frank. I think it's a good message. You can see clients trying to do it. Same thing with Databricks, same thing with BigQuery. We get a lot of same dynamic in BigQuery. It's good for a lot of things, but it's not everything you need to do. And there's ways for the ecosystem to play together. >> Well, what's interesting about BigQuery is, it is truly cloud native, as is Snowflake. You know, whereas Amazon Redshift was sort of Parexel, it's cobbled together now. It's great engineering, but BigQuery gets a lot of high marks. But again, there's limitations to everything. That's why companies like yours can exist. >> And that's why.. so back to the Supercloud. It allows me as a company to participate in that because I'm leveraging all the underlying pieces. Which we couldn't be doing what we're doing now, without leveraging the Supercloud concepts right, so... >> Ed, I really appreciate you coming by, help me wrap up today in RE:INFORCE. Always a pleasure seeing you, my friend. >> Thank you. >> All right. Okay, this is a wrap on day one. We'll be back tomorrow. I'll be solo. John Furrier had to fly out but we'll be following what he's doing. This is RE:INFORCE 2022. You're watching theCUBE. I'll see you tomorrow.

(upbeat music) >> My name's Courtney Pallota, I'm the Vice President of Marketing at ChaosSearch. We've partnered with theCUBE team to take every one of those assets, tailor them to meet whatever our needs were, and get them out and shared far and wide. And theCUBE team has been tremendously helpful in partnering with us to make that a success. >> theCUBE has been fantastic with us. They are thought leaders in this space. And we have a unique product, a unique vision, and they have an insight into where the market's going. They've had conference with us with data mesh, and how do we fit into that new realm of data access. And with our unique vision, with our unique platform, and with theCUBE, we've uniquely come out into the market. >> What's my overall experience with theCUBE? Would I do it again, would I recommended it to others? I said, I recommend theCUBE to everyone. In fact, I was at IBM, and some of the IBM executives didn't want to go on theCUBE because it's a live interview. Live interviews can be traumatic. But the fact of the matter is, one, yeah, they're tough questions, but they're in line, they're what clients are looking for. So yes, you have to be on ball. I mean, you're always on your toes, but you get your message out so crisply. So I recommend it to everyone. I've gotten a lot of other executives to participate, and they've all had a great example. You have to be ready. I mean, you can't go on theCUBE and not be ready, but now you can get your message out. And it has such a good distribution. I can't think of a better platform. So I recommended it to everyone. If I say ChaosSearch in one word, I'd say digital transformation, with a hyphen.

>> Welcome to theCUBE, I am Dave Vellante. And today we're going to explore the ebb and flow of data as it travels into the cloud and the data lake. The concept of data lakes was alluring when it was first coined last decade by CTO James Dixon. Rather than be limited to highly structured and curated data that lives in a relational database in the form of an expensive and rigid data warehouse or a data mart. A data lake is formed by flowing data from a variety of sources into a scalable repository, like, say an S3 bucket that anyone can access, dive into, they can extract water, A.K.A data, from that lake and analyze data that's much more fine-grained and less expensive to store at scale. The problem became that organizations started to dump everything into their data lakes with no schema on our right, no metadata, no context, just shoving it into the data lake and figure out what's valuable at some point down the road. Kind of reminds you of your attic, right? Except this is an attic in the cloud. So it's too big to clean out over a weekend. Well look, it's 2021 and we should be solving this problem by now. A lot of folks are working on this, but often the solutions add other complexities for technology pros. So to understand this better, we're going to enlist the help of ChaosSearch CEO Ed Walsh, and Thomas Hazel, the CTO and Founder of ChaosSearch. We're also going to speak with Kevin Miller who's the Vice President and General Manager of S3 at Amazon web services. And of course they manage the largest and deepest data lakes on the planet. And we'll hear from a customer to get their perspective on this problem and how to go about solving it, but let's get started. Ed, Thomas, great to see you. Thanks for coming on theCUBE. >> Likewise. >> Face to face, it's really good to be here. >> It is nice face to face. >> It's great. >> So, Ed, let me start with you. We've been talking about data lakes in the cloud forever. Why is it still so difficult to extract value from those data lakes? >> Good question. I mean, data analytics at scale has always been a challenge, right? So, we're making some incremental changes. As you mentioned that we need to see some step function changes. But in fact, it's the reason ChaosSearch was really founded. But if you look at it, the same challenge around data warehouse or a data lake. Really it's not just to flowing the data in, it's how to get insights out. So it kind of falls into a couple of areas, but the business side will always complain and it's kind of uniform across everything in data lakes, everything in data warehousing. They'll say, "Hey, listen, I typically have to deal with a centralized team to do that data prep because it's data scientists and DBAs". Most of the time, they're a centralized group. Sometimes they're are business units, but most of the time, because they're scarce resources together. And then it takes a lot of time. It's arduous, it's complicated, it's a rigid process of the deal of the team, hard to add new data, but also it's hard to, it's very hard to share data and there's no way to governance without locking it down. And of course they would be more self-serve. So there's, you hear from the business side constantly now underneath is like, there's some real technology issues that we haven't really changed the way we're doing data prep since the two thousands, right? So if you look at it, it's, it falls two big areas. It's one, how to do data prep. How do you take, a request comes in from a business unit. I want to do X, Y, Z with this data. I want to use this type of tool sets to do the following. Someone has to be smart, how to put that data in the right schema, you mentioned. You have to put it in the right format, that the tool sets can analyze that data before you do anything. And then second thing, I'll come back to that 'cause that's the biggest challenge. But the second challenge is how these different data lakes and data warehouses are now persisting data and the complexity of managing that data and also the cost of computing it. And I'll go through that. But basically the biggest thing is actually getting it from raw data so the rigidness and complexity that the business sides are using it is literally someone has to do this ETL process, extract, transform, load. They're actually taking data, a request comes in, I need so much data in this type of way to put together. They're literally physically duplicating data and putting it together on a schema. They're stitching together almost a data puddle for all these different requests. And what happens is anytime they have to do that, someone has to do it. And it's, very skilled resources are scanned in the enterprise, right? So it's a DBS and data scientists. And then when they want new data, you give them a set of data set. They're always saying, what can I add to this data? Now that I've seen the reports. I want to add this data more fresh. And the same process has to happen. This takes about 60% to 80% of the data scientists in DPA's to do this work. It's kind of well-documented. And this is what actually stops the process. That's what is rigid. They have to be rigid because there's a process around that. That's the biggest challenge of doing this. And it takes an enterprise, weeks or months. I always say three weeks or three months. And no one challenges beyond that. It also takes the same skill set of people that you want to drive digital transformation, data warehousing initiatives, motorization, being data driven or all these data scientists and DBS they don't have enough of. So this is not only hurting you getting insights out of your day like in the warehouses. It's also, this resource constraint is hurting you actually getting. >> So that smallest atomic unit is that team, that's super specialized team, right? >> Right. >> Yeah. Okay. So you guys talk about activating the data lake. >> Yep. >> For analytics. What's unique about that? What problems are you all solving? You know, when you guys crew created this magic sauce. >> No, and basically, there's a lot of things. I highlighted the biggest one is how to do the data prep, but also you're persisting and using the data. But in the end, it's like, there's a lot of challenges at how to get analytics at scale. And this is really where Thomas and I founded the team to go after this, but I'll try to say it simply. What we're doing, I'll try to compare and contrast what we do compared to what you do with maybe an elastic cluster or a BI cluster. And if you look at it, what we do is we simply put your data in S3, don't move it, don't transform it. In fact, we're against data movement. What we do is we literally point and set that data and we index that data and make it available in a data representation that you can give virtual views to end-users. And those virtual views are available immediately over petabytes of data. And it actually gets presented to the end-user as an open API. So if you're elastic search user, you can use all your elastic search tools on this view. If you're a SQL user, Tableau, Looker, all the different tools, same thing with machine learning next year. So what we do is we take it, make it very simple. Simply put it there. It's already there already. Point us at it. We do the hard of indexing and making available. And then you publish in the open API as your users can use exactly what they do today. So that's, dramatically I'll give you a before and after. So let's say you're doing elastic search. You're doing logging analytics at scale, they're lending their data in S3. And then they're ETL physically duplicating and moving data. And typically deleting a lot of data to get in a format that elastic search can use. They're persisting it up in a data layer called leucine. It's physically sitting in memories, CPU, SSDs, and it's not one of them, it's a bunch of those. They in the cloud, you have to set them up because they're persisting ECC. They stand up same by 24, not a very cost-effective way to the cloud computing. What we do in comparison to that is literally pointing it at the same S3. In fact, you can run a complete parallel, the data necessary it's being ETL out. When just one more use case read only, or allow you to get that data and make this virtual views. So we run a complete parallel, but what happens is we just give a virtual view to the end users. We don't need this persistence layer, this extra cost layer, this extra time, cost and complexity of doing that. So what happens is when you look at what happens in elastic, they have a constraint, a trade-off of how much you can keep and how much you can afford to keep. And also it becomes unstable at time because you have to build out a schema. It's on a server, the more the schema scales out, guess what? you have to add more servers, very expensive. They're up seven by 24. And also they become brutalized. You lose one node, the whole thing has to be put together. We have none of that cost and complexity. We literally go from to keep whatever you want, whatever you want to keep an S3 is single persistence, very cost effective. And what we are able to do is, costs, we save 50 to 80%. Why? We don't go with the old paradigm of sit it up on servers, spin them up for persistence and keep them up 7 by 24. We're literally asking their cluster, what do you want to cut? We bring up the right compute resources. And then we release those sources after the query done. So we can do some queries that they can't imagine at scale, but we're able to do the exact same query at 50 to 80% savings. And they don't have to do any tutorial of moving that data or managing that layer of persistence, which is not only expensive, it becomes brittle. And then it becomes, I'll be quick. Once you go to BI, it's the same challenge, but the BI systems, the requests are constant coming at from a business unit down to the centralized data team. Give me this flavor of data. I want to use this piece of, you know, this analytic tool in that desk set. So they have to do all this pipeline. They're constantly saying, okay, I'll give you this data, this data, I'm duplicating that data, I'm moving it and stitching it together. And then the minute you want more data, they do the same process all over. We completely eliminate that. >> And those requests are queue up. Thomas, it had me, you don't have to move the data. That's kind of the exciting piece here, isn't it? >> Absolutely no. I think, you know, the data lake philosophy has always been solid, right? The problem is we had that Hadoop hang over, right? Where let's say we were using that platform, little too many variety of ways. And so, I always believed in data lake philosophy when James came and coined that I'm like, that's it. However, HTFS, that wasn't really a service. Cloud object storage is a service that the elasticity, the security, the durability, all that benefits are really why we founded on-cloud storage as a first move. >> So it was talking Thomas about, you know, being able to shut off essentially the compute so you don't have to keep paying for it, but there's other vendors out there and stuff like that. Something similar as separating, compute from storage that they're famous for that. And you have Databricks out there doing their lake house thing. Do you compete with those? How do you participate and how do you differentiate? >> Well, you know you've heard this term data lakes, warehouse, now lake house. And so what everybody wants is simple in, easy in, however, the problem with data lakes was complexity of out. Driving value. And I said, what if, what if you have the easy in and the value out? So if you look at, say snowflake as a warehousing solution, you have to all that prep and data movement to get into that system. And that it's rigid static. Now, Databricks, now that lake house has exact same thing. Now, should they have a data lake philosophy, but their data ingestion is not data lake philosophy. So I said, what if we had that simple in with a unique architecture and indexed technology, make it virtually accessible, publishable dynamically at petabyte scale. And so our service connects to the customer's cloud storage. Data stream the data in, set up what we call a live indexing stream, and then go to our data refinery and publish views that can be consumed the elastic API, use cabana Grafana, or say SQL tables look or say Tableau. And so we're getting the benefits of both sides, use scheme on read-write performance with scheme write-read performance. And if you can do that, that's the true promise of a data lake, you know, again, nothing against Hadoop, but scheme on read with all that complexity of software was a little data swamping. >> Well, you've got to start it, okay. So we got to give them a good prompt, but everybody I talked to has got this big bunch of spark clusters, now saying, all right, this doesn't scale, we're stuck. And so, you know, I'm a big fan of Jamag Dagani and our concept of the data lake and it's early days. But if you fast forward to the end of the decade, you know, what do you see as being the sort of critical components of this notion of, people call it data mesh, but to get the analytics stack, you're a visionary Thomas, how do you see this thing playing out over the next decade? >> I love her thought leadership, to be honest, our core principles were her core principles now, 5, 6, 7 years ago. And so this idea of, decentralize that data as a product, self-serve and, and federated computer governance, I mean, all that was our core principle. The trick is how do you enable that mesh philosophy? I can say we're a mesh ready, meaning that, we can participate in a way that very few products can participate. If there's gates data into your system, the CTL, the schema management, my argument with the data meshes like producers and consumers have the same rights. I want the consumer, people that choose how they want to consume that data. As well as the producer, publishing it. I can say our data refinery is that answer. You know, shoot, I'd love to open up a standard, right? Where we can really talk about the producers and consumers and the rights each others have. But I think she's right on the philosophy. I think as products mature in this cloud, in this data lake capabilities, the trick is those gates. If you have to structure up front, if you set those pipelines, the chance of you getting your data into a mesh is the weeks and months that Ed was mentioning. >> Well, I think you're right. I think the problem with data mesh today is the lack of standards you've got. You know, when you draw the conceptual diagrams, you've got a lot of lollipops, which are APIs, but they're all unique primitives. So there aren't standards, by which to your point, the consumer can take the data the way he or she wants it and build their own data products without having to tap people on the shoulder to say, how can I use this?, where does the data live? And being able to add their own data. >> You're exactly right. So I'm an organization, I'm generating data, when the courageously stream it into a lake. And then the service, a ChaosSearch service, is the data is discoverable and configurable by the consumer. Let's say you want to go to the corner store. I want to make a certain meal tonight. I want to pick and choose what I want, how I want it. Imagine if the data mesh truly can have that producer of information, you know, all the things you can buy a grocery store and what you want to make for dinner. And if you'd static, if you call up your producer to do the change, was it really a data mesh enabled service? I would argue not. >> Ed, bring us home. >> Well, maybe one more thing with this. >> Please, yeah. 'Cause some of this is we're talking 2031, but largely these principles are what we have in production today, right? So even the self service where you can actually have a business context on top of a data lake, we do that today, we talked about, we get rid of the physical ETL, which is 80% of the work, but the last 20% it's done by this refinery where you can do virtual views, the right or back and do all the transformation need and make it available. But also that's available to, you can actually give that as a role-based access service to your end-users, actually analysts. And you don't want to be a data scientist or DBA. In the hands of a data scientist the DBA is powerful, but the fact of matter, you don't have to affect all of our employees, regardless of seniority, if they're in finance or in sales, they actually go through and learn how to do this. So you don't have to be it. So part of that, and they can come up with their own view, which that's one of the things about data lakes. The business unit wants to do themselves, but more importantly, because they have that context of what they're trying to do instead of queuing up the very specific request that takes weeks, they're able to do it themselves. >> And if I have to put it on different data stores and ETL that I can do things in real time or near real time. And that's game changing and something we haven't been able to do ever. >> And then maybe just to wrap it up, listen, you know 8 years ago, Thomas and his group of founders, came up with the concept. How do you actually get after analytics at scale and solve the real problems? And it's not one thing, it's not just getting S3. It's all these different things. And what we have in market today is the ability to literally just simply stream it to S3, by the way, simply do, what we do is automate the process of getting the data in a representation that you can now share an augment. And then we publish open API. So can actually use a tool as you want, first use case log analytics, hey, it's easy to just stream your logs in. And we give you elastic search type of services. Same thing that with CQL, you'll see mainstream machine learning next year. So listen, I think we have the data lake, you know, 3.0 now, and we're just stretching our legs right now to have fun. >> Well, and you have to say it log analytics. But if I really do believe in this concept of building data products and data services, because I want to sell them, I want to monetize them and being able to do that quickly and easily, so I can consume them as the future. So guys, thanks so much for coming on the program. Really appreciate it.

>>Hi, Brian, this is Dave Volante. Welcome to this cube conversation with Thomas Hazel was the founder and CTO of chaos surgeon. I'm also joined by ed Walsh. Who's the CEO Thomas. Good to see you. >>Great to be here. >>Explain Jason. First of all, what >>Jason, Jason has a powerful data representation, a data source. Uh, but let's just say that we try to drive value out of it. It gets complicated. Uh, I can search. We activate customers, data lakes. So, you know, customers stream their Jason data to this, uh, cloud stores that we activate. Now, the trick is the complexity of a Jason data structure. You can do all these complexity of representation. Now here's the problem putting that representation into a elastic search database or relational databases, very problematic. So what people choose to do is they pick and choose what they want and or they just stored as a blob. And so I said, what if, what if we create a new index technology that could store it as a full representation, but dynamically in a, we call our data refinery published access to all the permutations that you may want, where if you do a full on flatten, your flattening of its Jason, one row theoretically could be put into a million rows and relational data sort of explode, >>But then it gets really expensive. But so, but everybody says they have Jason support, every database vendor that I talked to, it's a big announcement. We now support Jason. What's the deal. >>Exactly. So you take your relational database with all those relational constructs and you have a proprietary Jason API to pick and choose. So instead of picking, choosing upfront, now you're picking, choosing in the backend where you really want us the power of the relational analysis of that Jaison data. And that's where chaos comes in, where we expand those data streams we do in a relational way. So all that tooling you've been built to know and love. Now you can access to it. So if you're doing proprietary APIs or Jason data, you're not using Looker, you're not using Tableau. You're doing some type of proprietary, probably emailing now on the backend. >>Okay. So you say all the tools that you've trained, everybody on you can't really use them. You got to build some custom stuff and okay, so, so, so maybe bring that home then in terms of what what's the money, why do the suits care about this stuff? >>The reason this is so important is think about anything, cloud native Kubernetes, your different applications. What you're doing in Mongo is all Jason is it's very powerful but painful, but if you're not keeping the data, what people are doing a data scientist is, or they're just doing leveling, they're saying I'm going to only keep the first four things. So think about it's Kubernetes, it's your app logs. They're trying to figure out for black Friday, what happens? It's Lilly saying, Hey, every minute they'll cut a new log. You're able to say, listen, these are the users that were in that system for an hour. And here's a different things. They do. The fact of the matter is if you cut it off, you lose all that fidelity, all that data. So it's really important that to have. So if you're trying to figure out either what happened for security, what happened for on a performance, or if you're trying to figure out, Hey, I'm VP of product or growth, how do I cross sell things? >>You need to know what everyone's doing. If you're not handling Jason natively, like we're doing either your, it keeps on expanding on black Friday. All of a sudden the logs get huge. And the next day it's not, but it's really powerful data that you need to harness for business values. It's, what's going to drive growth. It's what's going to do the digital transformation. So without the technology, you're kind of blind. And to be honest, you don't know. Cause a data scientist is kind of deleted the data on you. So this is big for the business and digital transformation, but also it was such a pain. The data scientists in DBS were forced to just basically make it simple. So it didn't blow up their system. We allow them to keep it simple, but yes, >>Both power. It reminds me if you like, go on vacation, you got your video camera. Somebody breaks into your house. You go back to Lucas and see who and that the data's gone. The video's gone because it didn't, you didn't, you weren't able to save it cause it's too >>Expensive. Well, it's funny. This is the first day source. That's driving the design of the database because of all the value we should be designed the database around the information. It stores not the structure and how it's been organized. And so our viewpoint is you get to choose your structure yet contain all that content. So if a vendor >>It says to kind of, I'm a customer then says, Hey, we got Jason support. What questions should I ask to really peel the onion? >>Well, particularly relational. Is it a relational access to that data? Now you could say, oh, I've ETL does Jason into it. But chances are the explosion of Jason permutations of one row to a million. They're probably not doing the full representation. So from our viewpoint is either you're doing a blob type access to proprietary Jason APIs or you're picking and choosing those, the choices say that is the market thought. However, what if you could take all the vegetation and design your schema based on how you want to consume it versus how you could store it. And that's a big difference with, >>So I should be asking how, how do I consume this data? Are you ETL? Bring it in how much data explosion is going to occur. Once I do this, and you're saying for chaos, search the answer to those questions. >>The answer is, again, our philosophy simply stream your data into your cloud object, storage, your data lake and with our index technology and our data refinery. You get to create views, dynamic the incident, whether it's a terabyte or petabyte, and describe how you want your data because consumed in a relational way or an elastic search way, both are consumable through our data refinery, which is >>For us. The refinery gives you the view. So what happens if someone wants a different view, I want to actually unpack different columns or different matrices. You able to do that in a virtual view, it's available immediately over petabytes of data. You don't have that episode where you come back, look at the video camera. There's no data there left. So that's, >>We do appreciate the time and the explanation on really understanding Jason. Thank you. All right. And thank you for watching this cube conversation. This is Dave Volante. We'll see you next time.

>>Welcome to the cube. I'm Dave Volante. Today, we're going to explore the ebb and flow of data as it travels into the cloud. In the data lake, the concept of data lakes was a Loring when it was first coined last decade by CTO James Dickson, rather than be limited to highly structured and curated data that lives in a relational database in the form of an expensive and rigid data warehouse or a data Mart, a data lake is formed by flowing data from a variety of sources into a scalable repository, like say an S3 bucket that anyone can access, dive into. They can extract water. It can a data from that lake and analyze data. That's much more fine-grained and less expensive to store at scale. The problem became that organizations started to dump everything into their data lakes with no schema on it, right? No metadata, no context to shove it into the data lake and figure out what's valuable. >>At some point down the road kind of reminds you of your attic, right? Except this is an attic in the cloud. So it's too big to clean out over a weekend. We'll look it's 2021 and we should be solving this problem by now, a lot of folks are working on this, but often the solutions at other complexities for technology pros. So to understand this better, we're going to enlist the help of chaos search CEO and Walsh and Thomas Hazel, the CTO and founder of chaos search. We're also going to speak with Kevin Miller. Who's the vice president and general manager of S3 at Amazon web services. And of course they manage the largest and deepest data lakes on the planet. And we'll hear from a customer to get their perspective on this problem and how to go about solving it, but let's get started. Ed Thomas. Great to see you. Thanks for coming on the cube. Likewise face. It's really good to be in this nice face. Great. So let me start with you. We've been talking about data lakes in the cloud forever. Why is it still so difficult to extract value from those data? >>Good question. I mean, a data analytics at scale is always been a challenge, right? So, and it's, uh, we're making some incremental changes. As you mentioned that we need to seem some step function changes, but, uh, in fact, it's the reason, uh, search was really founded. But if you look at it the same challenge around data warehouse or a data lake, really, it's not just a flowing the data in is how to get insights out. So it kind of falls into a couple of areas, but the business side will always complain and it's kind of uniform across everything in data lakes, everything that we're offering, they'll say, Hey, listen, I typically have to deal with a centralized team to do that data prep because it's data scientist and DBS. Most of the time they're a centralized group, sometimes are business units, but most of the time, because they're scarce resources together. >>And then it takes a lot of time. It's arduous, it's complicated. It's a rigid process of the deal of the team, hard to add new data. But also it's hard to, you know, it's very hard to share data and there's no way to governance without locking it down. And of course they would be more self-service. So there's you hear from the business side constantly now underneath is like, there's some real technology issues that we haven't really changed the way we're doing data prep since the two thousands. Right? So if you look at it, it's, it falls, uh, two big areas. It's one. How do data prep, how do you take a request comes in from a business unit. I want to do X, Y, Z with this data. I want to use this type of tool sets to do the following. Someone has to be smart, how to put that data in the right schema. >>You mentioned you have to put it in the right format, that the tool sets can analyze that data before you do anything. And then secondly, I'll come back to that because that's a biggest challenge. But the second challenge is how these different data lakes and data we're also going to persisting data and the complexity of managing that data and also the cost of computing. And I'll go through that. But basically the biggest thing is actually getting it from raw data so that the rigidness and complexity that the business sides are using it is literally someone has to do this ETL process extract, transform load. They're actually taking data request comes in. I need so much data in this type of way to put together their Lilly, physically duplicating data and putting it together and schema they're stitching together almost a data puddle for all these different requests. >>And what happens is anytime they have to do that, someone has to do it. And it's very skilled. Resources are scant in the enterprise, right? So it's a DBS and data scientists. And then when they want new data, you give them a set of data set. They're always saying, what can I add this data? Now that I've seen the reports, I want to add this data more fresh. And the same process has to happen. This takes about 60 to 80% of the data scientists in DPA's to do this work. It's kind of well-documented. Uh, and this is what actually stops the process. That's what is rigid. They have to be rigid because there's a process around that. Uh, that's the biggest challenge to doing this. And it takes in the enterprise, uh, weeks or months. I always say three weeks to three months. And no one challenges beyond that. It also takes the same skill set of people that you want to drive. Digital transformation, data, warehousing initiatives, uh, monitorization being, data driven, or all these data scientists and DBS. They don't have enough of, so this is not only hurting you getting insights out of your dead like that, or else it's also this resource constraints hurting you actually getting smaller. >>The Tomic unit is that team that's super specialized team. Right. Right. Yeah. Okay. So you guys talk about activating the data lake. Yep, sure. Analytics, what what's unique about that? What problems are you all solving? You know, when you guys crew created this, this, this magic sauce. >>No, and it basically, there's a lot of things I highlighted the biggest one is how to do the data prep, but also you're persisting and using the data. But in the end, it's like, there's a lot of challenges that how to get analytics at scale. And this is really where Thomas founded the team to go after this. But, um, I'll try to say it simply, what are we doing? I'll try to compare and stress what we do compared to what you do with maybe an elastic cluster or a BI cluster. Um, and if you look at it, what we do is we simply put your data in S3, don't move it, don't transform it. In fact, we're not we're against data movement. What we do is we literally pointed at that data and we index that data and make it available in a data representation that you can give virtual views to end users. >>And those virtual views are available immediately over petabytes of data. And it re it actually gets presented to the end user as an open API. So if you're elastic search user, you can use all your lesser search tools on this view. If you're a SQL user, Tableau, Looker, all the different tools, same thing with machine learning next year. So what we do is we take it, make it very simple. Simply put it there. It's already there already. Point is at it. We do the hard of indexing and making available. And then you publish in the open API as your users can use exactly what they do today. So that's dramatically. I'll give you a before and after. So let's say you're doing elastic search. You're doing logging analytics at scale, they're lending their data in S3. And then they're,, they're physically duplicating a moving data and typically deleting a lot of data to get in a format that elastic search can use. >>They're persisting it up in a data layer called leucine. It's physically sitting in memories, CPU, uh, uh, SSDs. And it's not one of them. It's a bunch of those. They in the cloud, you have to set them up because they're persisting ECC. They stand up semi by 24, not a very cost-effective way to the cloud, uh, cloud computing. What we do in comparison to that is literally pointing it at the same S3. In fact, you can run a complete parallel, the data necessary. It's being ETL. That we're just one more use case read only, or allow you to get that data and make this virtual views. So we run a complete parallel, but what happens is we just give a virtual view to the end users. We don't need this persistence layer, this extra cost layer, this extra, um, uh, time cost and complexity of doing that. >>So what happens is when you look at what happens in elastic, they have a constraint, a trade-off of how much you can keep and how much you can afford to keep. And also it becomes unstable at time because you have to build out a schema. It's on a server, the more the schema scales out, guess what you have to add more servers, very expensive. They're up seven by 24. And also they become brittle. As you lose one node. The whole thing has to be put together. We have none of that cost and complexity. We literally go from to keep whatever you want, whatever you want to keep an S3, a single persistence, very cost effective. And what we do is, um, costs. We save 50 to 80% why we don't go with the old paradigm of sit it up on servers, spin them up for persistence and keep them up. >>Somebody 24, we're literally asking her cluster, what do you want to cut? We bring up the right compute resources. And then we release those sources after the query done. So we can do some queries that they can't imagine at scale, but we're able to do the exact same query at 50 to 80% savings. And they don't have to do any of the toil of moving that data or managing that layer of persistence, which is not only expensive. It becomes brittle. And then it becomes an I'll be quick. Once you go to BI, it's the same challenge, but the BI systems, the requests are constant coming at from a business unit down to the centralized data team. Give me this flavor of debt. I want to use this piece of, you know, this analytic tool in that desk set. So they have to do all this pipeline. They're constantly saying, okay, I'll give you this data, this data I'm duplicating that data. I'm moving in stitching together. And then the minute you want more data, they do the same process all over. We completely eliminate that. >>The questions queue up, Thomas, it had me, you don't have to move the data. That's, that's kind of the >>Writing piece here. Isn't it? I absolutely, no. I think, you know, the daylight philosophy has always been solid, right? The problem is we had that who do hang over, right? Where let's say we were using that platform, little, too many variety of ways. And so I always believed in daily philosophy when James came and coined that I'm like, that's it. However, HTFS that wasn't really a service cloud. Oddish storage is a service that the, the last society, the security and the durability, all that benefits are really why we founded, uh, Oncotype storage as a first move. >>So it was talking Thomas about, you know, being able to shut off essentially the compute and you have to keep paying for it, but there's other vendors out there and stuff like that. Something similar as separating, compute from storage that they're famous for that. And, and, and yet Databricks out there doing their lake house thing. Do you compete with those? How do you participate and how do you differentiate? >>I know you've heard this term data lakes, warehouse now, lake house. And so what everybody wants is simple in easy N however, the problem with data lakes was complexity of out driving value. And I said, what if, what if you have the easy end and the value out? So if you look at, uh, say snowflake as a, as a warehousing solution, you have to all that prep and data movement to get into that system. And that it's rigid static. Now, Databricks, now that lake house has exact same thing. Now, should they have a data lake philosophy, but their data ingestion is not daily philosophy. So I said, what if we had that simple in with a unique architecture, indexed technology, make it virtually accessible publishable dynamically at petabyte scale. And so our service connects to the customer's cloud storage data, stream the data in set up what we call a live indexing stream, and then go to our data refinery and publish views that can be consumed the lasted API, use cabana Grafana, or say SQL tables look or say Tableau. And so we're getting the benefits of both sides, you know, schema on read, write performance with scheme on, right. Reperformance. And if you can do that, that's the true promise of a data lake, you know, again, nothing against Hadoop, but a schema on read with all that complexity of, uh, software was, uh, what was a little data, swamp >>Got to start it. Okay. So we got to give a good prompt, but everybody I talked to has got this big bunch of spark clusters now saying, all right, this, this doesn't scale we're stuck. And so, you know, I'm a big fan of and our concept of the data lake and it's it's early days. But if you fast forward to the end of the decade, you know, what do you see as being the sort of critical components of this notion of, you know, people call it data mesh, but you've got the analytics stack. Uh, you, you, you're a visionary Thomas, how do you see this thing playing out over the next? >>I love for thought leadership, to be honest, our core principles were her core principles now, you know, 5, 6, 7 years ago. And so this idea of, you know, de centralize that data as a product, you know, self-serve and, and federated, computer, uh, governance, I mean, all that, it was our core principle. The trick is how do you enable that mesh philosophy? We, I could say we're a mesh ready, meaning that, you know, we can participate in a way that very few products can participate. If there's gates data into your system, the CTLA, the schema management, my argument with the data meshes like producers and consumers have the same rights. I want the consumer people that choose how they want to consume that data, as well as the producer publishing it. I can say our data refinery is that answer. You know, shoot, I love to open up a standard, right, where we can really talk about the producers and consumers and the rights each others have. But I think she's right on the philosophy. I think as products mature in this cloud, in this data lake capabilities, the trick is those gates. If you have the structure up front, it gets at those pipelines. You know, the chance of you getting your data into a mesh is the weeks and months that it was mentioning. >>Well, I think you're right. I think the problem with, with data mesh today is the lack of standards you've got. You know, when you draw the conceptual diagrams, you've got a lot of lollipops, which are API APIs, but they're all, you know, unique primitives. So there aren't standards by which to your point, the consumer can take the data the way he or she wants it and build their own data products without having to tap people on the shoulder to say, how can I use this? Where's the data live and, and, and, and, and being able to add their own >>You're exactly right. So I'm an organization I'm generally data will be courageous, a stream it to a lake. And then the service, uh, Ks search service is the data's con uh, discoverable and configurable by the consumer. Let's say you want to go to the corner store? You know, I want to make a certain meal tonight. I want to pick and choose what I want, how I want it. Imagine if the data mesh truly can have that producer of information, you, all the things you can buy a grocery store and what you want to make for dinner. And if you'd static, if you call up your producer to do the change, was it really a data mesh enabled service? I would argue not that >>Bring us home >>Well. Uh, and, um, maybe one more thing with this, cause some of this is we talking 20, 31, but largely these principles are what we have in production today, right? So even the self service where you can actually have business context on top of a debt, like we do that today, we talked about, we get rid of the physical ETL, which is 80% of the work, but the last 20% it's done by this refinery where you can do virtual views, the right our back and do all the transformation need and make it available. But also that's available to, you can actually give that as a role-based access service to your end users actually analysts, and you don't want to be a data scientist or DBA in the hands of a data science. The DBA is powerful, but the fact of matter, you don't have to affect all of our employees, regardless of seniority. If they're in finance or in sales, they actually go through and learn how to do this. So you don't have to be it. So part of that, and they can come up with their own view, which that's one of the things about debt lakes, the business unit wants to do themselves, but more importantly, because they have that context of what they're trying to do instead of queuing up the very specific request that takes weeks, they're able to do it themselves and to find out that >>Different data stores and ETL that I can do things in real time or near real time. And that's that's game changing and something we haven't been able to do, um, ever. Hmm. >>And then maybe just to wrap it up, listen, um, you know, eight years ago is a group of founders came up with the concept. How do you actually get after analytics at scale and solve the real problems? And it's not one thing it's not just getting S3, it's all these different things. And what we have in market today is the ability to literally just simply stream it to S3 by the way, simply do what we do is automate the process of getting the data in a representation that you can now share an augment. And then we publish open API. So can actually use a tool as you want first use case log analytics, Hey, it's easy to just stream your logs in and we give you elastic search puppet services, same thing that with CQL, you'll see mainstream machine learning next year. So listen, I think we have the data lake, you know, 3.0 now, and we're just stretching our legs run off >>Well, and you have to say it log analytics. But if I really do believe in this concept of building data products and data services, because I want to sell them, I want to monetize them and being able to do that quickly and easily, so that can consume them as the future. So guys, thanks so much for coming on the program. Really appreciate it. All right. In a moment, Kevin Miller of Amazon web services joins me. You're watching the cube, your leader in high tech coverage.

>>president >>so called big data promised to usher in a new era of innovation where companies competed on the basis of insights and agile decision making. There's little question that social media giants, search leaders and e commerce companies benefited. They had the engineering shops and the execution capabilities to take troves of data and turned them into piles of money. But many organizations were not as successful. They invested heavily in data architecture is tooling and hyper specialized experts to build out their data pipelines. Yet they still struggle today to truly realize they're busy. Did data in their lakes is plentiful but actionable insights aren't so much chaos. Search is a cloud based startup that wants to change this dynamic with a new approach designed to simplify and accelerate time to insights and dramatically lower cost and with us to discuss his company and its vision for the future is cuba Lem Ed Walsh had great to see you. Thanks for coming back in the cube. >>I always love to be here. Thank you very much. It's always a warm welcome. Thank you. >>Alright, so give us the update. You guys have had some big funding rounds, You're making real progress on the tech, taking it to market what's new with chaos surgery. >>Sure. Actually even a lot of good exciting things happen. In fact just this month we need some, you know, obviously announced some pretty exciting things. So we unveiled what we consider the industry first multi model data late platform that we allow you to take your data in S three. In fact, if you want to show the image you can, but basically we allow you to put your data in S three and then what we do is we activate that data and what we do is a full index of the data and makes it available through open a P. I. S. And the key thing about that is it allows your end users to use the tools are using today. So simply put your data in your cloud option charge, think Amazon S three and glacier think of all the different data. Is that a natural act? And then we do the hard work. And the key thing is to get one unified delic but it's a multi mode model access so we expose api like the elastic search aPI So you can do things like search or using cabana do log analytics but you can also do things like sequel, use Tableau looker or bring relational concepts into cabana. Things like joins in the data back end. But it allows you also to machine learning which is early next year. But what you get is that with that because of a data lake philosophy, we're not making new transformations without all the data movement. People typically land data in S. Three and we're on the shoulders of giants with us three. Um There's not a better more cost effective platform. More resilient. There's not a better queuing system out there and it's gonna cost curve that you can't beat. But basically so people store a lot of data in S. Three. Um But what their um But basically what you have to do is you E. T. L. Out to other locations. What we do is allow you to literally keep it in place. We index in place. We write our hot index to rewrite index, allow you to go after that but published an open aPI S. But what we avoid is the GTL process. So what our index does is look at the data and does full scheme of discovery normalization, were able to give sample sets. And then the refinery allows you to advance transformations using code. Think about using sequel or using rejects to change that data pull the dead apartheid things but use role based access to give that to the end user. But it's in a format that their tools understand cabana will use the elasticsearch ap or using elasticsearch calls but also sequel and go directly after data by doing that. You get a data lake but you haven't had to take the three weeks to three months to transform your data. Everyone else makes you. And you talk about the failure. The idea that Alex was put your data there in a very scalable resilient environment. Don't do transformation. It was too hard to structure for databases and data. Where else is put it there? We'll show you how value out Largely un delivered. But we're that last mile. We do exactly that. Just put it in s. three and we activated and activate it with a piece that the tools of your analysts use today or what they want to use in the future. That is what's so powerful. So basically we're on the shoulders of giants with street, put it there and we light it up and that's really the last mile. But it's this multi model but it's also this lack of transformation. We can do all the transformation that's all to virtually and available immediately. You're not doing extended GTL projects with big teams moving around a lot of data in the enterprise. In fact, most time they land and that's three and they move it somewhere and they move it again. What we're saying is now just leave in place well index and make it available. >>So the reason that it was interesting, so the reason they want to move in the S three was the original object storage cloud. It was, it was a cheap bucket. Okay. But it's become much more than that when you talk to customers like, hey, I have all this data in this three. I want to do something with it. I want to apply machine intelligence. I want to search it. I want to do all these things, but you're right. I have to move it. Oftentimes to do that. So that's a huge value. Now can I, are you available in the AWS marketplace yet? >>You know, in fact that was the other announcement to talk about. So our solution is one person available AWS marketplace, which is great for clients because they've been burned down their credits with amazon. >>Yeah, that's that super great news there. Now let's talk a little bit more about data. Like you know, the old joke of the tongue in cheek was data lakes become data swamps. You sort of know, see no schema on, right. Oh great. I can put everything into the lake and then it's like, okay, what? Um, so maybe double click on that a little bit and provide a little bit more details to your, your vision there and your philosophy. >>So if you could put things that data can get after it with your own tools on elastic or search, of course you do that. If you don't have to go through that. But everyone thinks it's a status quo. Everyone is using, you know, everyone has to put it in some sort of schema in a database before they can get access to what everyone does. They move it some place to do it. Now. They're using 1970s and maybe 1980s technology. And they're saying, I'm gonna put it in this database, it works on the cloud and you can go after it. But you have to do all the same pain of transformation, which is what takes human. We use time, cost and complexity. It takes time to do that to do a transformation for an user. It takes a lot of time. But it also takes a teams time to do it with dBS and data scientists to do exactly that. And it's not one thing going on. So it takes three weeks to three months in enterprise. It's a cost complexity. But all these pipelines for every data request, you're trying to give them their own data set. It ends up being data puddles all over this. It might be in your data lake, but it's all separated. Hard to govern. Hard to manage. What we do is we stop that. What we do is we index in place. Your dad is already necessary. Typically retailing it out. You can continue doing that. We really are just one more use of the data. We do read only access. We do not change that data and you give us a place in. You're going to write our index. It's a full rewrite index. Once we did that that allows you with the refinery to make that we just we activate that data. It will immediately fully index was performant from cabana. So you no longer have to take your data and move it and do a pipeline into elasticsearch which becomes kind of brittle at scale. You have the scale of S. Three but use the exact same tools you do today. And what we find for like log analytics is it's a slightly different use case for large analytics or value prop than Be I or what we're doing with private companies but the logs were saving clients 50 to 80% on the hard dollars a day in the month. They're going from very limited data sets to unlimited data sets. Whatever they want to keep an S. Three and glacier. But also they're getting away from the brittle data layer which is the loosen environment which any of the data layers hold you back because it takes time to put it there. But more importantly It becomes brittle at scale where you don't have any of that scale issue when using S. three. Is your dad like. So what what >>are the big use cases Ed you mentioned log analytics? Maybe you can talk about that. And are there any others that are sort of forming in the marketplace? Any patterns that you see >>Because of the multi model we can do a lot of different use cases but we always work with clients on high R. O. I use cases why the Big Bang theory of Due dad like and put everything in it. It's just proven not to work right? So what we're focusing first use cases, log analytics, why as by way with everything had a tipping point, right? People were buying model, save money here, invested here. It went quickly to no, no we're going cloud native and we have to and then on top of it it was how do we efficiently innovate? So they got the tipping point happens, everyone's going cloud native. Once you go cloud native, the amount of machine generated data that you have that comes from the environment dramatically. It just explodes. You're not managing hundreds or thousands or maybe 10,000 endpoints, you're dealing with millions or billions and also you need this insight to get inside out. So logs become one of the things you can't keep up with it. I think I mentioned uh we went to a group of end users, it was only 60 enterprise clients but we asked him what's your capture rate on logs And they said what do you want it to be 80%, actually 78 said listen we want eight captured 80 200 of our logs. That would be the ideal not everything but we need most of it. And then the same group, what are you doing? Well 82 had less than 50%. They just can't keep up with it and every everything including elastic and Splunk. They work harder to the process to narrow and keep less and less data. Why? Because they can't handle the scale, we just say landed there don't transform will make it all available to you. So for log analytics, especially with cloud native, you need this type of technology and you need to stop, it's like uh it feels so good when you stop hitting your head against the wall. Right? This detail process that this type of scale just doesn't work. So that's exactly we're delivering the second use case uh and that's with using elastic KPI but also using sequel to go after the same data representation. And we come out with machine learning. You can also do anomaly detection on the same data representation. So for a log uh analytic use case series devops setups. It's a huge value problem now the same platform because it has sequel exposed. You can do just what we use the term is agile B. I people are using you think about look or tableau power bi I uh metabolic. I think of all these toolsets that people want to give and uh and use your business or coming back to the centralized team every single week asking for new datasets. And they have to be set up like a data set. They have to do an e tail process that give access to that data where because of the way just landed in the bucket. If you have access to that with role based access, I can literally get you access that with your tool set, let's say Tableau looker. You know um these different data sets literally in five minutes and now you're off and running and if you want a new dataset they give another virtual and you're off and running. But with full governance so we can use to be in B I either had self service or centralized. Self service is kind of out of control, but we can move fast and the centralized team is it takes me months but at least I'm in control. We allow you do both fully governed but self service. Right. I got to >>have lower. I gotta excel. All right. And it's like and that's the trade off on each of the pieces of the triangle. Right. >>And they make it easy, we'll just put in a data source and you're done. But the problem is you have to E T L the data source. And that's what takes the three weeks to three months in enterprise and we do it virtually in five minutes. So now the third is actually think about um it's kind of a combination of the two. Think about uh you love the beers and diaper stories. So you know, think about early days of terror data where they look at sales out data for business and they were able to look at all the sales out data, large relational environment, look at it, they crunch all these numbers and they figured out by different location of products and the start of they sell more sticker things and they came up with an analogy which everyone talked about beers and diapers. If you put it together, you sell more from why? Because afternoon for anyone that has kids, you picked up diapers and you might want to grab a beer of your home with the kids. But that analogy 30 years ago, it's now well we're what's the shelf space now for approximate company? You know it is the website, it's actually what's the data coming from there. It's actually the app logs and you're not capturing them because you can't in these environments or you're capturing the data. But everyone's telling, you know, you've got to do an E. T. L. Process to keep less data. You've got to select, you got to be very specific because it's going to kill your budget. You can't do that with elastic or Splunk, you gotta keep less data and you don't even know what the questions are gonna ask with us, Bring all the app logs just land in S. three or glacier which is the most it's really shoulders of giants right? There's not a better platform cost effectively security resilience or through but to think about what you can stream and the it's the best queuing platform I've ever seen in the industry just landed there. And it's also very cost effective. We also compress the data. So by doing that now you match that up with actually relatively small amount of relational data and now you have the vaccine being data. But instead it's like this users using that use case and our top users are always, they start with this one then they use that feature and that feature. Hey, we just did new pricing is affecting these clients and that clients by doing this. We get that. But you need that data and people aren't able to capture it with the current platforms. A data lake. As long as you can make it available. Hot is a way to do it. And that's what we're doing. But we're unique in that. Other people are making GTL IT and put it in a in 19 seventies and 19 eighties data format called a schema. And we avoided that because we basically make S three a hot and elected. >>So okay. So I gotta I want to, I want to land on that for a second because I think sometimes people get confused. I know I do sometimes without chaos or it's like sometimes don't know where to put you. I'm like okay observe ability that seems to be a hot space. You know of course log analytics as part of that B. I. Agile B. I. You called it but there's players like elastic search their star burst. There's data, dogs, data bricks. Dream EOS Snowflake. I mean where do you fit where what's the category and how do you differentiate from players like that? >>Yeah. So we went about it fundamentally different than everyone else. Six years ago. Um Tom hazel and his band of merry men and women came up and designed it from scratch. They may basically yesterday they purposely built make s free hot analytic environment with open A. P. I. S. By doing that. They kind of changed the game so we deliver upon the true promises. Just put it there and I'll give you access to it. No one else does that. Everyone else makes you move the data and put it in schema of some format to get to it. And they try to put so if you look at elasticsearch, why are we going after? Like it just happens to be an easy logs are overwhelming. You once you go to cloud native, you can't afford to put it in a loose seen the elk stack. L is for loosen its inverted index. Start small. Great. But once you now grow it's now not one server. Five servers, 15 servers, you lose a server, you're down for three days because you have to rebuild the whole thing. It becomes brittle at scale and expensive. So you trade off I'm going to keep less or keep less either from retention or data. So basically by doing that so elastic we're not we have no elastic on that covers but we allow you to well index the data in S. Tree and you can access it directly through a cabana interface or an open search interface. Api >>out it's just a P. >>It's open A P. I. S. It's And by doing that you've avoided a whole bunch of time cost, complexity, time of your team to do it. But also the time to results the delays of doing that cost. It's crazy. We're saving 50-80 hard dollars while giving you unlimited retention where you were dramatically limited before us. And as a managed service you have to manage that Kind of Clunky. Not when it starts small, when it starts small, it's great once at scale. That's a terrible environment to manage the scale. That's why you end up with not one elasticsearch cluster, dozens. I just talked to someone yesterday had 125 elasticsearch clusters because of the scale. So anyway, that's where elastic we're not a Mhm. If you're using elastic it scale and you're having problems with the retired off of cost time in the, in the scale, we become a natural fit and you don't change what your end users do. >>So the thing, you know, they had people here, this will go, wow, that sounds so simple. Why doesn't everybody do this? The reason is it's not easy. You said tom and his merry band. This is really hard core tech. Um and it's and it's it's not trivial what you've built. Let's talk about your secret sauce. >>Yeah. So it is a patented technology. So if you look at our, you know, component for architecture is basically a large part of the 90% of value add is actually S. Three, I gotta give S three full kudos. They built a platform that we're on shoulders of giants. Um But what we did is we purpose built to make an object storage a hot alec database. So we have an index, like a database. Um And we basically the data you bring a refinery to be able to do all the advanced type of transformation but all virtually done because we're not changing the source of record, we're changing the virtual views And then a fabric allows you to manage and be fully elastic. So if we have a big queries because we have multiple clients with multiple use cases, each multiple petabytes, we're spending up 1800 different nodes after a particular environment. But even with all that we're saving them 58%. But it's really the patented technology to do this, it took us six years by the way, that's what it takes to come up with this. I come upon it, I knew the founder, I've known tom tom a stable for a while and uh you know his first thing was he figured out the math and the math worked out. Its deep tech, it's hard tech. But the key thing about it is we've been in market now for two years, multiple use cases in production at scale. Um Now what you do is roadmap, we're adding a P. I. So now we have elasticsearch natural proofpoint. Now you're adding sequel allows you open up new markets. But the idea for the person dealing with, you know, so we believe we deliver on the true promise of Data Lakes and the promise of Data lakes was put it there, don't focus on transferring. It's just too hard. I'll get insights out and that's exactly what we do. But we're the only ones that do that everyone else makes you E. T. L. At places. And that's the innovation of the index in the refinery that allows the index in place and give virtual views in place at scale. Um And then the open api is to be honest, uh I think that's a game. Give me an open api let me go after it. I don't know what tool I'm gonna use next week every time we go into account they're not a looker shop or Tableau Sharp or quick site shop there, all of them and they're just trying to keep up with the businesses. Um and then the ability to have role based access where actually can give, hey, get them their own bucket, give them their own refinery. As long as they have access to the data, they can go to their own manipulation ends up being >>just, >>that's the true promise of data lakes. Once we come out with machine learning next year, now you're gonna rip through the same embassy and the way we structured the data matrices. It's a natural fit for things like tensorflow pytorch, but that's, that's gonna be next year just because it's a different persona. But the underlining architecture has been built, what we're doing is trying to use case that time. So we worked, our clients say it's not a big bang. Let's nail a use case that works well. Great R. O. I great business value for a particular business unit and let's move to the next. And that's how I think it's gonna be really. That's what if you think about gardener talks about, if you think about what really got successful in data, where else in the past? That's exactly it wasn't the big bang, it was, let's go and nail it for particular users. And that's what we're doing now because it's multi model, there's a bunch of different use cases, but even then we're focusing on these core things that are really hard to do with other relational only environments. Yeah, I >>can see why you're still because you know, you haven't been well, you and I have talked about the api economy for forever and then you've been in the storage world so long. You know what a nightmare is to move data. We gotta, we gotta jump. But I want to ask you, I want to be clear on this. So you are your cloud cloud Native talked to frank's Lukman maybe a year ago and I asked him about on prem and he's like, no, we're never doing the halfway house. We are cloud all the >>way. I think >>you're, I think you have a similar answer. What what's your plan on Hybrid? >>Okay. We get, there's nothing about technology, we can't go on, but we are 100 cloud native or only in the public cloud. We believe that's a trend line. Everyone agrees with us, we're sticking there. That's for the opportunity. And if you can run analytics, There's nothing better than getting to the public cloud like Amazon and he was actually, that were 100 cloud native. Uh, we love S three and what would be a better place to put this is put the next three and we just let you light it up and then I guess if I'm gonna add the commercial and buy it through amazon marketplace, which we love that business model with amazon. It's >>great. Ed thanks so much for coming back in the cube and participating in the startup showcase. Love having you and best of luck. Really exciting. >>Hey, thanks again, appreciate it. >>All right, thank you for watching everybody. This is Dave Volonte for the cube. Keep it right there.

>> Narrator: From around the globe it's theCUBE, with digital coverage of AWS re:Invent 2020. Special coverage sponsored by AWS Global Partner Network. >> Hello and welcome to theCUBE Virtual and our coverage of AWS re:Invent 2020 with special coverage of APN partner experience. We are theCUBE Virtual and I'm your host, Justin Warren. And today I'm joined by Ed Walsh, CEO of ChaosSearch. Ed, welcome to theCUBE. >> Well thank you for having me, I really appreciate it. >> Now, this is not your first time here on theCUBE. You're a regular here and I've loved it to have you back. >> I love the platform you guys are great. >> So let's start off by just reminding people about what ChaosSearch is and what do you do there? >> Sure, the best way to say is so ChaosSearch helps our clients know better. We don't do that by a special wizard or a widget that you give to your, you know, SecOp teams. What we do is a hard work to give you a data platform to get insights at scale. And we do that also by achieving the promise of data lakes. So what we have is a Chaos data platform, connects and indexes data in a customer's S3 or glacier accounts. So inside your data lake, not our data lake but renders that data fully searchable and available for analysis using your existing tools today 'cause what we do is index it and publish open API, it's like API like Elasticsearch API, and soon SQL. So give you an example. So based upon those capabilities were an ideal replacement for a commonly deployed, either Elasticsearch or ELK Stack deployments, if you're hitting scale issues. So we talk about scalable log analytics, and more and more people are hitting these scale issues. So let's say if you're using Elasticsearch ELK or Amazon Elasticsearch, and you're hitting scale issues, what I mean by that is like, you can't keep enough retention. You want longer retention, or it's getting very expensive to keep that retention, or because the scale you hit where you have availability, where the cluster is hard to keep up running or is crashing. That's what we mean by the issues at scale. And what we do is simply we allow you, because we're publishing the open API of Elasticsearch use all your tools, but we save you about 80% off your monthly bill. We also give you an, and it's an and statement and give you unlimited retention. And as much as you want to keep on S3 or into Glacier but we also take care of all the hassles and management and the time to manage these clusters, which ends up being on a database server called leucine. And we take care of that as a managed service. And probably the biggest thing is all of this without changing anything your end users are using. So we include Kibana, but imagine it's an Elastic API. So if you're using API or Kibana, it's just easy to use the exact same tools used today, but you get the benefits of a true data lake. In fact, we're running now Elasticsearch on top of S3 natively. If that makes it sense. >> Right and natively is pretty cool. And look, 80% savings, is a dramatic number, particularly this year. I think there's a lot of people who are looking to save a few quid. So it'd be very nice to be able to save up to 80%. I am curious as to how you're able to achieve that kind of saving though. >> Yeah, you won't be the first person to ask me that. So listen, Elastic came around, it was, you know we had Splunk and we also have a lot of Splunk clients, but Elastic was a more cost effective solution open source to go after it. But what happens is, especially at scale, if it's fall it's actually very cost-effective. But underneath last six tech ELK Stack is a leucine database, it's a database technology. And that sits on our servers that are heavy memory count CPU count in and SSDs. So you can do on-prem or even in the clouds, so if you do an Amazon, basically you're spinning up a server and it stays up, it doesn't spin up, spin down. So those clusters are not one server, it's a cluster of those servers. And typically if you have any scale you're actually having multiple clusters because you don't dare put it on one, for different use cases. So our savings are actually you no longer need those servers to spin up and you don't need to pay for those seen underneath. You can still use Kibana under API but literally it's $80 off your bill that you're paying for your service now, and it's hard dollars. So it's not... And we typically see clients between 70 and 80%. It's up to 80, but it's literally right within a 10% margin that you're saving a lot of money, but more importantly, saving money is a great thing. But now you have one unified data lake that you can have. You used to go across some of the data or all the data through the role-based access. You can give different people. Like we've seen people who say, hey give that, help that person 40 days of this data. But the SecOp up team gets to see across all the different law. You know, all the machine generated data they have. And we can give you a couple of examples of that and walk you through how people deploy if you want. >> I'm always keen to hear specific examples of how customers are doing things. And it's nice that you've thought of drawn that comparison there around what what cloud is good for and what it isn't is. I'll often like to say that AWS is cheap to fail in, but expensive to succeed. So when people are actually succeeding with this and using this, this broad amount of data so what you're saying there with that savings I've actually got access to a lot more data that I can do things with. So yeah, if you could walk through a couple of examples of what people are doing with this increased amount of data that they have access to in EKL Search, what are some of the things that people are now able to unlock with that data? >> Well, literally it's always good for a customer size so we can go through and we go through it however it might want, Kleiner, Blackboard, Alert Logic, Armor Security, HubSpot. Maybe I'll start with HubSpot. One of our good clients, they were doing some Cloud Flare data that was one of their clusters they were using a lot to search for. But they were looking at to look at a denial service. And they were, we find everyone kind of at scale, they get limited. So they were down to five days retention. Why? Well, it's not that they meant to but basically they couldn't cost-effectively handle that in the scale. And also they're having scale issues with the environment, how they set the cluster and sharding. And when they also denial service tech, what happened that's when the influx of data that is one thing about scale is how fast it comes out, yet another one is how much data you have. But this is as the data was coming after them at denial service, that's when the cluster would actually go down believe it or not, you know right. When you need your log analysis tools. So what we did is because they're just using Kibana, it was easy swap. They ran in parallel because we published the open API but we took them from five days to nine days. They could keep as much as they want but nine days for denial services is what they wanted. And then we did save them in over $4 million a year in hard dollars, What they're paying in their environment from really is the savings on the server farm and a little bit on the Elasticsearch Stack. But more importantly, they had no outages since. Now here's the thing. Are you talking about the use case? They also had other clusters and you find everyone does it. They don't dare put it on one cluster, even though these are not one server, they're multiple servers. So the next use case for CloudFlare was one, the next QS and it was a 10 terabyte a day influx kept it for 90 days. So it's about a petabyte. They brought another use case on which was NetMon, again, Network Monitoring. And again, I'm having the same scale issue, retention area. And what they're able to do is easily roll that on. So that's one data platform. Now they're adding the next one. They have about four different use cases and it's just different clusters able to bring together. But now what they're able to do give you use cases either they getting more cost effective, more stability and freedom. We say saves you a lot of time, cost and complexity. Just the time they manage that get the data in the complexities around it. And then the cost is easy to kind of quantify but they've got better but more importantly now for particular teams they only need their access to one data but the SecOP team wants to see across all the data. And it's very easy for them to see across all the data where before it was impossible to do. So now they have multiple large use cases streaming at them. And what I love about that particular case is at one point they were just trying to test our scale. So they started tossing more things at it, right. To see if they could kind of break us. So they spiked us up to 30 terabytes a day which is for Elastic would even 10 terabytes a day makes things fall over. Now, if you think of what they just did, what were doing is literally three steps, put your data in S3 and as fast as you can, don't modify, just put it there. Once it's there three steps connect to us, you give us readability access to those buckets and a place to write the indexy. All of that stuff is in your S3, it never comes out. And then basically you set up, do you want to do live or do you want to do real time analysis? Or do you want to go after old data? We do the rest, we ingest, we normalize the schema. And basically we give you our back and the refinery to give the right people access. So what they did is they basically throw a whole bunch of stuff at it. They were trying to outrun S3. So, you know, we're on shoulders of giants. You know, if you think about our platform for clients what's a better dental like than S3. You're not going to get a better cross curve, right? You're not going to get a better parallelism. And so, or security it's in your, you know a virtual environment. But if you... And also you can keep data in the right location. So Blackboard's a good example. They need to keep that in all the different regions and because it's personal data, they, you know, GDPR they got to keep data in that location. It's easy, we just put compute in each one of the different areas they are. But the net net is if you think that architecture is shoulders of giants if you think you can outrun by just sheer volume or you can put in more cost-effective place to keep long-term or you think you can out store you have so much data that S3 and glacier can't possibly do it. Then you got me at your bigger scale at me but that's the scale we'r&e talking about. So if you think about the spiked our throughput what they really did is they try to outrun S3. And we didn't pick up. Now, the next thing is they tossed a bunch of users at us which were just spinning up in our data fabric different ways to do the indexing, to keep up with it. And new use cases in case they're going after everyone gets their own worker nodes which are all expected to fail in place. So again, they did some of that but really they're like you guys handled all the influx. And if you think about it, it's the shoulders of giants being on top of an Amazon platform, which is amazing. You're not going to get a more cost effective data lake in the world, and it's continuing to fall in price. And it's a cost curve, like no other, but also all that resiliency, all that security and the parallelism you can get, out of an S3 Glacier is just a bar none is the most scalable environment, you can build an environment. And what we do is a thin layer. It's a data platform that allows you to have your data now fully searchable and queryable using your tools >> Right and you, you mentioned there that, I mean you're running in AWS, which has broad experience in doing these sorts of things at scale but on that operational management side of things. As you mentioned, you actually take that off, off the hands of customers so that you run it on their behalf. What are some of the areas that you see people making in trying to do this themselves, when you've gone into customers, and brought it into the EKL Search platform? >> Yeah, so either people are just trying their best to build out clusters of Elasticsearch or they're going to services like Logz.io, Sumo Logic or Amazon Elasticsearch services. And those are all basically on the same ELK Stack. So they have the exact same limits as the same bits. Then we see people trying to say, well I really want to go to a data lake. I want to get away from these database servers and which have their limits. I want to use a data Lake. And then we see a lot of people putting data into environments before they, instead of using Elasticsearch, they want to use SQL type tools. And what they do is they put it into a Parquet or Presto form. It's a Presto dialect, but it into Parquet and structure it. And they go a lot of other way to, Hey it's in the data lake, but they end up building these little islands inside their data lake. And it's a lot of time to transform the data, to get it in a format that you can go after our tools. And then what we do is we don't make you do that. Just literally put the data there. And then what we do is we do the index and a polish API. So right now it's Elasticsearch in a very short time we'll publish Presto or the SQL dialect. You can use the same tool. So we do see people, either brute forcing and trying their best with a bunch of physical servers. We do see another group that says, you know, I want to go use an Athena use cases, or I want to use a there's a whole bunch of different startups saying, I do data lake or data lake houses. But they are, what they really do is force you to put things in the structure before you get insight. True data lake economics is literally just put it there, and use your tools natively to go after it. And that's where we're unique compared to what we see from our competition. >> Hmm, so with people who have moved into ChaosSearch, what's, let's say pick one, if you can, the most interesting example of what people have started to do with, with their data. What's new? >> That's good. Well, I'll give you another one. And so Armor Security is a good one. So Armor Security is a security service company. You know, thousands of clients doing great I mean a beautiful platform, beautiful business. And they won Rackspace as a partner. So now imagine thousand clients, but now, you know massive scale that to keep up with. So that would be an example but another example where we were able to come in and they were facing a major upgrade of their environment just to keep up, and they expose actually to their customers is how their customers do logging analytics. What we're able to do is literally simply because they didn't go below the API they use the exact same tools that are on top and in 30 days replaced that use case, save them tremendous amount of dollars. But now they're able to go back and have unlimited retention. They used to restrict their clients to 14 days. Now they have an opportunity to do a bunch of different things, and possible revenue opportunities and other. But allow them to look at their business differently and free up their team to do other things. And now they're, they're putting billing and other things into the same environment with us because one is easy it's scale but also freed up their team. No one has enough team to do things. And then the biggest thing is what people do interesting with our product is actually in their own tools. So, you know, we talk about Kibana when we do SQL again we talk about Looker and Tableau and Power BI, you know, the really interesting thing, and we think we did the hard work on the data layer which you can say is, you know I can about all the ways you consolidate the performance. Now, what becomes really interesting is what they're doing at the visibility level, either Kibana or the API or Tableau or Looker. And the key thing for us is we just say, just use the tools you're used to. Now that might be a boring statement, but to me, a great value proposition is not changing what your end users have to use. And they're doing amazing things. They're doing the exact same things they did before. They're just doing it with more data at bigger scale. And also they're able to see across their different machine learning data compared to being limited going at one thing at a time. And that getting the correlation from a unified data lake is really what we, you know we get very excited about. What's most exciting to our clients is they don't have to tell the users they have to use a different tool, which, you know, we'll decide if that's really interesting in this conversation. But again, I always say we didn't build a new algorithm that you going to give the SecOp team or a new pipeline cool widget that going to help the machine learning team which is another API we'll publish. But basically what we do is a hard work of making the data platform scalable, but more importantly give you the APIs that you're used to. So it's the platform that you don't have to change what your end users are doing, which is a... So we're kind of invisible behind the scenes. >> Well, that's certainly a pretty strong proposition there and I'm sure that there's plenty of scope for customers to come and and talk to you because no one's creating any less data. So Ed, thanks for coming out of theCUBE. It's always great to see you here. >> Know, thank you. >> You've been watching theCUBE Virtual and our coverage of AWS re:Invent 2020 with special coverage of APN partner experience. Make sure you check out all our coverage online, either on your desktop, mobile on your phone, wherever you are. I've been your host, Justin Warren. And I look forward to seeing you again soon. (soft music)

>> From theCUBE Studios in Palo Alto in Boston, connecting with thought leaders all around the world, this is theCUBE Conversation. >> Hey, everybody, this is Dave Vellante, and welcome to this CXO Series. As you know, I've been running this series discussing major trends and CXOs, how they've navigated through the pandemic. And we've got some good news and some bad news today. And Ed Walsh is here to talk about that. Ed, how you doing? Great to see you. >> Great seeing you, thank you for having me on. I really appreciate it. So the bad news is Ed Walsh is leaving IBM as the head of the storage division (indistinct). But the good news is, he's joining a new startup as CEO, and we're going to talk about that, but Ed, always a pleasure to have you. You're quite a run at at IBM. You really have done a great job there. So, let's start there if we can before we get into the other part of the news. So, you give us the update. You're coming off another strong quarter for the storage business. >> I would say listen, they're sweet, heartily, but to be honest, we're leaving them in a really good position where they have sustainable growth. So they're actually IBM storage in a very good position. I think you're seeing it in the numbers as well. So, yeah, listen, I think the team... I'm very proud of what they were able to pull off. Four years ago, they kind of brought me in, hey, can we get IBM storage back to leadership? They were kind of on their heels, not quite growing, or not growing but falling back in market share. You know, kind of a distant third place finisher, and basically through real innovation that mattered to clients which that's a big deal. It's the right innovation that matters to the clients. We really were able to dramatically grow, grow all different four segments of the portfolio. But also get things like profitability growing, but also NPS growing. It really allowed us to go into a sustainable model. And it's really about the team. You heard I've talked about team all the time, which is you get a good team and they really nailed great client experiences. And they take the right offerings and go to market and merge it. And I'll tell you, I'm very proud of what the IBM team put together. And I'm still the number one fan and inside or outside IBM. So it might be bittersweet, but I actually think they're ready for quite some growth. >> You know Ed, when you came in theCUBE, right after you had joined IBM, a lot of people are saying, Ed Walsh joined an IBM storage division to sell the division. And I asked you on theCUBE, are you there to sell division? And you said, no, absolutely not. So it's always it seemed to me, well, hey, it's good. It's a good business, good cash flow business, got a big customer base, so why would IBM sell it? Never really made sense to me. >> I think it's integral to what IBM does, I think it places their client base in a big way. And under my leadership, really, we got more aligned with what IBM is doing from the big IBM right. What we're doing around Red Hat hybrid multi cloud and what we're doing with AI. And those are big focuses of the storage portfolio. So listen, I think IBM as a company is in a position where they're really innovating and thriving, and really customer centric. And I think IBM storage is benefiting from that. And vice versa. I think it's a good match. >> So one of the thing I want to bring up before we move on. So you had said you were seeing a number. So I want to bring up a chart here. As you know, we've been using a lot of data and sharing data reporting from our partner. ETR, Enterprise Technology Research, they do quarterly surveys. They have a very tight methodology, it's similar to NPS. But it's a net score, we call it methodology. And every quarter they go out and what we're showing here is the results from the last three quarter, specific to IBM storage and IBM net score in storage. And net scores is essentially, we ask people are you spending more, are you spending less, we subtract the less from the more and that's the net score. And you can see when you go back to the October 19, survey, you know, low single digits and then it dipped in the April survey, which was the height of the pandemic. So this was this is forward looking. So in the height of the pa, the lockdown people were saying, maybe I'm going to hold off on budgets. But then now look at the July survey. Huge, huge up check. And I think this is testament to a couple of things. One is, as you mentioned, the team. But the other is, you guys have done a good job of taking R&D, building a product pipeline and getting it into the field. And I think that shows up in the numbers. That was really a one of the hallmarks of your leadership. >> Yeah, I mean, they're the innovation. IBM is there's almost an embarrassment of riches inside. It's how do you get in the pipeline? We went from a typically about for four years, four and a half year cycles, not a two year cycle product cycle. So we're able to innovate and bring it to market much quicker. And I think that's what clients are looking for. >> Yeah, so I mean, you brought a startup mentality to the division and of course now, cause your startup guy, let's face it. Now you're going back to the startup world. So the other part of the news is Ed Walsh is joining ChaosSearch as the CEO. ChaosSearches is a local Boston company, they're focused on log analytics but more on we're going to talk about that. So first of all, congratulations. And tell us about your decision. Why ChaosSearch? And you know where you're out there? >> Yeah, listen, if you can tell from the way I describe IBM, I mean, it was a hard decision to leave IBM, but it was a very, very easy decision to go to Chaos, right. So I knew the founder, I knew what he was working on for the last seven years, right. Last five years as a company, and I was just blown away at their fundamental innovation, and how they're really driving like how to get insights at scale from your data lake in the cloud. But also and also instead, and statements slash cost dramatically. And they make it so simple. Simply put your data in your S3 or really Cloud object storage. But right now, it's, Amazon, they'll go the rest of clouds, but just put your data in S3. And what we'll do is we'll index it, give you API so you can search it and query it. And it literally brings a way to do at scale data analysts. And also login analytics on everything you just put into S3 basically bucket. It makes it very simple. And because they're really fundamental, we can go through it. Fundamental on hard technology that data layer, but they kept all the API. So you're using your normal tools that we did for Elastic Search API's. You want to do Glyfada, you want to do Cabana, or you want to do SQL or you want to do use Looker, Tableau, all those work. Which is that's a part of it. It's really revolutionary what they're doing as far as the value prop and we can explain it. But also they made it evolution, it's very easy for clients to go. Just run in parallel, and then they basically turn off what they currently have running. >> So data lakes, really the term became popular during the sort of early big data, Hadoop era. And, Hadoop obviously brought a lot of innovation, you know, leave the data where it is. Bring the compute to the data, really launched the Big Data initiative, but it was very complicated. You had, MapReduce and and elastic MapReduce in the cloud. And, it really was a big batch job, where storage was really kind of a second class citizen, if you will. There wasn't a lot of real time stuff going on. And then, Spark comes in. And still there's this very complicated situation. So it's sounds like, ChaosSearch is really attacking that problem. And the first use case, it's really going after is log analytics. Explain that a little bit more, please. >> Yeah, so listen, they finally went after it with this, it's called a data lake engine for scalable and we'll say log analytics firstly. It was the first use case to go after it. But basically, they allows for log analytics people, everyone does it, and everyone's kind of getting to scale with it, right. But if you asked your IT department, are you even challenged with scale, or cost, or retention levels, but also management overlay of what they're doing on log analytics or security log analytics, or all this machine data they're collecting? The answer be absolutely no, it's a nightmare. It starts easy and becomes a big, very costly application for our environments. And what Chaos does is because they deal with a real issue, which is the data layer, but keep the API's on top. And so people easily use the data insights at scale, what they're able to do is very simply run in parallel and we'll save 80% of your cost, but also get better data retention. Cause there's typically a trade off. Clients basically have this trade off, or it gets really expensive. It gets to scale. So I should just retain less. We have clients that went from nine day retention and security logs to literally four and five days. If they didn't catch it in that time, it was too late. Now what they're able to do is, they're able to go to our solution. Not change what they're doing applications, because you're using the same API's, but literally save 80% and this is millions and 10s of millions of dollars of savings, but also basically get 90 day retention. There's really limitless, whatever you put into your S3 bucket, we're going to give you access to. So that alone shows you that it's literally revolutions that CFO wins because they save money. The IT department wins because they don't that wrestle with this data technology that wasn't really built. It is really built 30 years ago, wasn't built for this volume and velocity of data coming in. And then the data analytics guys, hey, I keep my tool set but I get all the retention I want. No one's limiting me anymore. So it's kind of an easy win win. And it makes it really easy for clients to have this really big benefit for them. And dramatic cost savings. But also you get the scale, which really means a lot in security login or anything else. >> So let's dig into that a little bit. So Cloud Object Storage has kind of become the de facto bucket, if you will. Everybody wants it, because it's simple. It's a get put kind of paradigm. And it's cheap, but it's also got performance issues. So people will throw cash at the problem, they'll have to move data around. So is that the problem that you're solving? Is it a performance? You know, problem is it a cause problem or both? And explain that a little bit. >> Yeah, so it's all over. So basically, if you were building a data lake, they would like to just put all their data in one very cost effective, scalable, resilient environment. And that is Cloud Object Storage, or S3, or every cloud has around, right? You can do also on prem, everyone would love to do that. And then literally get their insights out of it. But they want to go after it with our tools. Is it Search or is it SQL, they want to go after their own tools. That's the vision everyone wants. But what everyone does now is because this is where the core special sauce what ChaosSearch provides, is we built from the ground up. The database, the indexing technology, the database technology, how to actually make your Cloud object storage a database. We don't move it somewhere, we don't cash it. You put it in the inside the bucket, we literally make the Cloud object storage, the database. And then around it, we basically built a Chaos fabric that allows you to spin up compute nodes to go at the data in different ways. We truly have separated that the data from the compute, but also if a worker nodes, beautiful, beauty of like containerization technology, a worker nodes goes away, nothing happens. It's not like what you do on Prem. And all sudden you have to rebuild clusters. So by fundamentally solving that data layer, but really what was interesting is they just published API's, you mentioned put and get. So the API's you're using cloud obvious sources of put and get. Imagine we just added to that API, your Search API from elastic, or your SQL interface. It's just all we're doing is extending. You put it in the bucket will extend your ability to get after it. Really is an API company, but it's a hard tech, putting that data layer together. So you have cost effectiveness, and scale simultaneously. But we can ask for instance, log analytics. We don't cash, nothing's on the SSD, nothing's on local storage. And we're as fast as you're running Elastic Search on SSDs. So we've solved the performance and scale issues simultaneously. And that's really the core fundamental technology. >> And you do that with math, with algorithms, with machine learning, what's the secret sauce? Yeah, we should really have I'll tell you, my founder, just has the right interesting way of looking at problems. And he really looked at this differently and went after how do you make a both, going after data. He really did it in a different way, and really a modern way. And the reason it differentiates itself is he built from the ground up to do this on object storage. Where basically everyone else is using 30 year old technology, right? So even really new up and coming companies, they're using Tableau, Looker, or Snowflake could be another example. They're not changing how the data stored, they always have to move it ETL at somewhere to go after it. We avoid all that. In fact, we're probably a pretty good ecosystem players for all those partners as we go forward. >> So your talking about Tom Hazel, you're founder and CTO and he's brought in the team and they've been working on this for a while. What's his background? >> Launched Telkom, building out God boxes. So he's always been in the database space. I can't do his in my first day of the job, I can't do justice to his deep technology. There's a really good white paper on our website that does that pretty well. But literally the patent technology is a Chaos index, which is a database that it makes your object storage, the database. And then it's really the chaos fabric that puts around in the chaos refinery that gives you virtual views. But that's one solution. And if you look for log analytics, you come in log in and you get all the tools you're used to. But underneath the covers, were just saving about 80% of overall cost, but also almost limitless retention. We see people going from literally have been reduced the number of logs are keeping because of cost, and complexity, and scale, down to literally a very small amount and going right back at nine days. You could do longer, but that's what we see most people go into when they go to our service. >> Let's talk about the market. I mean, as a startup person, you always look for large markets. Obviously, you got to have good tech, a great team. And you want large markets. So the, space that you're in, I mean, I would think it started, early days and kind of the decision support. Sort of morphed into the data warehouse, you mentioned ETL, that's kind of part of it. Business Intelligence, it's sort of all in there. If you look at the EDW market, it's probably around 18 to 20 billion. Small slice of that is data lakes, maybe a billion or a billion plus. And then you got this sort of BI layer on top, you mentioned a lot of those. You got ETL, you probably get up into the 30,35 billion just sort of off the top of my head and from my historical experience and looking at these markets. But I have to say these markets have traditionally failed to live up to the expectations. Things like 360 degree views of the customer, real time analytics, delivering insights and self service to the business. Those are promises that these industries made. And they ended up being cumbersome, slow, maybe requiring real experts, requiring a lot of infrastructure, the cloud is changing that. Is that right? Is that the way to look at the market that you're going after? You're a player inside of that very large team. >> Yeah, I think we're a key fundamental component underneath that whole ecosystem. And yes, you're seeing us build a full stack solution for log analytics, because there's really good way to prove just how game changing the technology is. But also how we publishing API's, and it's seamless for how you're using log analytics. Same thing can be applied as we go across the SQL and different BI and analytic type of platforms. So it's exactly how we're looking at the market. And it's those players that are all struggling with the same thing. How they add more value to clients? It's a big cost game, right? So if I can literally make your underlying how you store your data and mix it literally 80% more cost effective. that's a big deal or simultaneously saving 80% and give you much longer retention. Those two things are typically, Lily a trade off, you have to go through, and we don't have to do that. That's what really makes this kind of the underlying core technology. And really I look at log analytics is really the first application set. But or if you have any log analytics issues, if you talk to your teams and find out, scale, cost, management issues, it's a pretty we make it very easy. Just run in parallel, we'll do a PLC, and you'll see how easy it is you can just save 80% which is, 80% and better retention is really the value proposition you see at scale, right. >> So this is day zero for you. Give us the hundred day plan, what do you want to accomplish? Where are you going to focus your priorities? I mean, obviously, the company's been started, it's well funded, but where are you going to focus in the next 100 days? >> No, I think it's building out where are we taking the next? There's a lot of things we could do, there's degrees of freedom as far as where we'd go with this technology is pretty wide. You're going to see us be the best log analytic company there. We're getting, really a (mumbling) we, you saw the announcement, best quarter ever last quarter. And you're seeing this nice as a service ramp, you're going to see us go to VPC. So you can do as a service with us, but now we can put this same thing in your own virtual private data center. You're going to see us go to Google, Azure, and also IBM cloud. And the really, clients are driving this. It's not us driving it, but you're going to see actually the client. So we'll go into Google because we had a couple financial institutions that are saying they're driving us to go do exactly that. So it's more really working with our client sets and making sure we got the right roadmap to support what they're trying to do. And then the ecosystem is another play. How to, you know, my core technology is not necessarily competitive with anyone else. No one else is doing this. They're just kind of, hey, move it here, I'll put it on this, you know, a foundational DV or they'll put it on on a presto environment. They're not really worried about the bottom line economics, which is really that's the value prop and that's the hard tech and patented technology that we bring to this ecosystem. >> Well, people are definitely worried about their cloud bills. The the CFO saying, whoa, cause it's so easy to spin up, instances in the cloud. And so, Ed it really looks like you're going after a real problem. You got some great tech behind you. And of course, we love the fact that it's another Boston based company that you're joining, cause it's more Boston based startups. Better for us here at the East Coast Cube, so give us a give us your final thoughts. What should we look for? I'm sure we're going to be being touched and congratulations. >> No, hey, thank you for the time. I'm really excited about this. I really just think it's fundamental technology that allows us to get the most out of everything you're doing around analytics in the cloud. And if you look at a data lake model, I think that's our philosophy. And we're going to drive it pretty aggressively. And I think it's a good fundamental innovation for the space and that's the type of tech that I like. And I think we can also, do a lot of partnering across ecosystems to make it work for a lot of different people. So anyway, so I guess thank you very much for the time appreciate. >> Yeah, well, thanks for coming on theCUBE and best of luck. I'm sure we're going to be learning a lot more and hearing a lot more about ChaosSearch, Ed Walsh. This is Dave Vellante. Thank you for watching everybody, and we'll see you next time on theCUBE. (upbeat music)

>>From the cube studios in Palo Alto in Boston. It's the cube covering IBM thing brought to you by IBM. >>Hi everybody. We're back. This is Dave Volante for the cube and you watching our continuous coverage of the IBM thing, 2020 digital event experience. And ed Walsh is here as the general manager. So the IBM storage division and software defined infrastructure. Ed, last time you were about to four feet to my left. I wish you were face to face but this'll, this'll have to do. Thanks for coming on the new normal. I like to call this maybe the new abnormal as some of us are still in lockdown but is the new normal. So we'll see more of this. So welcome it. I embrace it. No. So had you, you've obviously seen a number of, of downturns. You've run a lot, a lot of businesses, you've been on rocket ship businesses, you've been at IBM for a couple of stints. Obviously we've never seen anything like this. >>When did you first start getting visibility, uh, that this was going to be an issue? Obviously you guys have presence in China, okay. In AP. Uh, but when did you start to see it and what was your first move for the team? Yeah, sure. And so, uh, yeah, I've had the opportunity to lead a couple businesses and that was it. Okay. One, 2008. Ah, and this is, it is very different. But as far as our visibility on this, um, we have a worldwide and I'll say awesome. Right. Okay. So we saw this as far as a supply chain issue, um, and we came into it hot from Q4. We had a very good Q4 so I came into it hot or something. Why? So we are tracking it early and then we started to see the issues in China in late January. Then of course they shut down, came back to open after the Chinese new year in to be honest, they weren't quite back. >>So we were watching it almost as a support. Right. Main challenge. Yes. We do a lot of business in China, so we were also watching that, but it was light chain. But every single day managing that supply chain, I get out and give a compliment to my team. Uh, I don't think anyone has a better supply chain, but then of course quickly moved and everyone says, well, you should have seen it. This happened really fast. So it's a, it's different than other crises because it actually has to do with humans in life. Okay. All the other crisis were financial crisis. These, and we largely just manage the business through it and you're worried about your employees from the stress level, but you don't worry about the employees by the health level. So, uh, so we did see it early with supply chain that quickly gotten demand. And to be honest, when Italy went down, well, when Italy had the challenges that it happened so fast, when it shut down, uh, that was kind of a big wake up call for us. >>Mmm. You saw IBM respond very quickly. Um, everyone was at home almost immediately, even in countries weren't set up for it really took care of our people. But then we immediately, you saw the IBM was going to work really helping our clients. So we saw it kind of early, but it went from a hundred percent supply chain to a demand issue. And then we did have different real uh, interesting is a bad word, but interesting supply chain challenges as well. What it went on different countries stopping shipping's coming in, had to get a government approvals to get things. Mmm. So it was a good partnership with some of our um, get things where they need be in the right time. Ah. But it was probably a, I'll remember this quarter for a lot of different reasons. Um, and it worked out good for us. But uh, to be honest, it was, it is different from the other crisis's because it wasn't just a financial issue, which I think were just getting into actually, um, it was human and you saw different, two of our best regions were Italy and Spain that you think, Whoa, why? >>You know, you think the thing about other than going on in the quarter and but it was a relationship. It was, you know, we got our, the IB members got safe real quick, but then we quickly got them to engage with the clients, but we didn't Bush and was natural. Next thing you know that trust, I think there was a flight back to quality. You saw these different companies and that was the things they had to get done. Um, but it was, it was pretty amazing quarter to me. It was more seeing the team, you see your teams reacted. Crisis is in challenges in different ways and sometimes they paralyzed and we didn't see that at all in the team, which was pretty intelligent. Um, but we it coming from the beginning, call it before this we saw supply chain did, we came into Q1 hot on supply. So we kind of saw our early and we're already doing drills. So we saw it kind of right when it was hitting. Okay. >>But it was interesting you used the term interesting the challenging because it was sort of not only day to day for you, it was probably like minute by minute, hour by hour, country by country, region by region. How did you change the way in which you communicated to your teams or did you >>well so quickly? Um, so one I think culture, so I've been in a couple different companies, big and small. Mmm. I've seen different cultures react and the IBM culture is one that I've, I kind of look back and on this last quarter just because it's very customer intimacy. You don't have to, if the customer's in trouble, you can't stop them from running to help the clients. So we saw a natural, you know, we, IBM made sure they oil is refined to have one at home. Well we saw them quickly go after it. So most of it, any indication you do see it if these crisises um, you see some groups kind of freeze and, and you have to kind of walk them through it and make sure one, they're okay. This, this one was different yet to make sure your team was okay. Um, both mentally, physically, and their families. >>And it was a different stress level, was very personal and effected all of them. Where are the financial crises? In fact, it didn't affect everyone as much. It was more sterile. Uh, this one was wow, really different from a leadership. Um, but it's all the same. You have to get the team together and make sure they're healthy, happy, a healthy and mentally healthy too. And then you have to get people to kind of how do you go drive and help clients out. In this case it was helping you make sure your clients are okay, they're healthy, and then what can we do to help them? And I think that became more natural. And then of course it's Viber, Katelyn's drive, the business supply chain, which is I would say with any of the different um, challenges. But it's all communication. Well on this one, it was really had to check with the team often. >>We also had this new normal, I call this the new abnormal, which, you know, all of a sudden you can't meet with people so you couldn't get people physically together. So I call abnormal cause we're still, we'll get to the new normal, we'll use a lot more remote type of communication. But it was, I've never been so busy and I'm on video calls with all my teams every day. You see people using different tools to communicate like Slack, but also a lot more video. Uh, so it's communication, communication, which is the same thing. It's all the same thing with teams getting together, getting your direction. Well in this one it was mixture. They're safe first and then move on. Same thing with clients. Make sure let's say. Yeah. And that was what was fundamentally different about this. Um, Hey, what's up? Yeah. You know, and we were both grinders. >>I always joke, I work a half day every day. It doesn't matter which 12 hours the same way I have it twice. I'd take 12 hour days in a heartbeat these days. I mean, it's just really been crazy and I have to agree that the teams around the world at our, at our client space, of course the cube teams have barely really stepped up. But I want to ask you about the quarter. You're right. You came in hot in December, meaning you had a really good Q4. I, you know, I reached out to Tom Rosamilia last week, members said, Hey, nice announcement. And he said, did you cover it? I said, I did. And I sent them my breaking analysis. I, I really dug into the life cycles of the Z and how it affects, you know, IBM's overall business. And I predicted this is going to go on for several quarters where IBM has done a real chill tailwind, not only in, in systems hardware, but also, you know, the storage piece of the system's hardware business. >>We saw that last 40 accrued 19% and storage 60. Yeah. In, in, in Z hardware. Pretty amazing what's going on. Unpack. Okay. The quarter for us a little bit. Yeah. So if it wasn't for the crisis, I think all that would be plate. We had some announcements okay. Across the entire source portfolio. So what we do for storage for Z big announcements in Q3, uh, directly aligned with what we do with the new store. You know, the new Z, uh, you get a lot of value. One-on-one is three. So a lot of senators, I think it's different platform. So hit the demand and what clients are trying to do. Mmm. Bring a new, you know, uh, cloud development platforms, you know, native cloud development, but also using cloud. So there's a whole bunch of different things we brought to that platform. But we also launched new AI platforms, so stores for AI and big data. >>Uh, and then it the one we launched our new distributor. So we're kind of coming in from an offering set in fact water, uh, you know, 19% growth. Um, I think it's like speaks volumes no on the offering. Yes. But more how are we reacting to their clients more than anything else? I think it was a, Ashley's I talked about earlier, it was an interesting quarter. I think it's clients were responding to the flight equality, but also who's engaging with them the right way. So we do have a company absolutely refresh offerings across. In fact, this quarter, every single one of our offerings, every single new offerings group. Yeah. It's more of a, if you have the right offerings meet in the market, helping them with it, it correct after two, right. Your own journey. The cloud, moving, modernizing your environment. We need to free up our teams. >>We did a dramatic simplification on but what we do with storage or Z, but also distributed storage and what we do for storage AI and a big focus on cyber resiliency. Those are hitting what I'll say the market was in Q4 but they happen to also be hitting the market for what's going on now the noodles. So a lot of the simplification was that, how do you remote manage, how do you do things? One of the biggest things we do to our clients is, and we have all these tools, we give you a lot of things for free baseline, but we also have these increase the pro versions. We're just said, take them, I use them because it allows you to monitor and manage your environment better remotely. It was all web based. Uh, and that was one of the biggest things to do. But that is hidden the market. >>That's, that's the new normal. And we did that across those Z distributed storage. Mmm. But also what we did in a cyber resiliency in AI. I want to hit on a couple of those points. I mean, I'm going to start with the cyber resiliency because we were one of the first to report with our, with our partner ETR, our data partner that the work from home offset it was somewhat cushioning the downturn. I mean it's ugly, but chill worked from home pivot and that included, uh, uh, solutions around ransomware, data protection, cyber resiliency. So yep. Investment, actually 20% of the CIO is that we surveyed actually by not spending more in 2020, because of there wasn't zoom and WebEx, it was, there was other infrastructure around it, VDI, et cetera. So you're seeing that, uh, it sounds like, well, maybe talk a little bit about, so the cyber resiliency, and I'm especially interested in the context of going forward, feels like this is going to be one of those permanent things. >>You know, clients might sacrifice some near term profitability to have more flexibility and resiliency in their business and not rely so much on just narrow dr but more business continuance. No, I think you agree. In fact, um, we've always been, you know, a leader in business continuance. We still are. But cyber resiliency is yes. What a million different factories hovering from a ransomware or uh, um, you know, malware incident is different fundamentally different tool sets than what you're doing. You need to have a copy of your data of course, but very different than when if you were dr single server come up and running. Okay. You see us and mostly I think we're ahead of it because as IBM, we're the largest outsource firm in the world. So we actually live with these incidents as IBM. So in normal storage you hear about them and typically it's a storage issue. >>That issue that came back running. We are living with what we do or how to, our storage or outsourcing or strategic outsourcing group. And so we're putting into all of our products a lot of unique things from cyber resiliency. So what we did for storage for Z, it literally is a safe card. Copies an offering that little gifty 500 recover points. Yeah. Separated administratively and physically. So you're really able to literally, internal and external threats, protect yourself best in class. No one else has a solution set. We did the same thing and distributed. So, but in distributed, what we're trying to do is help people, not only, I used the term, left the boom and right up, boom, left the boom is before incident. How do you prepare? How do you have the right backup recovery? How do you have the right tool sets? Recover points? >>How do you protect yourself? How do you make sure you're um, you know, monitoring for ransomware? Every single night we'll get back power tools. Okay? The right of boom is once you do get hit, you go into this incident response situation where eyes drawn, your lights are on you. How do you give the humans, uh, the right cool. So they can react the right way and be quick. So also storage plays a huge role with ransomware and malware. Also. You get into, all right, the boom hits, you get the call, it's from the CEO. You got to fix it. You need new tools. Right? What recover point do you go back to? Um, it's iterative in nature. Uh, well yeah, it hit on, I got a call on Friday, but I don't know when the malware got and it was a Wednesday or Tuesday. It might be different per system. >>It's an internet process. You need the right tools, you use all your copies, primary storage, secondary storage for sure. He copies VR copies and find out what's your best recover point. And it's imperative you have to Lily bring up environments, you have to have fence network capabilities and all your tools to allow you to literally bring them up quickly in succession or altogether find them. That's recover point you get to as soon as you can. So those are the things I think we're leading. And we launched all this before this issue. Well we also saw an increase in malware in our client set. So to be honest, you know, even with all this crisis that we're seeing an increase and in malware, ransomware is where the storage infrastructure layer really matters in the incident response capability where if you have an incident, someone stole your data sets and typically storage guys that they call now IBM has great solution sets around their AI, direct driven. >>The ability is to allow you to protect yourself there. But this is on ransomware. It's something that storage plays a huge role. We do undistributed we do on mainframe with specialized solution sets. No one else in the industry is doing that. And of course back, uh, and recovery. Yeah. Quick recovery and orchestrated fashion. That's what we do around spectrum protect all day long. Right. Okay. Yeah. Last time we met. Oh, okay. You shared with us your, your consolidation strategy, your big, you know, announcement, uh, last fall, uh, and obviously, you know, great board or 90% growth. Well, a lot of that was drafting off the Z and the, you know, the hundred, but, but I'm wondering how that, how that consolidation work. We talked about the challenges of doing that know yep. The importance of that, how others are going to have to respond. And we're seeing that in the industry for a lot of the large portfolio players. >>But how did that, you know, how's that going? Can you give us, what, can you tell us about the progress there terms of its uptake and adoption? Sure, sure. So really what we did is we kind of looked at the industry and said everyone's adding too much complexity. You know, the whole industry is based on having a high end mid range and low end storage environment and the high end did everything custom and silk concrete performance, but you had to pay a price for it. And then the whole industry is based upon just get each of the next gen. So if you're a high end about problem is every client has high end, mid range and low in storage. So you have dual vendor strategy, but what you do is you have to, the whole industry is just getting to the next high end. Uh, you see EMC, Dell hashtag next generation, midbrain storage, the whole industry, including in the past, IBM was structure and getting you there. >>So we basically announced no more of that. Doesn't make sense. It used to, it no longer makes sense. We drive a lot of innovation what we're doing with Silicon, but software and we need to one platform, one platform that allow you at different price points down the stack from low end, mid range and high end, well without compromise. What's a dramatic simplification, right? Uh, that was a well-respected, you know, I would say we got an unbelievable response from that. And you saw a dramatic growth. So you kind of hit upon, we grew across all of our segments. Yes. We had a good growth on what we do for stores for Z. Well, we had an equally good growth at, as we did on distribute storage. So if you have physical environments, virtual environments, VMware, hyper V containers, public cloud, hybrid cloud, our distributed storage portfolio. >>So one of the biggest increases. Mmm. And we, again, we grew in every one of these segments. So one the simplification. Okay. Chapter two, how do you free up your team? How do you modernize your applications so you can innovate? Mmm. critical. You're free of your team. So that one thing that we also did a lot of, you know, Billy do remote management. I made it very simple to use Mmm. And simple to support, which also helps them the new normal, but it hit the right tone with it, our partners, but also our clients. And you saw a pretty massive uptick after the February announcement. So it was only half a quarter. We saw quite a large lift. I want to ask you about the storage for big data and AI as well. There seems to be a new emerging workload. You got all this data out there collected and Hadoop and analytics over the last 10 years. >>Now you're applying, we've talked about this, the new innovation cocktail. You got data AI and okay, well it gives you the scale whether it's on grammar in the public cloud, uh, but there seems to be a new workload where you get up what kind of a data store. You've got the analytic workloads that are in there. You've got some data science tooling, uh, and other, you know, AI that, that seems to be an emerging workload beyond, um, just kind of infrastructure as a service. But okay, really new way to get insights out of data, data, wonderful insights or not yet. So talk about that workload and how that is, is powering your business. And what are you seeing there? Well, I think this is where I see IBM, uh, really I'm helping clients with this journey to building smarter businesses cause AI is going to be in every workload. >>You're bringing up very specific workloads around machine learning, learning, bring customer on Silicon, like GPS into it, on these big data Lake. Uh, how do you take a data swamp and make a data Lake? Um, okay. Uh, what I'll say is IBM's doing this and we use the term ladder, the AI, and there's no AI without IAA information architecture. You have to have the right infrastructure to do it. We also see different groups having random acts of AI, a data scientist and the visionary does something is kind of interesting. Another group does something interesting and maybe a third. It's like the early days of data warehousing, but they're not able to take it together and bring it to, they can infuse AI across all the processes in a company and have one single view of the truth. Do we see people going through this natural progression, some start independently, a fight technology then bring it together. >>So everything we're doing from, I'll talk about what we're doing is storage infrastructure servers, but also across what we're doing, you know, are um, Mmm cloud pack for data offering and make it very simple for you to pull and get the use case out of it. But for storage is about when you want to bring it together, you need the right performance. But we bar none have the best source for AI. And data. It's based upon our, you know, Lily, um, award-winning. Yeah. Scale up a file system called GPFS or spectrum scale. It runs the largest AI supercomputers in the world. The same as X software, but you can buy it to your device that we launched it in December, which is feller. ESS, um, 3000 is a single all flash array. It's a cluster, but you can no compromise. You go from that device and the largest AI supercomputer in the world configuration, exact same technology, hardware and software that we do. >>Floyd. So now you can start small and grow and then we're helping along. How do you get the value out of it? So that's typically where storage ends. I gave you the best platform you can possibly have, cost effective, small, and you can scale to the biggest thing you want to do. The next thing we're doing, which people say, well that's not storage and why are you doing that? We're doing things called spectrum discover. It's managing your metadata and making your data scientists the most productive possible. They spent any 80% of the time literally just understanding the data, tagging the data, organizing it so they know what they're doing with, cause if you don't have the right AI data sets, you really can't get the outcome. Okay. But we have what's called spectrum discover works across a whole bunch of other products, but also all of our portfolio, both object storage file system block allows you to look at an environment, organize it, and save dramatic amount of time for data scientists. >>And of course that's easy feed into all the things we do around cloud pack for data, which is where IBM has really put a lot of these open source and our own tools together so you can move forward pretty quickly. The key thing is how does IBM help you not technology. We know what you want to accomplish, let's help you but not limit you by we're letting you use all the different open source. Yes. I just want allow you to move forward and help you in that journey. And it is a journey and we're meeting clients where they are because everyone's on it different. Yeah. I guess segment of the journey and how do we help you go through it and from a storage, uh, you're seeing that environment really double every quarter. Mmm. For the people that are looking for it, no one really touches us. >>Mmm. In fact, our number two and three customers, Mmm. Competitors in the space use the same software that we OEM so we're in a very good position when it comes to stores for AI, big data. So they say it's better to be lucky than good. I say it's, it's better to be good and lucky. And so, you know, we're not going back it's not happening. we've got this new abnormal, as you call it, and you've done a lot of the hard work in terms of rationalizing the port folio. You've done the R and D and you started this years ago and it took a long time. Mmm. But I wonder if you could just talk about why you feel like you're in a good position coming out of this thing and who knows how we're going to come out of it, but what are the critical components that you feel you have in your arsenal that will make you stronger and more competitive or you know, relative to, you know, the, uh, the landscape out there, your thoughts? >>Yeah. So now this is going to sound, uh, well good. So all these different issues we've been through all these Bryce disease we've been through in our careers. Um, there's an old adage, if you can last room and you get resourced, you can come out stronger. And it's very true. So you can grow, you can do the right things, but you have to have the right offerings. Sometimes that's low, lucky you entered, right? I think we perfectly with the right innovation that did take us years ago, but we're hitting the current market. But also what I'll say is the new normal market. Mmm. And I think that's an opportunity. And I've always said, listen, the world doesn't need another storage. Right. Well, they're looking for solutions around the source challenges and I think what we've done around product portfolio with, we use the term offerings was the offerings around it with a different software allows you to actually, we're really free, you know, if it's really chapter two now we're trying to do monetize your core infrastructure, you need to free up your team so they can innovate. >>We're going to do that dramatically in what we're doing. Storage, they help you with that journey to cloud either OnPrem or into the public cloud or really what we see is a hybrid multicloud fabric happening, but also we do cyber resiliency as we built it from the or. So I think we're good hitting it, right? Mmm. Now the new normal is all the things that it has to be simple, it has to be rope managed and those are all the things we made massive investments across every one of our portfolio items. They just got launched a launch in the last two quarters. So I think we're in good stead. But to be honest, in these times, as we talked earlier, you work harder. You've got to really embrace the client feedback. Mmm. I think IBM is a good position to do that. Also with the greater IBM, we see vigor, Mmm. Opportunity set to find out how to help clients. >>Okay. We're the number one AI company in the world. So we're seeing what clients really want to do with AI and how they. There's actually holding it back. Number one outsourcer. We're seeing how people are really dealing with cyber resiliency and especially now where ransomware, where storage really impacts you. We're seeing exactly how to do it and what tools push forward and that's where you're seeing very unique opportunities in these times. If you can have the right product, the right go to market and do very well and more importantly you'd do it by helping clients. If you can help clients through this, do you come out stronger? I think some other people's storage, it becomes more challenging. I don't think people just want you know, the next flash array. I think they're looking for solution sets a companies to help them get through and get to the really the new, I think we're going to get to the new normal. I think this is a new abnormal, I can't call it normal. When we're all locked away, the new normal is going to be much faster. You're gonna have to go faster. So I think IBM and the IBM storage is aligned with let's help you with the cloud journey. Let's help you build our businesses. We'll make sure cyber resiliency built in there. Well, we're going to, you're seeing it across every division of IBM, step up and help you in that. Mmm. In that direction. That's what I think is differentiated. Why I'm excited about >>what we're doing. IBM in general, but also, yeah, again, storage is perfectly aligned with that overall mission and it's, it's kind of exciting to see it kind of play out in front of class. Well, I think you're right. I think the last decade was a lot of, it was about the all flash data center and, and the future is about powering innovation infrastructure for machine intelligence. Uh, and, and really getting insights out of data scaling. Uh, ed ed Walsh. Always great to have you on the, uh, hopefully we can do this, you know, a little closer face to face, maybe six feet apart. Um, and then eventually we could shake hands or high five or whatever it works. Thanks so much for coming to the Cuba. It's great to see you looking good and stay safe. Hey, thank you. Stay safe. All right. And thank you for watching everybody. This is Dave Volante for the cube and our continuous coverage of the IBM, that 20, 20 digital events experience. I'll be right back. Sorry for the short break.

(upbeat music) >> From the Silicon Valley Media Office in Boston Massachusetts, it's theCUBE. Now here's your host, Dave Vellante. >> Hello everyone, and welcome to this exclusive CUBE conversation. Here's the setup. The storage industry has been drowning in complexity for years. Companies like Pure Storage and Nutanix, you know they reached escape velocity last decade, primarily because they really understood well how to deliver great products, that were simpler to use. But as we enter the 2020's, virtually every player in the storage business is trying to simplify it's portfolio. And the mandate is coming from customers, that are under huge pressure to operationalize and bring to market their major digital initiatives. They simply can't spend time managing infrastructure that the way they used to. They have to reallocate resources up the stack, so to speak to more strategic efforts. Now, as you know post the acquisition of EMC by Dell, we have followed closely, and been reporting on their efforts to manage the simplification of the storage portfolio under the leadership of Jeff Clark. IBM is one of those leading companies, along with Dell EMC, NetApp, and HPE that are under tremendous pressure to continue to simplify their respective portfolios. IBM as a company, has declared the dawn of a new era. They call it Chapter II of Digital and AI. Whereas, the company claims it's all about scaling and moving from experimentation to transformation. Chapter II, I will tell you unquestionably is not about humans managing complex storage infrastructure. Under the leadership of General Manager, Ed Walsh, the companies storage division has aligned with this Chapter II vision, and theCUBE has been able to secure an exclusive interview with Ed, who joins me today. Great to see you my friend. >> Thanks very much for having me. >> So, you're very welcome. And you heard my narrative. How did we get here? How did the industry get so complex? >> I like the way you kicked it off, because I think you nailed it. It's just how the storage industry has always been. And there was a reason for it twenty years ago, but it's almost, it's run its course, and I could tell you what were now seeing, but everyone there's always a difference between high end solutions sets, and low end solution sets. In fact their different, there's custom silicon on the high end. So think about EMC Matrix in the day, it was the ultimate custom hardware and software combination. And then the low end storage, well it didn't have any of that. And then there's a mid tier. But we actually, everything is based upon it. So you think about the right availability, the right price port, feature function, availability features. It made sense that you had to have that unique thing. So, what's happened is, we're all doing sustaining innovation. So we're all coming out with the next high end array for you. EMC's next one is Hashtag, Next Generation storage, right, mid-range. So they're going to redo their midrange. And then low end, but they never come together, and this is where the complexity is, you're nailing it. So no one is a high end or a low end shop, they basically use it all, but what they're having to do, is they have to manage and understand each one of those platforms. How to maintain it, it's kind of specialized. How to report on it, how to automate, each the automation requirements are different, but different API to actually automate it. Now the minute you say, now help me modernize that and bring me to a hybrid multi-cloud, now you're doing kind of a complex thing over multiple ways, and against different platforms, which are all completely different. And the key thing is, in the past it made sense to a have high end silicon with high end software, and it made sense. And different low end, and basically, because of some of the innovation we've driven, no longer do you have to do that. There's one platform that allows you to have one platform to meet those different requirements, and dramatically simplify what you're doing for enterprises. >> So, we're going to talk a little bit more about what you guys are announcing. But how do you know when you get there, to this land of simple? >> One it's hard to get there, we can talk about that too. But it's a, when a client, so we just had a call this morning with our board advisor for storage, our division. And they're kind of the bigs of the bigs. Up on the need, more on the high end side, just so you know the sample size. But literally, in the discussion we were talking about the platform simplification, how do you get to hybrid cloud, what we're going to do with the cyber incident response type of capabilities have resiliency. And literally in the call they are already emailing their team, saying we need to do something more strategic, we need to do that, we need to look at this holistically. They love the simplicity. Everything we just went through, they can't do anymore. Especially in Chapter II, it's about modernizing your existing mission critical enterprises, and then put them in the context of Hybrid multi-cloud. That's hard, you can't do it with all these different platforms, so they're looking for, let me spend less. Like you said, to get my team to do up-stack things, they definitely don't want to be managing different disparate storage organizations. They want to move forward and use that freed up resource to do other things, so. When I see big companies literally jumping at it, and giving the example. You know I want to talk about the cyber resiliency thing, I've had four of those this week. That's exactly what we need to have done, so it's just, I haven't had a conversation yet that clients aren't actually excited about this, and it's actually pretty straightforward. >> So I'll give you the benefit of the doubt, and again we'll get there, but assuming your there. Why do you think it took you so long? You kind of mentioned it's hard. >> So, transformations are never easy, and typically whoever is the transformation engine, gets shot in the back of the head, right. So it's really hard to get teams to do something different. So imagine every platform, EMC has nine now, right. So it is through acquisition of others, you have VP's, you know. VP of development, offering and maybe sales, and then you have whole teams, where you have founders you've acquired. So you have real people, that they love their platform, and there's no way they're going to give it up. They always come up with the next generation, and how it's going to solve all ills, but it's a people transformation. How do you get we're going to take three and say, hey, it's one platform. Now to do that it's a operational transformation challenge. It's actually driving the strategy, you don't do it in matter of a week, there's development to make sure that you can actually meet all the different use cases, that will take you literally years to do, and have a new platform. But, I think it's just hard to do. Now, anyone that's going to do that, let's say you know EMC or HP wants to do it. They're going to have to do the same thing we did, which is going to take them years of development. But also, it's managing that transition and the people involved, or the founders you've acquired, or it just it's amazing. In fact, it's the most wonderful part of my job is dealing with people, but it can frustrate you. >> So we've seen this over the years, look at NetApp, right with waffle, it was one size fits all for years, but they just couldn't cover all markets. And then they were faced with TAM expansion, of course now the portfolio expands. Do you think -- >> And now they have three and -- >> And David Scott at HPE, Storage VP at the time used to talk about how complex EMC's portfolio was, and you see HPE has to expand the portfolio. >> We all did, including IBM. >> Do you think Pure will have to face the same sort of -- >> We are seeing Pure with three, right. And that's without the file, so I'm just talking about what we do for physical, virtual, and container workloads and cloud. If you start going to what we're going to scale up to object we all have our own there too. And I'm not even counting the three to get to that. So you see Pure doing the exact the same thing, because they are trying to expand their TAM. And you have to do some basic innovation to have a platform actually meet the requirements, of the high end requirements, the mid range, and the entry level requirements. It's not just saying, I'm going to have one, you're actually have to do a lot of development to do it. >> All right, let's get to the news. What are you guys announcing? >> So basically, we're announcing a brand new, a dramatic simplification of our distributed storage. So, everything for non-Z. If you're doing physical boxes, bare metal, Linux. You're doing virtual environments, VMware environments, hyper-V, Power VM, or if you're doing container workloads or into the cloud. Our platforms are now one. One software, one API to manage. But we're going to actually, we're going to do simplification without compromise. We're going to give you want you need. You're going to need an entry level packaging, midrange and high end, but it's going to be one software allows you to meet every single price requirement and functionality. And we'll be able to do some surprises on the upside for what we're bringing out to you, because we believe in value in automation. We can up the value we bring to our clients, but also dramatically take out the cost complexity. But one thing we're getting rid of, is saying the need, the requirement to have a different hardware software platform for high end, midrange and low end. It's one hardware and software platform that gets you across all those. And that's where you get a dramatic simplification. >> So same OS? >> Same OS? >> Normally, you'd do, you'd optimize the code for the high end, midrange and low end. Why are you able to address all three with one OS? How are you able to do that? >> It took us three and half years, it was actually, I will talk about a couple innovation pieces. So, on the high end you have customized silicon, we did, everyone does, we had a Texas Memory Systems acquisition. It was the flash drawer 2U, about 375 TB, uncompressed de dup, pretty big chunky, you had to buy big chunks. So it was on the high end. >> That was the unit of granularity, right. >> But it gave you great value, but also you had great performance, latency better than you get in NVMe today, before NVMe. But you get inline compression, encryption, so it was wonderful. But it was really ultra high end. What we did was we took that great custom silicon, and we actually made it onto what it looks like a custom, or to be a standard NVMe SSD. So you take a Samsung NVMe, or a WD and you compare it to what we call our flash core module. They look the same and they go interchangeably into the NVMe standard slot. But what's in there is the same silicon, that was on this ultra high end box. So we can give the high end, exactly what we've did before. Ultra low latency, better than NVMe, but also you can get inline compression de dup and the were leveling, and the stuff that you expect in the custom silicon level. But we can take this same NVMe drive and we can put it in our lowest end model. Average sale price $15,000. Allows you to literally, no compromise on the high end, but have unbelievable surprises on the midrange and the low end, where now we can get the latency and the performance and all those benefits, to be honest on a much lower box. >> Same functionality? >> Same functionality, so you lose nothing. Now that took a lot of work, that wasn't easy. You're talking about people, there was roadmaps that had to be changed. We had to know that we were going to do that, and stick to our guns. But that'll be one. Other things is, you know you're going to get some things on the upside that you're not expecting, right. Because it's custom silicon, right, I might have a unique price performance. But also cost advantages, so I'm going to have best price performance or density across the whole product line. But also, I'm going to do things like, on the high end you used to unbelievable operational resiliency. Two site, three site, hyper-swap, you know two boxes that would act like one. Have a whole outage, or a site outage and you don't really miss a transaction, or multi-sites. But we're going to be able to do that on the low end and the midrange as well. Cyber resiliency is a big deal. So I talked about Operational Resiliency. It's very different coming back when it's cyber. But cyber incident response becomes key, so we're going to give you special capabilities there which are not available for anyone in the industry. But is cyber incident response only a high end thing, or is it a low end thing. No, it's across everywhere. So I think we're going to shock on the upside a lot of it, was the development to make sure the code stack, but also the hardware, we can at least say no compromise if you want entry-level. I'm going to meet anyone at that mote. In fact, because the features of it, I'm able to compete at an unfair level against everyone on the low end. So you say, midrange and high end, but you're not losing anything because your losing the custom silicon. >> So let's come back to the cyber piece, what exactly is that? >> All right, so, listen, this is not for data breaches. So if a data breach happens, they steal your database or they steal your customer name, you have to report to, you know you have to let people know. But it's typically than I call the storage guy and say hey, solve it. It was stolen at a different level. Now the ones that doesn't hit the media, but happens all the time actually more frequently. And it definitely, gets called down to the operations team and the storage team is for cyber or malicious code. They've locked up your system. Now they didn't steal data, so it's not something you have to report. So what happens is call comes down, and you don't know when they got you. So it's an iterative process, you have to literally find the box, bring up, maybe it's Wednesday, oh, bring it up, give it to application group, nope, it's there. Bring up Tuesday... it's an iterative process. >> It's like drilling for oil, a 100 years ago, nope, not it, drill another hole. >> So what happens is, if it's cyber without the right tools, you use your backup, one of our board advisors, literally major bank, I had four of those, I'll give you one. It took me 33 hours to bring back a box. It was a large database 30 TB, 33 hours. Now why did you backup, why didn't he use his primary storage against DR copies of everything. Well they didn't have the right tool sets, so what we were able to do is, tape is great for this air gap, but it takes time to restore and come back up and running. The modern day protection we have like Veeam or Cohesity allows the recovery being faster, because your mounting backup copies faster. But the fastest is your primary snapshot and your replicated DR snapshots. And if you can leverage those, the reason people don't leverage it, and we came upon this, almost accidentally. We were seeing our services brethren from IBM doing, IBM SO or outsourcing GTS, when they did have a hit. And what they want to do is, bring up your snapshots, but if you bring up a snapshot and you're not really careful, you start crashing production workloads, because it looks like the VM that just came up. So you need to have, and we're providing the software that allows you to visualize what your recovery points are. Allows you to orchestrate bringing up environments but more importantly, orchestrate into a fenced network environment, so it's not going to step on production workloads and address this. But allows you to do that, and provide a URL to the different business users, that they can come and say yes, it's there or it's not. So even if you don't use this software before this incident, it gives you visibility, orchestration, and then more importantly a fence, a safe fence network, a sandbox to bring these up quickly and check it out, and easily promote to production. >> So that's your safe zone? >> Safe zone, but it's just not there. You know you start bringing up snapshots, it's not like a DR case, where you're bringing things up, you have to be really smart, because you bring it up, and checking out. So without that, they don't want to trust to use the snapshots, so they just don't use primary storage. With it, it becomes the first thing you do. Because you hope you got it within a week, or week and a half of your snapshots. And it's in the environment for ninety days, now you're going to tape. Now if you do this, if you put this software in place before an incident, now you get more values, you can do orchestrated DR testing. Because where doing this orchestrated, bring up application sets it's not a VM, it's sets of VM's. Fenced network, bring it up, does it work. You can use it for Test/Dev data, you can use it for automatic DR. But even if you don't set it up, we're going to make it available so you can actually come back from these cyber incidents much faster. >> And this is the capability that I get on primary storage. Because everybody's targeting you know the backup corpus for ransomware and things of that nature. This is primary storage. >> And we do put it on our backups. So our backups allows you to do the exact same thing and do the bootable copies. And so if you have our backup product, you could already do this on primary. But, what we're saying is, regardless of who your using, we're still saying you need to do backup, you need to air a cup your backup. 'cause you know Want to Cry was in the environment for 90 days, you know your snapshots are only for a week or two. So the fact of the matter is that you need it, but in this case, if you're using the other guys, you can also, we're going to give it just for this tool set. >> How does immutability does it factor? I know like for instance AWS Reinvent they announced an immutability capability. I think IBM may have that, because of the acquisition that you made years ago, Clever Safe was fundamental to that, their architecture. Is that a way to combat ransomware? >> So immutability is obviously not just changes. So ransomware and you know malware typically is either encrypting or deleting things. Encrypting is what they do, but they have the key, so. The fact of the matter is that they're deleting things. So if it's immutable, than you can't change it. Now if you own the right controls, you can delete it, but you can't change it, they can't encrypt it on you. That becomes critical. So what you're looking for, is we do like for instance all of our flash system allows you to do these snapshots, local or remote that allow you to have, go to immutable copies either in Amazon, we support that or locally on our object storage, or in IBM's cloud. It allows you to do that. So the different platforms have this immutability that our software allows you integrate with. So I think immutability is kind of critical. >> How about consumption models? The way in which your packaging and pricing. People want to, the cloud is sort of change the way we think about this, how have you responded to that? >> So, you hit upon our Chapter II. We, IBM, actually resonates to the clients. In Chapter I, we are doing some lift and shift, and we're doing some new use cases in the cloud. And they had some challenges but it worked in general. But we're seeing the next phase II, is looking at the 80% of your key workloads, your mission critical workloads, and basically how you transfer those in. So basically, as you look at your Chapter 2, you're going to do the modernization, and you might move those into the cloud. So if you're going to move into the cloud, you might say, I'd like to modernize my storage, free my team up, because it's simple, I don't have to do a lot of things. But you need to simplify so you can now, modernize so you can transform. But, I'm going to be in the cloud in 18 months, so I don't want to modernize my storage. So what we have, is of course we have so you can buy things, you can lease things, we have a utility model, that is great for three to five years. But we have now a subscription model, which think of just cloud pricing. No long term commitment. Use what you use, up and down, and if it goes to zero, call us we'll pick it up, and there's no expense to you. So, no long term commitments and returns. So in 14 months, I've done my modernization, you've helped me free up my team. Let me go, and then we'll come and pick it up, and your bill stops that day. >> Cancel at anytime? >> Yeah, cancel at anytime. >> Do you expect people to take advantage of that? Is there a ton of demand at this point in time? >> I think everyone is on their own cloud journey. We talk a lot about meeting the client where they are, right. So how do I meet them where they're at. And everyone is on their own journey, so a lot of people are saying, hey, why would I do anything here, I need to get there. But if they can modernize and simplify what they're doing, and again these are your mission critical. We're not talking, this is how you're running your business, if we can make it better in the mean time, and then modernize it, get it in containers, get it into a new platform, that makes all the sense in the world. And because if we can give them a flexible way, say it's cheaper than using cloud storage, like in Amazon or IBM cloud. But you can use it on-prem, free you up, and then at anytime, just return it, that's a big value that people say, you know what, you're right, I'm going to go do that. You're able to give me cloud based pricing, down to zero when I'm done with it. Now I can use that to free up my team, that's the value equation. I don't think it's for everyone. But I think for a segment of the market, I think it's critical. And I think IBM's kind of perfectly positioned to do it with a balance sheet to help clients out. >> So how do you feel about this? Obviously, you've put a lot of work into it. You seem pretty excited. Do you feel as though this is going to help re-energize your business, your customer base, and how do you think competitors are going to respond? >> Good question. So, I think simplification, especially we can talk about value equation. I think I can add more value to you Mr. Customer. I can bring things you're not expected, right, and we're get to this cyber in a second, that would be one of the things they would not expected. And reduce the costs and complexity. So we've already done this a couple of times, so we did it with our Mainframe storage launch in the fall. It bar none, the best box for that workload. Lowest latency, most integration, encrypt, pervasive encryption, encryption in flight. But also, we took it from nine variants, to two. Because we could. We go, why did you need all those, we'll there's reasons for it in the past, but no longer. We also got rid of all the hard disk drives. We also add a little non-volatile cache and allowed you to get rid of all those battery backups. All these custom things that you used to have on this high end box. And now it's dramatically simpler, better. And by the way, no one asked, hey what are my other seven variants went. It was simpler, it was better, faster, but then it was the best launch we've had in the history of the product line. It think we can add better value and simplify for our clients. So that's what we'll do. You asked about how people respond. Listen, they're going to have to go through the same thing we did, right. A product line has people behind it, and it's really hard, or a founder behind it. You mentioned a couple, they're acquiring companies. I think they're going to have to go this, it's a transformational journey, that they'll have to go through. It's not as simple as doing a PowerPoint. I couldn't come to you and say, I can simplify without compromise. I can help you on the low end, the midrange, high end with same platform unless I did a lot of fundamental design work to make sure I could do that. Flash core modules being one of them, right. So I think it's going to be hard. It'll be interesting, well, they're going to have to go through the same thing I did, how about that. >> Usually when you make a major release like that, you're able to claim Top Gun, at least for a while with things like latency, and bandwidth and IOP's and performance. Are you able to make that claim? >> So, basically you saw it in the launch today. But basically you saw the latency which is one, because we're bringing a custom silicon down, our latency you'll see like I'll give you Pure bragging on their websites, their lowest latency is 70 microseconds, which by the way is pretty, you know. It's gonna be 150 microseconds, pretty good bragging rights. We're at 70 microseconds, but that's on the X90 using storage class memory. So literally we are 2x faster than on latency, how fast can you respond to something. But we can do it not only on our high end box, but we can also do it on our average sale price $15,000 box. Because I'm bringing that silicon up and down. So we can do the latency, now EMC the highest and PowerMax box. Two big chassis put together, that can do 100 microseconds. Again, still we're 70 microseconds, so we're 30% faster. And that's epitomized of the high end custom silicon software. So latency we got it. IOP's, so look at the biggest baddest two boxes of EMC, they'll do you know 15 million IOPs on their website. We'll do 18 million IOPs, but instead of two racks, it's 8U. It is 12x better IOPs per rack space, if you want to look at it that way. Throughput, which if you could do, it's all about building for our businesses. It's all about journey of the cloud and building for our businesses, everyone's trying to do this. Throughput in analytics becomes everything, and we you can do analytics in everything. Your DBA's are going to run analytics, so throughput matters. Ours is for every one of our boxes, that you can kind of add up and cluster out, it's 45 Gb/s. Pure, for instance their bragging rights, is 18, and they can't cluster anymore. So what we're able to do is on any of the, and most of those are high end, but I'll say, I can do the same thing up and down my line, because of where I'm bringing the custom silicon. So on bragging rights, and that's just kind of website, big bragging rights, I think we got a cold, and if you look price performance, and just overall price per capacity, we're inline to be the most the cost effective across everyone. >> Yeah, up and down the line, it's very interesting, it's kind of unique. >> And then you mentioned resiliency, I'll tell you that's the hottest thing, so. You mentioned the cyber incident response, that is something that we did on the Mainframe. So, we did the last Mainframe cycle, we allow you to do the same thing, and it literally drove all the demand for the product sets. It's already the number one thing people want to talk about, because it becomes a you're right, I needed that this week, I needed it last week. So, I think that's going to really drive demand? >> What worries you? >> (laughs) On this launch, not much. I think it's how fast and far we can get this message out. >> Wow, okay, so execution, obviously. You feel pretty confident about that, and yeah, getting the word out. Letting people know. Well, congratulations Ed. >> No, thank you very much, I appreciate it. I appreciate you coming in. And thank you for watching everybody. This is Dave Vellante for theCUBE. We'll see you next time. (upbeat music)

>> Live from San Francisco. It's the cube covering IBM thing twenty nineteen brought to you by IBM. >> Welcome back to Moscow and everybody. The new, improved, shiny Mosconi Center. I'm Dave Lamont with Student of Men. This is Day one of IBM thinking you're watching the Cube, the leader and live tech coverage. Ed Walsh is here. He's the general manager of of IBM Storage and Software to find it. Great to see you again. Always. Pleasure. Thanks for coming on. I love the venue, You know, I agree. It's a saddle. Las Vegas. No offense to our friend. Thie shy. It's been a long time coming, Oscar, honey, but it looks really good. I agree. Three thousand people expected. So you must be excited. >> No, I think we have a lot of things that you're going to see announcements. Also, I think you're going to see some refinement of the overall message. I think it's going to exciting week. So it's kind of I'm I'm talking to before all the keynotes. But it's it's an interesting week, for sure. So >> so tease a little bit. What can you tell us is >> Okay, so you know my background, I've been outside IBM, coming to run storage,  I've said a couple of different times. My strategy is to drive the overall storage but also get more aligned with what we're trying to overall at IBM, because that's the strength of IBM, right? Really help the clients move forward And infrastructure matters, and what you're seeing is I think the market's coming our direction at IBM. And I'll give you a couple of things. You're gonna hear a lot about, you know, hybrid multicloud. Say AI at scale, right, and you're going to see that messaging, but that's where the markets come at us. You saw Red Hat talk about it, but all of our competitors are now doing that as well. But when it comes to once, you start a hybrid multicloud and we think it's like we're going to talk about that chapter two right? Chapter one was the first twenty percent of workloads and was all about these application driven events. And you know think about Office 365, etc. But the eighty percent of the workloads were still on premises, and there's a reason they're on premises. But what now is that people that next phase going leaving by the the organizations that have mission, critical data and how to do that? And there's a role for hybrid, which plays perfectly for us. >> And can you help help connect the >> dots for us because we've launched, you know, software to find wave come through through storage, but still many people on the outside, they'd be like, Okay, well, storage is a bunch of boxes sitting in my data center on all my new After being built here. I'm sass ify ing things. They're there, and it feels like death by a thousand cuts, too. The traditional storage markets help explain kind of what the modern storage market is. Data is at the center of everything, so we know that that's a huge thing that elwin for storage but help bring us inside your business. >> So I think in general everyone's trying to data driven, and it's easy to say hard to do and everyone at the platform going do that. It's a hybrid multi cloud and hybrid being the reason we're using the terminology, the industry. But also there is a rule for on premises. And how do you easily connect but getting the same, you know, agility and performance and cost benefit on Prem in an extended the right time for Cloud. So where you see us, we're looking, we focus on. We deal with a lot of clients somewhat advanced and some would say more laggards. And so we share loss stories where people are being successful being dad driven and we see it fall in tow. Kind of three. What I'll say is we try to success. Criteria are areas, right? So one is people just modernizing, going from traditional to go private clouds, making sure they extend use of benefits of public cloud and be more data driven. So some people are spending money to save money, and some people are spending money to make money. But even in traditional environments, you see the CEO having a bigger voice and we need to push him and show him how to do that right way. Also see another section, People really driving a I in general. We see that and definitely not hybrid people to cloud environment where we see a large on premises. But you're always looking for the different data. Sets are going to be in the multi cloud that they need. Bring it together and we see that being a very interesting and affected. How do you start with someone doing? We believe we have the best storage for a I, but we can scale from the smallest to the biggest super computer in the world. Right? Driving, You know, one point five terribly. It's a second, you know, huge monsters, but you can start small. But what we find is people are just getting started with the meets clients where they are so part of it in these different areas, meet them where they are, and there are different parts of the journey on AI is having them get going. But then really, how do you scale? You're gonna hear a lot about this. How do you scale AI? Everyone has these random acts of AI or machine learning, but they can't scale them for the business, let alone across the enterprise, which is where everyone needs to get to. And that's where we're really focused on the offering set. So from a storage that would be where we do the AI. And of course, the third one is just containers in general. Which, by the way, they intersect because a lot of things going to in containers going intersect with a especially when you go multicloud, but there's a whole different Hey, let me modernize my application infrastructure, and it's a different conversation with client. So we see people being very successful, and that's where you're seeing from a storage development. Investments is were going into that direction, helping clients those different. >> Why is a scale so difficult? Is it the silo data silo problem? >> So first of all, there's a coldness about a I everyone. It's a black box is mysterious. It's it's really just computer science. I mean, it's a process between time I'm eating their own deep learning. It is. You're you're doing stats, you're just driving you being in a river or that using custom, I silica do a faster, like abuse. Uh, the other one is it's easy, and it's not so in infrastructure matters. So when you get going, what we see is people just give a developed You give a particular data scientists on environments. You say You bring the data and you need and we'll help you with the governance of strategy, but still getting something to just be valuable to the business. Then what happens is they see another random act of a ay or another area that it becomes data silo, but they're trying their best to stay away from the data scientist. Given the right support, which I think the right thing is, the biggest thing is not having all controlled and centralized. You wantto let the business units drive. But then what happens is you have that almost like data warehousing in the past. You have these islands, and now you don't have any trusted, true source of the truth. And you don't have ability to get you force everyone to do the hard bit. The hard bit is actually having the right data. Do all the eighty percent of cleansing that dead guy having the right governance and security about that? Andi, you're adding to overtime. What if you do a couple applications and it's not on the first one? You do. But once you do the second third, maybe fifth deadly by the eighteenth application, you want to bring that together in a shared platform. And that's where IBM storage plays. And that's where we're truly differentiated. Compare the storage industry so we have no assets that no one else has. Like what we do. A spectrum scale where we can, Luli scale up from just individual server toe half rack and we can take the same environment into largest, eh? I submit computers in the world, but the key thing is, you need. You can have a without a information architecture, and that's on the software side. But it definitely has to be in the infrastructure, and we're doing a cross hybrid multi class. We're doing that on Prem. But trust me, eyes absolutely in the cloud so we can extend those environments and run the same thing in any of the public clouds. And >> what's the storage enabler eyes? It is its software, defined as you mentioned architecture. What is the linchpin there? >> Well, so one it is a softer to find. So in this particular area, where we're helping people is our file system called spectrum scale. But it allows us to do from the very small toe largest environments, right that allows you to scale, and it's also runs all different asset. So it's unstructured, be able to run a dupe Native spark. But you have the file you're able to block, able to bring it together, able, start small, but you're able to scale and keep up the thing about a eyes you go from, I want to collect that information. Get after it. You also need metadata. So we have products like Spectrum Discover to show you the metadata so we can actually track it, you know, So it doesn't become too junk drawer in the sky that we've seen with that a lot of data lease. But then it's interesting. You have to go in tow, actually do the training, and that's for using custom silicon. You know, G pews, and that dramatically changes a performance you require. So thes GP is used to run it sixty gigabytes a second. How they're one hundred fifty gigabytes a second. We no storage store. Traditional stories that you get from, you know, the environment we might get from pure net after emcee. They don't run it, run it sub twenty gigabytes a second. So how do you do this? It's a different architectures, actually, based upon a true scale out what we've seen in the largest super computers in the world. But you're able to bring that environment so we can actually do bring in all the data works, get under one governance and strategy. But then you can actually keep up with the performance of the true influencing and driving GPS that once you have a trusted source, you can scale us out. Lily, the largest, super covers the world so we can We can show you scale on the exact same components started half rack and goto. The biggest thing is in the world, but the key thing is right. You need to actually have a performance. So if you have this data back, plaintiff, you call that Now people still take a lot of the data. They'll bring it into the servers in the crunch it with GPS. They say, Well, okay, your stories doesn't need tohave that performance. But what you find is once you have a common back plane, which, how IBM did it. Now you have different business units almost hub and spoke, grabbing the data. But they have one true source of data they will get after it. They're able to get their other data that other groups are looking for. But now they're able to now scale it into the enterprise that because something is just a I call a pike, also your doing applications and they just want to have a P. I called in the same environment, and those have to be fast, cause now you're influencing, so it's sub seconds. But you need that performance. So what you need is by bring it all together. You can either do data silos, which are easy up front. But by the time you the second third, you do all that same work over and over, and you don't trust its source. You gotta bring it together. But you can't bring it together in any storage, and we don't bring it together on the same storage we do for of'em, where environment to others is different storage. And it's made specifically for this environment. And it's something that, actually IBM is no leaps and bounds above everyone supercomputer. They do these type of analysis, and we're like two x our best competitors benchmarks, by the way, that computer use spectrum scale so But it is a different architecture. But if you don't put together on, and I would also say that when you get started, no one starts with the big, in fact, that that's almost a mistake. What you want to do is let the data scientists have the creative driving get business outcome, but they need to be thinking ahead. How do you bring it together? So you have a shared because again, The way you're really gonna drive across enterprise. All that processes is actually having soon AP I calls come in and which are not going to their own environment altogether. Makes sense. >> Yeah, you've mentioned a couple times infrastructure matters, and I wantto wanna tie that into the eighty percent Sure it was at this very venue in two thousand nine when Paul Marat said is the CEO of the M where we're going to run any workload. Any application? Virtualized and a lot of people were skeptical, and I remember the time thinking about mainframes. I kind of did that. Um, >> and you can say that I can >> How you're talking about the eighty percent and, you know, Veum, where I think largely proved that that you could run that at high performance. Att. Least adequate performance. Now you're seeing a similar discussion around cloud. But it's somewhat different because of some of the things that you were just mentioning it. What does that world look like? Obviously, hybrid fits in. You mentioned the red Hat acquisition. That's key. Part of idea mes go forward. You know, multi cloud strategy. So should we think about what you know what similar and what's different than a sort of V M wear virtual ization, mainframe virtual ization days. >> Okay, that's interesting. And then you're tying into the cloud adoption as well, right? So and we do think about twenty in this idea. See, about twenty percent of workers have gone, but eighty percent are waiting, and that's that I never thought of that way. It's a good analogy. Virtual ization. That easy stuff went first, and then what you have to do is have the databases. Remember, that was a big issue. You couldn't do that for years, but then also, you move. It's like, Why would you do it? But what's happening is what we're seeing. Is this the mission? Critical workloads, And they're either regulated industries, but it's for different reasons. They're running in different places. But it might be a security concern. It might be scaled that he might be regulated industries, but there's reasons. Or maybe after reef, actually applications. Actually one cloud native because lifted shift was it the same economics as you thought. So what we think is the next eighty percent is not going to lead by the application, you know, or things like Office three sixty five we actually think is going to be people putting the real mission critical workloads. And that's a different conversation. That's where we think the market's complaints what we do at I BM and infrastructure, where you need to have the technology, but also the expertise and industry moving on. Then security becomes key concern. >> So way mentioned red hat here and you can't say too much, but we know about kind of the cloud native modern, you know, multi cloud stuff. But Red had also has, you know, quite a bit of a storage portfolio, you know, seven cluster acquisitions, open source. Wondering what you can just, you know, as an IBM or tell us about what you think of that portfolio. >> So you know, we can talk about what's gonna happen afterwards, but also I think we made it very clear we believe redheads and used the inn where I think it's a good analogy. We're going to keep it. They're going to be independent. They serve a world we're not going to change it. And I think that's a very important part of the message. But we can't talk about the assets. And I did my own kickoff to my team. My partners and I used a kid around. That was a storage acquisition, you know? So it was my thirty four billion dollars right? So but I think it has a good play. It's completely complimentary to what we do. They have some great to two technologies, but also we bring things to them. They're interesting, but a conversation when someone strikes, they were my going and for strategy. We believe containers are critical. We believe Lennox is critical. If you look at what people are doing on the cloud, it is theirs. It's already cross. It's mostly Lennox and the enterprises have chosen red hat. Now, if you think of what we could do with that particular environment, tohave the conversation make relevance about what we could do to help you on Prim. But now you can run the same thing on prime. You can put it literally anywhere. Now that's a strength of red hat would bring us on a story side. They have great assets. I'm kind of salivating to help him out with that now what they don't do with some of things that we can add to them. Right? So I don't think we're commenting any road map, I think, but they haven't. What we have is, to be honest, complimentary, if that makes >> sense. Well, and I think you you're familiar with our old troop private cloud nomenclature. It's evolving to true hybrid cloud, and what you just described is true hybrid cloud. Run it wherever you want. You're agnostic to where it runs, but don't want that cloud operating model yet to be the underpinning of the experience. >> You don't want me locked in, so that's where I think if you look a hybrid, you want to make sure. And I think it allows us to that. I think IBM is synergistic to it. I think we can bring a lot of how do you bring the integration capabilities that IBM brings on applications and help these mission critical environments that have need some industry expertise. To do that? Bring them to the cloud itself. And it doesn't matter where in the cloud. >> Yeah, at one of the things we were commenting on the open is, you know, the hybrid multicolored world. It's complicated, and IBM has a, you know, strong history with services to help drive that gives a little bit of your insight is toe what IBM brings tea. Kind of that multi cloud environment. >> It's almost too much. Right? So what we're doing is really working on the overall. How do we simplify? So we're going to meet the client where they are, So everyone's at different. You have to almost find out where they are on the journey and then even a particular client. You'll find different business units on different parts of journey so we can help him anywhere from helping figure out, you know, architect, where they would go. We could have moved to the cloud. We can actually help them manage a cloud, and they're also going to meet them where they are. So we have. If you think about what we could do with a I, we have full Aye aye stacks enterprise capabilities. But some people choose to just use their own open source and we can help them. In fact, are you see, our multi club manager allows you to manage, regardless of what you've young for your build environment. So what we're gonna do is meet clients where they are and help them do the last mile. And then we're servers and support were ableto, You know, if you look at what we do, gts were the largest red hat support organization. You look what we do with GPS. We can help people build up their own platforms and given overall struck shen and how to go drive a ay at scale in their environments. So I think I think it does play to us. And I think the red acquisition just ads. I think one of these three. >> Well, and I think, you know, we were talking about in our open that just even IBM giant application, modernization opportunity, right? I mean, because we tend to think about, you know, a I and leading edge, but there's just so much modernization opportunity and a lot of guys, you know, they don't want to go it alone with open source. The fact that IBM is there, you know, with that big blue blanket I called It's. Okay, we're gonna help you through your modernization initiatives. You know, we think a big deal. You know, we're excited about that chapter. Think about Red >> Hat that does a lot of consulting, but they have been discipline. They help you get rail going once you start Doc and open stack rights to be open shifts. Now, that is a whole different way. Tau. Look at your environment, your applications, and that takes a higher level. So it's like one and one is three. They don't do a lot of that now, right so >> well, and it's it's instant developers to That's the other thing. We said just that, while how many developers ready with a million? We're talking about Sisko before with a half a million, which is great. But you're talking eight million. Andi, you know, despite IBM's efforts around, you know, blue mix. So that was a heavy slog. Now, all of a sudden, you got eight million developers. It is. Look, Red hat. Lennox is running the Internet. You know we know this, so that's exciting. I want My last question is you've made a career early part of your career in and taking startups and getting them to a point where they could be acquired very, very successful career there. Then you joined IBM, spent some time at M. I t. You know, getting even smarter when you sit back and look at the industry, the storage industry in particular somehow that despite the trend toward, you know, cloud and bigger is better. You still see the specialist, you know, popping up all the place. You see guys like pure. You see, guys like Nutanix and you know, we saw that early with the comm Palance and the three powers and the ice Alonso, it's okay. Maybe this is the last way. But somehow, storage innovation and it continues to occur. Do you think we're seeing sort of the end of that sort of storage? Startup crazed can can independent storage companies continue to survive? What do your thoughts as an industry observer? >> So I think, is more difficult. But there's plenty of innovation. So you're seeing it as we just help people get to their journey. You're going to see different technologies that even IBM against your portfolio changed rather dramatically to keep up with the trends. And you need to do. It s Oh, I don't think it's over with, so I never want to quoted that I think there's no innovation left and there's a role for you saw storage, you know, grow last year, right? So it was. There's always been growth areas, but it's been flat also really took off on a lot of that is because we're doing >> on a I, >> which is not your average, is definitely not what pure does as faras for storage, right? But you're going to see, I think I'm going to see innovation. I think you're going to see that continue, but I think it's harder and harder for these independent firms, mostly when they scale. I think it's the innovation piece, and I think you're seeing guys like us and I am seeing others innovate very quickly and you can tell innovation is speeding up with investment cause we have to >> get our clients are demanding it, and the VCs keep pouring money in. I mean, you're seeing that in the data protection space and >> data protection isn't now cool, right? So Waseem all time. I think in general, if you look at data protection becomes now your secret weapon. When you talk about being dad driven in a classical environment, you can get copies your data at the AP Eyes anywhere in environment. So I think it's a really big play, so >> well and it opens up new opportunities beyond just back up right for whether it's Dev ops or maybe disaster recovery ransom, where even analytics? Because the backup Corpus is, you know, has all the data, and it's a lot of possibilities. Thanks so much for coming. I think we're going to see you also on Wednesday, right? And looking forward to that. So thank you. All right. Keep it right to everybody. We'll be back with our next guest. You're watching the Cube from day one. IBM thinking Mosconi right back.

For Las Vegas, it's the cube covering vm world 2018, brought to you by vm ware and its ecosystem partners. Hello everyone. Welcome back to the cubes exclusive coverage live coverage here in Las Vegas for Vm world 2018. I'm John furrier with Dave Volante cohost of the cube. Our next guest ed walls. Cube alumni is also the general manager of storage and software defined infrastructure for IBM. Great to see you. Yeah, likewise. So we just were talking to Michael Dell earlier and again, the open ecosystem of vm world again, continues to be a little bit more better vibes this year. It seems to be an uplift around, hey, on premise as is working now, it seems to have some visibility. It's not gloom and doom for all this data center, but certainly some work going on. You guys were out front and software defined infrastructure and storage from so leader. Yep. What's the update? What's going on? Well, I think the, uh, the upbeat here that we're seeing from vm world is the same. We're seeing across the industry there You don't wear a boom happening and being data driven, is a, almost a renaissance. which plays a storage heavily. You do have to have right infrastructure to pull this off either on premises or in a hybrid or a multicloud at ECC. I like Lvm, we're kind of clear, you know the nomenclature around hybrid is homogeneous across the cloud and multicloud is all different types, which we see both so I think it's helping our business. You're seeing us grow since I've been You're seeing a large explosion and what we're doing on flash mostly because the here a growing pretty consistently above market. performance does matter. We did a major new and non. So you've seen our, our innovation pipelines alive and really kicking in and it really plays what clients are trying to really be more data driven of the drivers of your growth suddenly successes. What are the key drivers? Yeah, so I think it has to come down to offerings if you don't have the right offerings to help people to get. We use the term, you know, during the cloud, but it's really turning more of a journey to being data driven and it has to be in a cloud context and that is all about having the right offerings. We do focus in different areas for innovation, what we're doing, our flash, and it's not just flashed for individual raise, but where that's going as far as how you're going to access the data, you know, it's very different and modernizing the cognitive infrastructure or what you're going to do in these new cloud apps, data center compared to what you can do on ai infrastructure, but they need different things. But it's all about being more data driven, being more agile, but also really taking the complexity down. So, and I think when we're doing the exact same thing, all the announcements this week, we're all about making it easier to be data driven regardless of where you're at, which is what we're seeing and security to continues to pound this at the core, which is you're going to see the same thing from ibm. We're going to talk about technology innovation. We talked about the industry context, but it has to be the trust and security at the core and that's where I think that's why ibm has been doing so well. You know, it's interesting when I've been in this business a long time, you've been in the storage business awhile. Me, me a little longer than you guys, but John and I were talking about this the other day, the stores, everybody's been saying infrastructure is dead. I've been hearing that for 35 years. Tommy Rosen believe infrastructure matters, but it's consistently a growth market and a 60 percent plus gross margin product. Now you've got so many people don't understand that. I hear people that well, how come the VC's are putting so much money into storage? Backup is now data particular booming. Yep. Lot of companies say, well, we're going to divest of this or that or get out. You know, you've seen some companies say, oh well you're going to go here. It's just a consistent performer. Your thoughts. So I agree and and decided just storage, but I would say in most stores plays into that, right? It's all about being data driven and eventually it has to land someplace for performance, but it's the agility piece is that you're seeing us in the other industry. Leaders really drive that you can You mentioned backup, you know, actually leverage this data in different ways. so we'll talk about storage rays need to be flexible and fast, but backup ends up really turning into an interesting segment which you're seeing us lead in is how do you get more effective? All those copies of data instead of being some, you know, insurance policy, you hope you never get to and it might take you a long time to get the data back. Now these instantaneous copies for recovery, but also for how do you leverage that for being data driven. So all of a sudden if you're trying to be doing digital transformation, you have to deal with the infrastructure on prem or in the cloud. You're looking for ways to do that easier. We use it on a journey to cloud. It seems more and more it's a journey to being data driven, but it has to be in cloud context, but it plays in a storage or ugly it. You need the right infrastructure to build on to be honest. Sure. Yeah. The data driven thing and you're pounding that Mr Hardy, which I love by the way, it's not new for you guys, but one of the things we've been talking about from day one, you know our wrap data's at the center of the value proposition, but we were talking early on years ago about data being a real part of the development process now apps, so one of things gelsinger kind of talked about apps are now the new networks or something around networking networks and they also sent security. NSX has putting security of around the apps. Decoupling that from say network security. She's starting to see data in the APP component. Highly coupled with the application, new models around how that's stored and retrieved. Yep. Service Meshes and things going on around the application. How has that changed or have you been vectoring to that place where IBM. So one of the investment themes is how to do that and there's a couple different reasons. How do you become more data driven, so all that data, how do you use it to get better insights, right? Uh, and then all the data is typically on, you know, in your infrastructure, a lot of this on prem or in the cloud. So one is getting more insights, right? So you need to flexibly go after those copies. And then, uh, the other thing is how to use it to do better APP development. So you want to do Dev ops, which is how do you just get better quality, faster delivery for a pipeline. But a lot of times if you can't bring the data, bringing, like even if you do a test of a mobile app, you need to test it with the production data sets. So how do you do that? Flexibility and data is a hardest part of all this stuff. So we're doing on ai, how do you get application driven to go after it and the right performance. But we see Dev ops, you can see across our entire product line an API layer, yes, we have apis across all of our storage for our products and software, but a separate API layer to allow you to do a lot of these things with, not to replace any devops tools put to enable those devops tools or Coobernetti's or whatever orchestration engine to drive the flexibility of your will. I'll say mission critical, either primary storage or secondary storage cause. So programmability is critical through the apis infrastructure's code. If you don't have the right API layer and we play grandma will. That's exactly okay. And it's in place across our entire portfolio today. So you don't have to have an API for this device or that device or that backup? No, it's an API layer that covers them all. Voices you guys perfectly for the growth of containers, Coobernetti's and service measures. So as Dev ops starts to move up the stack, you need to have an under the hood programmable model and that is the software defined. So you guys are saying you have that today. Okay. So let's go. It was a big. We had major investments to get there and also do it in such a way that it's a consistent api layer that is abstracted away from the infrastructure you never want to give in general, you don't want get developers access to lower end technology and your system of record data, but you do need to give them access and have the rights, you know, security rounded access control. But it has to be api driven because sometimes there's not a human. It's, it's Dev ops. Okay, so let's stay on that for a secretary we made because I had. You're known as a business whiz kid. Oh, okay. But, but people don't realize how technical you are, which probably drives a lot of the people that work for you crazy. But when you think about the ascendancy of virtualization, it changed the way in which, for example, you had to do data protection and then one of your old companies, you know, you're popularize a source side district location, which made a lot of sense because you were taking 10 servers down to one. Right? As you go into this world of cloud and multicloud. Yep. You were just touching upon architecturally some of the things you What are the key components sort of architecturally that you've been driving have to do with microservices, etc. in your r and d pipeline, which we've noticed over the last two years that you've accelerated at ibm? So we talked about this data driven multi cloud architecture and the only reason for it, it's not a how do you go across a very broad portfolio that ibm storage has and how do you have a way to say we're going to be able to give you a modern, agile and flexible infrastructure that allows you to participate in modernizing your existing environment or allow you to ai, you know, lilly industry leading type, the ai technologies or these new apps. But hoW do you give clients a way to say this portfolio allows you to do that across your entire portfolio? So one is this api layers. You need to not only have rich apis around the storage self, but a lot of times you don't want to give those to developers or other people. So we need a separate api layer, which we put across both our primary storage and secondary storage. That was one that gives you the agility to do almost anything and it, and it doesn't compete with all these orchestration or dev ops tools. It enables them, it's the last mile if you would a second thing, you need to be software defined that gives you a way to api but also literally be able to move things, a flexibility but also investment protection and then you need some core innovation, right? So we still make it a lot of hardware, so we're making flash technologies that keep the low latency at workload with encryption on the card, at line speed with ddup, etc. All the things on the card. So you'll see us innovating on the technology side, but it's also having the agility and a and a flexibility. So you'll see that as a theme and we can have it in. We see clients adopting it in three areas. It's either modernizing traditional, I would say this is all about modernizing nutritional application more agile mode, private cloud or multi cloud. infrastructure and how to make it faster, The other thing is ai in it. It is huge right now. Getting insights right? So how did you machine learning deep learning, true ai on prem or in the cloud and this different technologies to get that insight. So you saw what we did. The largest ai supercomputer in the world was designed to use 100 percent ibm storage, actually ibm systems with a power, a power ai environments. That's a great point about this community is really an it footprint kind of app and its operation. So ai maybe ai ops but they're not. ai is not a core competency like tensor flow. So they need simplicity tools to do that easier. Right. So and I think that's what vm ware is doing with a lot of their announcements, just making it easier to deploy, but that's where ibm's been doing. Driving that pretty aggressively with our software side, our cloud side, but also what we're doing in infrastructure itself aliens and I was saying I think ai is not core comps in this community. essentially in this ai. So we just had an argument, not argument and discussion on the cube yesterday with jim colby, And he had brought up a good point about ai operations. Ai's going to automate a lot of things, but I was saying that's under the hood is a general purpose. Ai tensor flows, all these cools that developers want. Those are the guys who want programmable infrastructure. The guys who want ai ops is going to be Infrastructure guys. So really both are very important. under the hood, making things work faster. A very important. But there's different personas at the mechanic fix the engine. We do you get your ai, is it, is it sort of homegrown and your where's it come from? division is a little bit from the watson group, So, uh, we like our clusters. So we have obviously watson, which is very high end of what we do on a cognitive and since you're very deep learning, but we'll use all the open source tools. different than normal ai or machine learning, So what you're seeing, what we're doing in our, um, we did the latest product launch our flashlights from 9,100, which is granted hardware, innovation and vme through and through. We talk about ai infused, so it allows you to have better service and support. So what are we using machine learning, you know, we say deep learning of all the coal home data allows you to do a lot of analysis. Yes. It's an ibm's cloud. Yes. We use some of the things that we're using in watson, but it's all the tools you also see in our power ai for systems or what we're doing. Icp, ibm will go all the way to say here's all the open source tools you want to but you're wrestling with all the open source will help you put together a tool use mr. Customer, without limiting it, but allows you to kind of move forward Let's move forward to. You'd be data driven, and said, are wrestling with that technology. will make it easier to deploy. So you see that in our storage but in systems and really kind of an ibm theme if you would. So cloud, native compute foundation, I already asked you a question on the customer impact and you know, we cover the linux foundation heavily, cncf, you know, doing all this coobernetti's so it's been great being there from the beginning. So, but a lot of, I've noticed a pattern, there's a lot of talking about new players seeing their software defined infrastructure that would find storage. you mentioned earlier here that you guys made a significant investment. So the question is how do customers, potential buyers of this journey of going digital data, data driven, how do they determine the people who are saying it and actually doing it? What's required? I mean because that's ultimately the trying to squint through the noise as dave says, what, who's got what? So you've got multiple years doing this. If someone says hey, I got them to solidify storage. I just launched last week. Yeah, they might not have the trajectory. So how do customers test? Who's got the real deal? It's actually a real. You can. I'm looking at the floor across the way. So you have to get past the hype, right? So, so this is where I like being ibm, so why didn't I do the ceo the next gig or why did they come? and I thInk I think storage or infrastructure is a, I use the term Big boy, big girl gaMe. I think it's more than just building the next era is how to bring all this together and make it easier for clients, deliberate in the way they want. So I think the will see a lot of these point products people come up and say, I am the leader of this or a leader at that. And you saw a couple of stores guy say I'm the best ai, look the my benchmark within video, right? Um, but then you'll look at like an ibm, maybe not aS aggressive and the marketing of the infrastructure, but we're building the largest, you know, super cues, ai in the world. And we can take the exact same technology and give you a quarter rack or half rack or full rack of the exact same technology knowing you can scale it. So sometimes you have to say is, you know, it's not just a, the point, hey, um, you kind of get dig under the, I would say the, uh, culture of question you could customer what's the investment you've made? I mean, going ask like how many years you've been doing it, how does the customer and truly know someone's really going to be software defined and positioned for that data driven integration, that holistic package. Is there a, well, why don't we be, you know, so software defined, you know, the easy thing is, and this gets a little technical, but just, okay, show me your proposal and tell me how I can run the exact same infrastructure in a cloud of my choice. And you know, either on prem or not or on software is that software to find. And there's a lot of [inaudible] then Now that is self is not maybe the biggest thing, you know, that not suffered a fine. but who can you partner with, um, that can help you. Not only, okay, I'm going to get your next array. That's kind of a tactical, but who can I partner with to kind of go forward and you can't partner with 20 different firms. Who can you partner with to kind of get you from where you are to kind of where you want to get to. So I guess my questions are, I would start asking him how can you help me in a broader scope? And that's where ibm does well. The cloud is a good one. Can you run the storage on premise and on any cloud seamlessly route? What we did in the last launches flashes now you 100. We also say it's multi cloud enabled and what we did is we put all the software in the bundle. We have these validated designs that show you how to exactly do all the major use cases in any cloud and allow you to actually show you test environments. You do this cookbook at works but also the software go try it. But it's kinda how do you make it real compared to put a bunch of clouds and a configuration. But then it's the use cases. So we do a lot with the leaders and laggards, I mean we're with our leaders, but then we have some people that are really struggling to keep up and some of it is just showing the way forward and it's a, you gotta have a broad enough portfolio Otherwise you end up with a point product here and that point park there to get them over time, the right location. and I think that's where no one has enough team to keep up. But you also look at the developers and our developers abstracted away and does the storage to the performance and versatile for developers without even touching it back to your infrastructure as code comment. Oh well now the next one. So can I run, you know, you know, tell me whatever technology you ansible, if you're gonna run a whatever a workflow you got to use for your pipeline, will it work for the api? Should write without opening up the whole storage api. You can, you can drive the car, not even look at the engine. That's the key storage. Right? And it has to be that way. And then can I get all the key abilities in the same function? So it's a good question. I'll think about it more. well, I didn't mean to put you on the spot there. It's just good. It's good. Question number one question we get from people is how do you tell the pretenders when the players, that's ultimately what customers are. And they do bake offs when you see bake off. We had to, we had an auto nation earlier. Listen less, right? But what's the new bakeoff just run code on it. Right? So the world's changing and thanks for. Come on real quick. What's your thoughts on vm world 2018 this year? What's the vibe that you're feeling? What's the overall sentiment? What's your view? I think it's overall very positive. You mentioned early on it feels like an uplifting. I think you're seeing that across infrastructure and infrastructure does matter. I think uh, you know, there's big data economy, you, you hear a boom and it's going pretty well. So I think it's exciting and I think you feel among the ecosystem, any predictions for next year? What we'll see because we like to roll the predictions from next year, predictions for 2018 next year at vm world. What? What's going to happen? Oh geez. I think more of the same. I think you're gonna see even more and more how to be data driven. Yeah. The clouds have given during the cloud, but how to be more data driven and we're going to make it even easier and easier for our clients to do that and the ecosystems driving that and I think you can see more and more of that. Here he goes. This is robin. Thanks for coming on this cubes. Coverage here live in las vegas. I'm john furrier. dave. A lot. They stayed with us. We're on only day two. We've got three days of wall to wall coverage, two sets, tons of interviews, a lot of great people, a lot of great content. A lot of data we've got that data was sharing with you here. We're data driven on the cube. Stay with us. We'll be back after this short break. Okay.

>> Announcer: Live from downtown San Francisco, it's theCUBE covering IBM Chief Data Officer Strategy Summit 2018, brought to you by IBM. (upbeat music) >> Welcome back to San Francisco, everybody. You're watching theCUBE, the leader in live tech coverage. We're covering the IBM Chief Data Officer Strategy Summit #ibmcdo. Ed Walsh is here. He's the General Manager of IBM Storage, and Steven Eliuk who's the Vice President of Deep Learning in the Global Chief Data Office at IBM, Steven. >> Yes, sir. >> Good to see you again. Welcome to The CUBE. >> Pleasure to be here. So there's a great story. We heard Inderpal Bhandari this morning talk about the enterprise data blueprint and laying out to the practitioners how to get started, how to implement, and we're going to have a little case study as to actually how you're doing this. But Ed, set it up for us. >> Okay, so we're at this Chief Data Officer Summit in the Spring, we do it twice a year and really get just Chief Data Officers together to think through their different challenges and actually share. So that's where we're at the Summit. And what we've, as IBM, as kind of try to be a foot forward, be that cognitive enterprise and showing very transparently what we're doing at our organization be more data-driven. And we've talked a bunch of different times. Everyone needs to be data-driven. Everyone wants to be data-driven, but it's really challenging for organizations. So what we're doing is with this blueprint which we're showing as a showcase, in fact you can actually physically come in and see our environment. But more importantly we're being very transparent on all the different components, high-level processes, what we did in governance, but also down to the Lilly Technology level and sharing that with our... Not because they want to do all of it, but maybe they want to do some of it or half of it, but it would be a blueprint that's worked. And then we're being transparent about what we're getting internally for our own transformation as IBM. Because really if we looked at this as a platform, it's really an enterprise cognitive data platform that all of IBM uses on all our transformation work. So our client, in fact, is Steven, and I think you can give what are we doing. By the way, it also, same type of infrastructure allows you to do what we did in the national labs, the largest supercomputers in the world, same infrastructure and the same thing we're trying to do, is make it easier for people to get insights from the data at scale in the enterprise. So that's why I want to bring Steven on. >> I joked with Inderpal. I said, "Well, if you can do it at IBM, "if you can do it there you can do it anywhere," (Ed laughing) because he's point oh. We're at a highly complex organization. So Steven, take us through how you got started and what you're doing. >> For sure, so I'm what's referred to probably as a difficult customer. So because we're so multifaceted we have so many different use cases internally in the orders of hundreds, it doesn't mean that I can just say, "Hey, this is a specific pattern that I need, Ed. "You need to make sure your hardware is sufficient in this area," because the next day I'm going to be hitting him and say, "Hey Ed, I need you to make sure "that it's also efficient in terms of bandwidth as well." And that's the beauty of working in this domain, is that I have those hundreds of use cases and it means that I'm hitting low latency requirements, bandwidth requirements, extensibility requirements because I have a huge number of headcount that I'm bringing on as well. And if I'm good now I don't have to worry about in six months to be stating, "Hey, I need to roll out new infrastructure "so I can support these new data scientists "and effectively so that they can get outcomes quicker." And I'd need to make sure that all the infrastructure behind the scenes is extensible and supports my users. And what I don't want them to have to worry about specifically is how that infrastructure works. I want them to focus on those use cases, those enterprise use cases, and I want them to touch as many of those use cases as possible. >> So Inderpal laid out sort of his five things that a CDO should do. He starts with develop a clear data strategy. So as the doer in the organization, how'd you go about doing that? Presumably you participated in that data strategy, but you're representing the lines of business presumably to make sure that it's of value to them. You can accelerate business value, but how did you start? I mean that's a big challenge, chewy. >> For sure, yeah, it's a huge challenge. And I think effectively curating, locating, governing, and quality aspects of that data is one of the first aspects. And where does that data reside, though, and how do we access it quickly? How does it support structured and unstructured data effectively? Those are all really important questions that had to come to light. And that's some of the approaches that we took. We look at the various business units and we look at are they curating the data correctly? Is it the data that we need? Maybe we have to augment that curation process before we actually are able to kind of apply new techniques, new machine-learning techniques, to that use case. There's a number of different aspects that kind of get rolled into that, and bringing effective storage and effective compute to the table really accelerates us in that journey. >> So Ed, what are the fundamental aspects of the infrastructure that supports this sort of emerging workload? >> Yeah, no, good question. And some of it is what we're going to talk about, what's a storage layer and what's a compute layer, but also what are the tools we're putting in place to use a lot of these open-source toolsets and make it easier for people to use but also use that underlying infrastructure better. So if you look at the high level, we use a storage infrastructure that is built for these AI workloads which is closer to an HPC workload. So the same infrastructure we use, we use the term ESS or elastic storage server. It's a combination. It's a turnkey solution, half rack, full rack. But it can start very small and grow to the biggest supercomputers in the world like what we're doing in the national labs, like the largest top five supercomputers in the world. But what that is is a file system called Spectrum Scale. Allows you to scale up at the performance but also low latency, gets added to the metadata but also high throughput. So we can do layers on that either on flash being all the hot tiers'll be on flash because it's not just the throughput you need which is high. So our lowest end box's close to like what, 26 gigabytes a second. Our highest one like national labs is 4.9 terabytes a second throughput. But it's also the low latency quick access. So we have a storage infrastructure but then we also have high-performance compute. So what we have is our Power Systems, our POWER9 Systems with GPUs, and the idea is how do you, we use the term feed the beast? How do you have the right throughput or IOPS to get the data close to that CPU or the GPU? The Power Systems have a unique bandwidth, so it's not like what you just find from a Comodo, the Intel servers. It's a much faster throughput, so it allows us to actually get data between the GPU CPU in storage or memory very fast. So you can get these deep learning times, and maybe you can share some of that. The learning times go up dramatically, so you get the insight. And then we're also putting layers on top which are IBM Cloud Private, is basically how do you have a hybrid cloud container-based service that allows you to move things seamlessly across and not have to wrestle with how to put all these things together either so it works seamlessly between a public cloud and private cloud? Then we have these toolsets, and I talked about this last time. It might not seem like storage or what you have in APU but we use the term PowerAI, is taking all these machine-learning tools because everyone always used open source. But we make them one more scale but also to ease your use. So how do you use a bunch of great GPUs and CPUs, great throughput, and how do you scale that? A lot of these tools were basically to be run on one CPU. So to be distributed, key research from IBM allows you to actually with PowerAI take the same TensorFlow workflows or dot dot dot and run it across a grid dramatically changing what you're doing from learning times. But anyway you can probably give more, I think, but it's a multiple layer. It's not one thing but it's not what you use for digital storage infrastructure, compute infrastructure for normal workloads. It is custom so you can't... A lot of people try to deploy maybe their NAS storage box and maybe it's flash and try to deploy it. And you can get going that way but then you hit a wall real quick. This is purposely built for AI. >> So Beth Smith was on earlier. She threw out a stat. She said that 85% of their, based on some research, I'm not sure if it was IBM or Forrest or Gartner, said 85% of customers they talked to said AI will be a competitive advantage but only 20% can use it today at scale. So obviously scale is a big challenge, and I want to ask you to comment on another potential challenge. We always talk about elastic infrastructure. You scale up, scale down, or end of month, okay. We sometimes use this concept of plastic infrastructure. Basically plastic maintains its shape because these workloads are so diverse. I don't want to have to rip down my infrastructure and bring in a new one every time my workload changes. So I wonder if you can talk about the sort of requirements from your perspective both in terms of scale and in terms of adaptability to changing workloads. >> Well, I think one of the things that Ed brought up that's really, really important is these open-source frameworks assume that it's running on a single system. They assume that storage is actually local, and that's really the only way that you get really effective throughput from it, is if it's local. So extending it via PowerAI, via these appliances and so forth means that you can use petabytes of storage at a distance and still have good throughput and not have those GP utilization coming down because these are very expensive devices. So if the storage is the blocker, is their controller and he's limiting that flow of data then ultimately you're not making the most effective use of those very expensive computational mediums. But more importantly it means that your time from ideation to product is slowed down, so you're not able to get those business outcomes. That means your competitor could get those business outcomes if they don't have it. And for me what's really important is I mentioned this briefly earlier, is that I need those specialists to touch as much of the data or as much as those enterprise use cases as possible. At the end of the year it's not about touching three use cases. It's the touching three this year, five, ten, more and more and more. And with the infrastructure being storage and computation, all of that is key attributes to kind of seeing that goal. >> Without having to rip that down and then repurpose building it every time. >> Steven: Yeah. >> And just being able to deal with the grid as a grid and you can place workloads across a grid. >> 100%. >> That's our Spectrum compute products that we've been doing for all the major banks in the world to do that and take these workloads and place them across a grid is also a key piece of this. So we always talk about the infrastructures being hey, Ed, that's not storage or infrastructure. No, you need that. And that's why it's part of my portfolio to actually build out the overall infrastructure for people to build on prim but also talk about everything we did with you on prim is hybrid. It's goes to the Cloud natively because some workloads we believe will be on the Cloud for good reasons, and you need to have that part of it. So everything we're going with you is hybrid cloud today, not in the future, today. >> No, 100%, and that's one of the requirements in our organization that we call A-1 architecture. If we write it for our own prim we have to be able to run it on the Cloud and it has to have the same look and feel and painted glass and things like that as well. So it means we only have to write it once, so we're incredibly efficient because we don't have to write it multiple times for different types of infrastructure. Likewise we have expectations from the data scientists that the performance all still have to be up to par as well. We want to really be moving the computation directly to where the data resides and we know that it's not just on prim, it's not in the Cloud, it's a hybrid scenario. >> So don't hate me for asking you this, Ed, but you've only been here for a couple years. Did you just stumble into this? You got this vast portfolio, you got this tooling, you got cloud. You got a part of your organization saying we got to do on prim. The other part's saying we got to do public. Or was this designed to the workload? Was kind of a little bit of both? >> Well, I think luck is good, but it's a embarrassment of riches inside IBM between our primary research, some of the things we were just talking about. How do you run these frameworks in a distributed fashion and not designed that way and do it performing at scale? That's our primary, that's research. That's not even in my group. What we're doing is for workload management. That's in storage, but we have these toolsets. The key thing is work with the clients to figure out what they're trying to do. Everyone's trying to be data-driven, so as we looked at what you need to do to be truly data-driven, it's not just having faster storage although that's important. It's not about the throughput or having to scale up. It's not about having just the CPUs. It's not just about having the open frameworks, but it's how to put that all together that we're invisible. In fact you said it earlier. He doesn't want his users to know at all what's underneath. He just wants to run their workload. You have people from my organization because I'm one of your customers. You're my customer but we go to you and say, "We're trying to use your platform "for a 360 view of the client," and our not data scientists, not data engineers, but ops team can use his platform. So anyway, so I actually think it's because IBM has its broad portfolio that we can bring together. And when IBM shows up which we're showing up in AI together in the Cloud, that's when you see something that we can truly do that you can't get from other organizations. And it's because of the technology differentiation we have from the different groups, but also the industry contacts that we bring. >> 100%. >> And also when you're dealing with data it is the trust. We can engage the clients at a high level and help them because we're not a single-product company. We might be more complex, but when we show up and bring the solution set we can really differentiate. And I think that's when IBM shows up. It's pretty powerful. >> And I think it's moved from "trust me" as well to "show me," and we're able to show it now because we're eating what we're producing. So we're showing. They called it a blueprint. We're using that effectively inside the organization. >> So now that you've sort of built this out internally you spend a lot of time with clients kind of showing them or...? >> Probably 15% of my time. >> So not that much. >> No, no, because I'm in charge of internal transformation operations. They're expecting outcomes from us. But at the same time there's clients that are in the exact same boat. The realization that this is really interesting. There's a lot of noise, a lot of interesting stuff in AI out there from Google, from Facebook, from Amazon, from all, Microsoft, but image recognition isn't important to me. How do I do it for my own organization? I have legacy data from 50 years. This is totally different, and there's no Git repo that I can go to and download them all and use it. It's totally custom, and how do I handle that? So it's different for these guys. >> What's on your wishlist? What's on Ed's to do list? >> Oh geez, uh... I want it so simple for my data scientists that they don't have to worry about where the data's coming from. Whether it be a traditional relational database or an object store, I want it to feed that data effectively and I don't want to have to have them looking into where the data is to make sure the computation's there. I want it just to flow effortlessly. That's really the wishlist. Likewise, I think if we had new accelerators in general outside the box, not something from the traditional GPU viewpoint, maybe data flow or something in new avant-garde-type stuff, that would be interesting because I think it might open up a new train of thought in the area just like GPUs did for us. >> Great story. >> Yeah I know, I think it's... So we're talking about AI for business, and I think what you're seeing is we're trying to showcase what IBM's doing to be really an AI business. And what we've done in this platform is really a showcase. So we're trying to be as transparent as possible not because it's the only way to do it but it's a good example of how a very complex business is using AI to get dramatically better and everyone's using the same kind of platform. >> Well, we learned, we effectively learned being open is much better than being closed. Look at the AI community. Because of its openness that's where we're at right now. And following the same lead we're doing the same thing, and that's why we're making everything available. You can see it and we're doing it, and we're happy to talk to you about it. >> Awesome, all right, so Steven, you stay here. >> Yeah. >> We're going to bring Sumit on and we're going to drill down into the cognitive platform. >> That's good. This guy, thanks for setting it up. I really, really appreciate it. >> Thank you very much. >> All right, good having you guys. All right, keep it right there, everybody. We'll be back at the IBM CDO Strategy Summit. You're watching theCUBE. (upbeat music) (telephone dialing) (modem connecting)