>> connecting with thought leaders all around the world, this is a CUBE Conversation. Hi, everybody this is Dave Vellante of theCUBE, and welcome back to my CXO series. I've been running this through really since the start of the COVID-19 crisis to really understand how leaders are dealing with this pandemic. Sri Ambati is here, he's the CEO and founder of H20. Sri, it's great to see you again, thanks for coming on. >> Thank you for having us. >> Yeah, so this pandemic has obviously given people fits, no question, but it's also given opportunities for companies to kind of reassess where they are. Automation is a huge watchword, flexibility, business resiliency and people who maybe really hadn't fully leaned into things like the cloud and AI and automation are now realizing, wow, we have no choice, it's about survival. Your thought as to what you're seeing in the marketplace. >> Thanks for having us. I think first of all, kudos to the frontline health workers who have been ruthlessly saving lives across the country and the world, and what you're really doing is a fraction of what we could have done or should be doing to stay away the next big pandemic. But that apart I think, I usually tend to say BC is before COVID. So if the world was thinking about going digital after COVID-19, they have been forced to go digital and as a result, you're seeing tremendous transformation across our customers, and a lot of application to kind of go in and reinvent their business models that allow them to scale as effortlessly as they could using the digital means. >> So, think about, doctors and diagnosis machines, in some cases, are helping doctors make diagnoses, they're sometimes making even better diagnosis, (mumbles) is informing. There's been a lot of talk about the models, you know how... Yeah, I know you've been working with a lot of healthcare organizations, you may probably familiar with that, you know, the Medium post, The Hammer and the Dance, and if people criticize the models, of course, they're just models, right? And you iterate models and machine intelligence can help us improve. So, in this, you know, you talk about BC and post C, how have you seen the data and in machine intelligence informing the models and proving that what we know about this pandemic, I mean, it changed literally daily, what are you seeing? >> Yeah, and I think it started with Wuhan and we saw the best application of AI in trying to trace, literally from Alipay, to WeChat, track down the first folks who were spreading it across China and then eventually the rest of the world. I think contact tracing, for example, has become a really interesting problem. supply chain has been disrupted like never before. We're beginning to see customers trying to reinvent their distribution mechanisms in the second order effects of the COVID, and the the prime center is hospital staffing, how many ventilator, is the first few weeks so that after COVID crisis as it evolved in the US. We are busy predicting working with some of the local healthcare communities to predict how staffing in hospitals will work, how many PPE and ventilators will be needed and so henceforth, but that quickly and when the peak surge will be those with the beginning problems, and many of our customers have begin to do these models and iterate and improve and kind of educate the community to practice social distancing, and that led to a lot of flattening the curve and you're talking flattening the curve, you're really talking about data science and analytics in public speak. That led to kind of the next level, now that we have somewhat brought a semblance of order to the reaction to COVID, I think what we are beginning to figure out is, is there going to be a second surge, what elective procedures that were postponed, will be top of the mind for customers, and so this is the kind of things that hospitals are beginning to plan out for the second half of the year, and as businesses try to open up, certain things were highly correlated to surgeon cases, such as cleaning supplies, for example, the obvious one or pantry buying. So retailers are beginning to see what online stores are doing well, e-commerce, online purchases, electronic goods, and so everyone essentially started working from home, and so homes needed to have the same kind of bandwidth that offices and commercial enterprises needed to have, and so a lot of interesting, as one side you saw airlines go away, this side you saw the likes of Zoom and video take off. So you're kind of seeing a real divide in the digital divide and that's happening and AI is here to play a very good role to figure out how to enhance your profitability as you're looking about planning out the next two years. >> Yeah, you know, and obviously, these things they get, they get partisan, it gets political, I mean, our job as an industry is to report, your job is to help people understand, I mean, let the data inform and then let public policy you know, fight it out. So who are some of the people that you're working with that you know, as a result of COVID-19. What's some of the work that H2O has done, I want to better understand what role are you playing? >> So one of the things we're kind of privileged as a company to come into the crisis, with a strong balance and an ability to actually have the right kind of momentum behind the company in terms of great talent, and so we have 10% of the world's top data scientists in the in the form of Kaggle Grand Masters in the company. And so we put most of them to work, and they started collecting data sets, curating data sets and making them more qualitative, picking up public data sources, for example, there's a tremendous amount of job loss out there, figuring out which are the more difficult kind of sectors in the economy and then we started looking at exodus from the cities, we're looking at mobility data that's publicly available, mobility data through the data exchanges, you're able to find which cities which rural areas, did the New Yorkers as they left the city, which places did they go to, and what's to say, Californians when they left Los Angeles, which are the new places they have settled in? These are the places which are now busy places for the same kind of items that you need to sell if you're a retailer, but if you go one step further, we started engaging with FEMA, we start engaging with the universities, like Imperial College London or Berkeley, and started figuring out how best to improve the models and automate them. The SEER model, the most popular SEER model, we added that into our Driverless AI product as a recipe and made that accessible to our customers in testing, to customers in healthcare who are trying to predict where the surge is likely to come. But it's mostly about information right? So the AI at the end of it is all about intelligence and being prepared. Predictive is all about being prepared and that's kind of what we did with general, lots of blogs, typical blog articles and working with the largest health organizations and starting to kind of inform them on the most stable models. What we found to our not so much surprise, is that the simplest, very interpretable models are actually the most widely usable, because historical data is actually no longer as effective. You need to build a model that you can quickly understand and retry again to the feedback loop of back testing that model against what really happened. >> Yeah, so I want to double down on that. So really, two things I want to understand, if you have visibility on it, sounds like you do. Just in terms of the surge and the comeback, you know, kind of what those models say, based upon, you know, we have some advanced information coming from the global market, for sure, but it seems like every situation is different. What's the data telling you? Just in terms of, okay, we're coming into the spring and the summer months, maybe it'll come down a little bit. Everybody says it... We fully expect it to come back in the fall, go back to college, don't go back to college. What is the data telling you at this point in time with an understanding that, you know, we're still iterating every day? >> Well, I think I mean, we're not epidemiologists, but at the same time, the science of it is a highly local response, very hyper local response to COVID-19 is what we've seen. Santa Clara, which is just a county, I mean, is different from San Francisco, right, sort of. So you beginning to see, like we saw in Brooklyn, it's very different, and Bronx, very different from Manhattan. So you're seeing a very, very local response to this disease, and I'm talking about US. You see the likes of Brazil, which we're worried about, has picked up quite a bit of cases now. I think the silver lining I would say is that China is up and running to a large degree, a large number of our user base there are back active, you can see the traffic patterns there. So two months after their last research cases, the business and economic activity is back and thriving. And so, you can kind of estimate from that, that this can be done where you can actually contain the rise of active cases and it will take masking of the entire community, masking and the healthy dose of increase in testing. One of our offices is in Prague, and Czech Republic has done an incredible job in trying to contain this and they've done essentially, masked everybody and as a result they're back thinking about opening offices, schools later this month. So I think that's a very, very local response, hyper local response, no one country and no one community is symmetrical with other ones and I think we have a unique situation where in United States you have a very, very highly connected world, highly connected economy and I think we have quite a problem on our hands on how to safeguard our economy while also safeguarding life. >> Yeah, so you can't just, you can't just take Norway and apply it or South Korea and apply it, every situation is different. And then I want to ask you about, you know, the economy in terms of, you know, how much can AI actually, you know, how can it work in this situation where you have, you know, for example, okay, so the Fed, yes, it started doing asset buys back in 2008 but still, very hard to predict, I mean, at this time of this interview you know, Stock Market up 900 points, very difficult to predict that but some event happens in the morning, somebody, you know, Powell says something positive and it goes crazy but just sort of even modeling out the V recovery, the W recovery, deep recession, the comeback. You have to have enough data, do you not? In order for AI to be reasonably accurate? How does it work? And how does at what pace can you iterate and improve on the models? >> So I think that's exactly where I would say, continuous modeling, instead of continuously learning continuous, that's where the vision of the world is headed towards, where data is coming, you build a model, and then you iterate, try it out and come back. That kind of rapid, continuous learning would probably be needed for all our models as opposed to the typical, I'm pushing a model to production once a year, or once every quarter. I think what we're beginning to see is the kind of where companies are beginning to kind of plan out. A lot of people lost their jobs in the last couple of months, right, sort of. And so up scaling and trying to kind of bring back these jobs back both into kind of, both from the manufacturing side, but also lost a lot of jobs in the transportation and the kind of the airlines slash hotel industries, right, sort of. So it's trying to now bring back the sense of confidence and will take a lot more kind of testing, a lot more masking, a lot more social empathy, I think well, some of the things that we are missing while we are socially distant, we know that we are so connected as a species, we need to kind of start having that empathy for we need to wear a mask, not for ourselves, but for our neighbors and people we may run into. And I think that kind of, the same kind of thinking has to kind of parade, before we can open up the economy in a big way. The data, I mean, we can do a lot of transfer learning, right, sort of there are new methods, like try to model it, similar to the 1918, where we had a second bump, or a lot of little bumps, and that's kind of where your W shaped pieces, but governments are trying very well in seeing stimulus dollars being pumped through banks. So some of the US case we're looking for banks is, which small medium business in especially, in unsecured lending, which business to lend to, (mumbles) there's so many applications that have come to banks across the world, it's not just in the US, and banks are caught up with the problem of which and what's growing the concern for this business to kind of, are they really accurate about the number of employees they are saying they have? Do then the next level problem or on forbearance and mortgage, that side of the things are coming up at some of these banks as well. So they're looking at which, what's one of the problems that one of our customers Wells Fargo, they have a question which branch to open, right, sort of that itself, it needs a different kind of modeling. So everything has become a very highly good segmented models, and so AI is absolutely not just a good to have, it has become a must have for most of our customers in how to go about their business. (mumbles) >> I want to talk a little bit about your business, you have been on a mission to democratize AI since the beginning, open source. Explain your business model, how you guys make money and then I want to help people understand basic theoretical comparisons and current affairs. >> Yeah, that's great. I think the last time we spoke, probably about at the Spark Summit. I think Dave and we were talking about Sparkling Water and H2O our open source platforms, which are premium platforms for democratizing machine learning and math at scale, and that's been a tremendous brand for us. Over the last couple of years, we have essentially built a platform called Driverless AI, which is a license software and that automates machine learning models, we took the best practices of all these data scientists, and combined them to essentially build recipes that allow people to build the best forecasting models, best fraud prevention models or the best recommendation engines, and so we started augmenting traditional data scientists with this automatic machine learning called AutoML, that essentially allows them to build models without necessarily having the same level of talent as these great Kaggle Grand Masters. And so that has democratized, allowed ordinary companies to start producing models of high caliber and high quality that would otherwise have been the pedigree of Google, Microsoft or Amazon or some of these top tier AI houses like Netflix and others. So what we've done is democratize not just the algorithms at the open source level. Now, we've made it easy for kind of rapid adoption of AI across every branch inside a company, a large organization, also across smaller organizations which don't have the access to the same kind of talent. Now, third level, you know, what we've brought to market, is ability to augment data sets, especially public and private data sets that you can, the alternative data sets that can increase the signal. And that's where we've started working on a new platform called Q, again, more license software, and I mean, to give you an idea there from business models endpoint, now majority of our software sales is coming from closed source software. And sort of so, we've made that transition, we still make our open source widely accessible, we continue to improve it, a large chunk of the teams are improving and participating in building the communities but I think from a business model standpoint as of last year, 51% of our revenues are now coming from closed source software and that change is continuing to grow. >> And this is the point I wanted to get to, so you know, the open source model was you know, Red Hat the one company that, you know, succeeded wildly and it was, put it out there open source, come up with a service, maintain the software, you got to buy the subscription okay, fine. And everybody thought that you know, you were going to do that, they thought that Databricks was going to do and that changed. But I want to take two examples, Hortonworks which kind of took the Red Hat model and Cloudera which does IP. And neither really lived up to the expectation, but now there seems to be sort of a new breed I mentioned, you guys, Databricks, there are others, that seem to be working. You with your license software model, Databricks with a managed service and so there's, it's becoming clear that there's got to be some level of IP that can be licensed in order to really thrive in the open source community to be able to fund the committers that you have to put forth to open source. I wonder if you could give me your thoughts on that narrative. >> So on Driverless AI, which is the closest platform I mentioned, we opened up the layers in open source as recipes. So for example, different companies build their zip codes differently, right, the domain specific recipes, we put about 150 of them in open source again, on top of our Driverless AI platform, and the idea there is that, open source is about freedom, right? It is not necessarily about, it's not a philosophy, it's not a business model, it allows freedom for rapid adoption of a platform and complete democratization and commodification of a space. And that allows a small company like ours to compete at the level of an SaaS or a Google or a Microsoft because you have the same level of voice as a very large company and you're focused on using code as a community building exercise as opposed to a business model, right? So that's kind of the heart of open source, is allowing that freedom for our end users and the customers to kind of innovate at the same level of that a Silicon Valley company or one of these large tech giants are building software. So it's really about making, it's a maker culture, as opposed to a consumer culture around software. Now, if you look at how the the Red Hat model, and the others who have tried to replicate that, the difficult part there was, if the product is very good, customers are self sufficient and if it becomes a standard, then customers know how to use it. If the product is crippled or difficult to use, then you put a lot of services and that's where you saw the classic Hadoop companies, get pulled into a lot of services, which is a reasonably difficult business to scale. So I think what we chose was, instead, a great product that builds a fantastic brand, that makes AI, even when other first or second.ai domain, and for us to see thousands of companies which are not AI and AI first, and even more companies adopting AI and talking about AI as a major way that was possible because of open source. If you had chosen close source and many of your peers did, they all vanished. So that's kind of how the open source is really about building the ecosystem and having the patience to build a company that takes 10, 20 years to build. And what we are expecting unfortunately, is a first and fast rise up to become unicorns. In that race, you're essentially sacrifice, building a long ecosystem play, and that's kind of what we chose to do, and that took a little longer. Now, if you think about the, how do you truly monetize open source, it takes a little longer and is much more difficult sales machine to scale, right, sort of. Our open source business actually is reasonably positive EBITDA business because it makes more money than we spend on it. But trying to teach sales teams, how to sell open source, that's a much, that's a rate limiting step. And that's why we chose and also explaining to the investors, how open source is being invested in as you go closer to the IPO markets, that's where we chose, let's go into license software model and scale that as a regular business. >> So I've said a few times, it's kind of like ironic that, this pandemic is as we're entering a new decade, you know, we've kind of we're exiting the era, I mean, the many, many decades of Moore's law being the source of innovation and now it's a combination of data, applying machine intelligence and being able to scale and with cloud. Well, my question is, what did we expect out of AI this decade if those are sort of the three, the cocktail of innovation, if you will, what should we expect? Is it really just about, I suggest, is it really about automating, you know, businesses, giving them more agility, flexibility, you know, etc. Or should we should we expect more from AI this decade? >> Well, I mean, if you think about the decade of 2010 2011, that was defined by software is eating the world, right? And now you can say software is the world, right? I mean, pretty much almost all conditions are digital. And AI is eating software, right? (mumbling) A lot of cloud transitions are happening and are now happening much faster rate but cloud and AI are kind of the leading, AI is essentially one of the biggest driver for cloud adoption for many of our customers. So in the enterprise world, you're seeing rebuilding of a lot of data, fast data driven applications that use AI, instead of rule based software, you're beginning to see patterned, mission AI based software, and you're seeing that in spades. And, of course, that is just the tip of the iceberg, AI has been with us for 100 years, and it's going to be ahead of us another hundred years, right, sort of. So as you see the discovery rate at which, it is really a fundamentally a math, math movement and in that math movement at the beginning of every century, it leads to 100 years of phenomenal discovery. So AI is essentially making discoveries faster, AI is producing, entertainment, AI is producing music, AI is producing choreographing, you're seeing AI in every walk of life, AI summarization of Zoom meetings, right, you beginning to see a lot of the AI enabled ETF peaking of stocks, right, sort of. You're beginning to see, we repriced 20,000 bonds every 15 seconds using H2O AI, corporate bonds. And so you and one of our customers is on the fastest growing stock, mostly AI is powering a lot of these insights in a fast changing world which is globally connected. No one of us is able to combine all the multiple dimensions that are changing and AI has that incredible opportunity to be a partner for every... (mumbling) For a hospital looking at how the second half will look like for physicians looking at what is the sentiment of... What is the surge to expect? To kind of what is the market demand looking at the sentiment of the customers. AI is the ultimate money ball in business and then I think it's just showing its depth at this point. >> Yeah, I mean, I think you're right on, I mean, basically AI is going to convert every software, every application, or those tools aren't going to have much use, Sri we got to go but thanks so much for coming to theCUBE and the great work you guys are doing. Really appreciate your insights. stay safe, and best of luck to you guys. >> Likewise, thank you so much. >> Welcome, and thank you for watching everybody, this is Dave Vellante for the CXO series on theCUBE. We'll see you next time. All right, we're clear. All right.

>> Starting the record, Dave in five, four, three. Hi, everybody this is Dave Vellante, theCUBE, and welcome back to my CXO series. I've been running this through really since the start of the COVID-19 crisis to really understand how leaders are dealing with this pandemic. Sri Ambati is here, he's the CEO and founder of H20. Sri, it's great to see you again, thanks for coming on. >> Thank you for having us. >> Yeah, so this pandemic has obviously given people fits, no question, but it's also given opportunities for companies to kind of reassess where they are. Automation is a huge watchword, flexibility, business resiliency and people who maybe really hadn't fully leaned into things like the cloud and AI and automation are now realizing, wow, we have no choice, it's about survival. Your thought as to what you're seeing in the marketplace. >> Thanks for having us. I think first of all, kudos to the frontline health workers who have been ruthlessly saving lives across the country and the world, and what you're really doing is a fraction of what we could have done or should be doing to stay away the next big pandemic. But that apart I think, I usually tend to say BC is before COVID. So if the world was thinking about going digital after COVID-19, they have been forced to go digital and as a result, you're seeing tremendous transformation across our customers, and a lot of application to kind of go in and reinvent their business models that allow them to scale as effortlessly as they could using the digital means. >> So, think about, doctors and diagnosis machines, in some cases, are helping doctors make diagnoses, they're sometimes making even better diagnosis, (mumbles) is informing. There's been a lot of talk about the models, you know how... Yeah, I know you've been working with a lot of healthcare organizations, you may probably familiar with that, you know, the Medium post, The Hammer and the Dance, and if people criticize the models, of course, they're just models, right? And you iterate models and machine intelligence can help us improve. So, in this, you know, you talk about BC and post C, how have you seen the data and in machine intelligence informing the models and proving that what we know about this pandemic, I mean, it changed literally daily, what are you seeing? >> Yeah, and I think it started with Wuhan and we saw the best application of AI in trying to trace, literally from Alipay, to WeChat, track down the first folks who were spreading it across China and then eventually the rest of the world. I think contact tracing, for example, has become a really interesting problem. supply chain has been disrupted like never before. We're beginning to see customers trying to reinvent their distribution mechanisms in the second order effects of the COVID, and the the prime center is hospital staffing, how many ventilator, is the first few weeks so that after COVID crisis as it evolved in the US. We are busy predicting working with some of the local healthcare communities to predict how staffing in hospitals will work, how many PPE and ventilators will be needed and so henceforth, but that quickly and when the peak surge will be those with the beginning problems, and many of our customers have begin to do these models and iterate and improve and kind of educate the community to practice social distancing, and that led to a lot of flattening the curve and you're talking flattening the curve, you're really talking about data science and analytics in public speak. That led to kind of the next level, now that we have somewhat brought a semblance of order to the reaction to COVID, I think what we are beginning to figure out is, is there going to be a second surge, what elective procedures that were postponed, will be top of the mind for customers, and so this is the kind of things that hospitals are beginning to plan out for the second half of the year, and as businesses try to open up, certain things were highly correlated to surgeon cases, such as cleaning supplies, for example, the obvious one or pantry buying. So retailers are beginning to see what online stores are doing well, e-commerce, online purchases, electronic goods, and so everyone essentially started working from home, and so homes needed to have the same kind of bandwidth that offices and commercial enterprises needed to have, and so a lot of interesting, as one side you saw airlines go away, this side you saw the likes of Zoom and video take off. So you're kind of seeing a real divide in the digital divide and that's happening and AI is here to play a very good role to figure out how to enhance your profitability as you're looking about planning out the next two years. >> Yeah, you know, and obviously, these things they get, they get partisan, it gets political, I mean, our job as an industry is to report, your job is to help people understand, I mean, let the data inform and then let public policy you know, fight it out. So who are some of the people that you're working with that you know, as a result of COVID-19. What's some of the work that H2O has done, I want to better understand what role are you playing? >> So one of the things we're kind of privileged as a company to come into the crisis, with a strong balance and an ability to actually have the right kind of momentum behind the company in terms of great talent, and so we have 10% of the world's top data scientists in the in the form of Kaggle Grand Masters in the company. And so we put most of them to work, and they started collecting data sets, curating data sets and making them more qualitative, picking up public data sources, for example, there's a tremendous amount of job loss out there, figuring out which are the more difficult kind of sectors in the economy and then we started looking at exodus from the cities, we're looking at mobility data that's publicly available, mobility data through the data exchanges, you're able to find which cities which rural areas, did the New Yorkers as they left the city, which places did they go to, and what's to say, Californians when they left Los Angeles, which are the new places they have settled in? These are the places which are now busy places for the same kind of items that you need to sell if you're a retailer, but if you go one step further, we started engaging with FEMA, we start engaging with the universities, like Imperial College London or Berkeley, and started figuring out how best to improve the models and automate them. The SaaS model, the most popular SaaS model, we added that into our Driverless AI product as a recipe and made that accessible to our customers in testing, to customers in healthcare who are trying to predict where the surge is likely to come. But it's mostly about information right? So the AI at the end of it is all about intelligence and being prepared. Predictive is all about being prepared and that's kind of what we did with general, lots of blogs, typical blog articles and working with the largest health organizations and starting to kind of inform them on the most stable models. What we found to our not so much surprise, is that the simplest, very interpretable models are actually the most widely usable, because historical data is actually no longer as effective. You need to build a model that you can quickly understand and retry again to the feedback loop of back testing that model against what really happened. >> Yeah, so I want to double down on that. So really, two things I want to understand, if you have visibility on it, sounds like you do. Just in terms of the surge and the comeback, you know, kind of what those models say, based upon, you know, we have some advanced information coming from the global market, for sure, but it seems like every situation is different. What's the data telling you? Just in terms of, okay, we're coming into the spring and the summer months, maybe it'll come down a little bit. Everybody says it... We fully expect it to come back in the fall, go back to college, don't go back to college. What is the data telling you at this point in time with an understanding that, you know, we're still iterating every day? >> Well, I think I mean, we're not epidemiologists, but at the same time, the science of it is a highly local response, very hyper local response to COVID-19 is what we've seen. Santa Clara, which is just a county, I mean, is different from San Francisco, right, sort of. So you beginning to see, like we saw in Brooklyn, it's very different, and Bronx, very different from Manhattan. So you're seeing a very, very local response to this disease, and I'm talking about US. You see the likes of Brazil, which we're worried about, has picked up quite a bit of cases now. I think the silver lining I would say is that China is up and running to a large degree, a large number of our user base there are back active, you can see the traffic patterns there. So two months after their last research cases, the business and economic activity is back and thriving. And so, you can kind of estimate from that, that this can be done where you can actually contain the rise of active cases and it will take masking of the entire community, masking and the healthy dose of increase in testing. One of our offices is in Prague, and Czech Republic has done an incredible job in trying to contain this and they've done essentially, masked everybody and as a result they're back thinking about opening offices, schools later this month. So I think that's a very, very local response, hyper local response, no one country and no one community is symmetrical with other ones and I think we have a unique situation where in United States you have a very, very highly connected world, highly connected economy and I think we have quite a problem on our hands on how to safeguard our economy while also safeguarding life. >> Yeah, so you can't just, you can't just take Norway and apply it or South Korea and apply it, every situation is different. And then I want to ask you about, you know, the economy in terms of, you know, how much can AI actually, you know, how can it work in this situation where you have, you know, for example, okay, so the Fed, yes, it started doing asset buys back in 2008 but still, very hard to predict, I mean, at this time of this interview you know, Stock Market up 900 points, very difficult to predict that but some event happens in the morning, somebody, you know, Powell says something positive and it goes crazy but just sort of even modeling out the V recovery, the W recovery, deep recession, the comeback. You have to have enough data, do you not? In order for AI to be reasonably accurate? How does it work? And how does at what pace can you iterate and improve on the models? >> So I think that's exactly where I would say, continuous modeling, instead of continuously learning continuous, that's where the vision of the world is headed towards, where data is coming, you build a model, and then you iterate, try it out and come back. That kind of rapid, continuous learning would probably be needed for all our models as opposed to the typical, I'm pushing a model to production once a year, or once every quarter. I think what we're beginning to see is the kind of where companies are beginning to kind of plan out. A lot of people lost their jobs in the last couple of months, right, sort of. And so up scaling and trying to kind of bring back these jobs back both into kind of, both from the manufacturing side, but also lost a lot of jobs in the transportation and the kind of the airlines slash hotel industries, right, sort of. So it's trying to now bring back the sense of confidence and will take a lot more kind of testing, a lot more masking, a lot more social empathy, I think well, some of the things that we are missing while we are socially distant, we know that we are so connected as a species, we need to kind of start having that empathy for we need to wear a mask, not for ourselves, but for our neighbors and people we may run into. And I think that kind of, the same kind of thinking has to kind of parade, before we can open up the economy in a big way. The data, I mean, we can do a lot of transfer learning, right, sort of there are new methods, like try to model it, similar to the 1918, where we had a second bump, or a lot of little bumps, and that's kind of where your W shaped pieces, but governments are trying very well in seeing stimulus dollars being pumped through banks. So some of the US case we're looking for banks is, which small medium business in especially, in unsecured lending, which business to lend to, (mumbles) there's so many applications that have come to banks across the world, it's not just in the US, and banks are caught up with the problem of which and what's growing the concern for this business to kind of, are they really accurate about the number of employees they are saying they have? Do then the next level problem or on forbearance and mortgage, that side of the things are coming up at some of these banks as well. So they're looking at which, what's one of the problems that one of our customers Wells Fargo, they have a question which branch to open, right, sort of that itself, it needs a different kind of modeling. So everything has become a very highly good segmented models, and so AI is absolutely not just a good to have, it has become a must have for most of our customers in how to go about their business. (mumbles) >> I want to talk a little bit about your business, you have been on a mission to democratize AI since the beginning, open source. Explain your business model, how you guys make money and then I want to help people understand basic theoretical comparisons and current affairs. >> Yeah, that's great. I think the last time we spoke, probably about at the Spark Summit. I think Dave and we were talking about Sparkling Water and H2O or open source platforms, which are premium platforms for democratizing machine learning and math at scale, and that's been a tremendous brand for us. Over the last couple of years, we have essentially built a platform called Driverless AI, which is a license software and that automates machine learning models, we took the best practices of all these data scientists, and combined them to essentially build recipes that allow people to build the best forecasting models, best fraud prevention models or the best recommendation engines, and so we started augmenting traditional data scientists with this automatic machine learning called AutoML, that essentially allows them to build models without necessarily having the same level of talent as these Greek Kaggle Grand Masters. And so that has democratized, allowed ordinary companies to start producing models of high caliber and high quality that would otherwise have been the pedigree of Google, Microsoft or Amazon or some of these top tier AI houses like Netflix and others. So what we've done is democratize not just the algorithms at the open source level. Now, we've made it easy for kind of rapid adoption of AI across every branch inside a company, a large organization, also across smaller organizations which don't have the access to the same kind of talent. Now, third level, you know, what we've brought to market, is ability to augment data sets, especially public and private data sets that you can, the alternative data sets that can increase the signal. And that's where we've started working on a new platform called Q, again, more license software, and I mean, to give you an idea there from business models endpoint, now majority of our software sales is coming from closed source software. And sort of so, we've made that transition, we still make our open source widely accessible, we continue to improve it, a large chunk of the teams are improving and participating in building the communities but I think from a business model standpoint as of last year, 51% of our revenues are now coming from closed source software and that change is continuing to grow. >> And this is the point I wanted to get to, so you know, the open source model was you know, Red Hat the one company that, you know, succeeded wildly and it was, put it out there open source, come up with a service, maintain the software, you got to buy the subscription okay, fine. And everybody thought that you know, you were going to do that, they thought that Databricks was going to do and that changed. But I want to take two examples, Hortonworks which kind of took the Red Hat model and Cloudera which does IP. And neither really lived up to the expectation, but now there seems to be sort of a new breed I mentioned, you guys, Databricks, there are others, that seem to be working. You with your license software model, Databricks with a managed service and so there's, it's becoming clear that there's got to be some level of IP that can be licensed in order to really thrive in the open source community to be able to fund the committers that you have to put forth to open source. I wonder if you could give me your thoughts on that narrative. >> So on Driverless AI, which is the closest platform I mentioned, we opened up the layers in open source as recipes. So for example, different companies build their zip codes differently, right, the domain specific recipes, we put about 150 of them in open source again, on top of our Driverless AI platform, and the idea there is that, open source is about freedom, right? It is not necessarily about, it's not a philosophy, it's not a business model, it allows freedom for rapid adoption of a platform and complete democratization and commodification of a space. And that allows a small company like ours to compete at the level of an SaaS or a Google or a Microsoft because you have the same level of voice as a very large company and you're focused on using code as a community building exercise as opposed to a business model, right? So that's kind of the heart of open source, is allowing that freedom for our end users and the customers to kind of innovate at the same level of that a Silicon Valley company or one of these large tech giants are building software. So it's really about making, it's a maker culture, as opposed to a consumer culture around software. Now, if you look at how the the Red Hat model, and the others who have tried to replicate that, the difficult part there was, if the product is very good, customers are self sufficient and if it becomes a standard, then customers know how to use it. If the product is crippled or difficult to use, then you put a lot of services and that's where you saw the classic Hadoop companies, get pulled into a lot of services, which is a reasonably difficult business to scale. So I think what we chose was, instead, a great product that builds a fantastic brand, that makes AI, even when other first or second.ai domain, and for us to see thousands of companies which are not AI and AI first, and even more companies adopting AI and talking about AI as a major way that was possible because of open source. If you had chosen close source and many of your peers did, they all vanished. So that's kind of how the open source is really about building the ecosystem and having the patience to build a company that takes 10, 20 years to build. And what we are expecting unfortunately, is a first and fast rise up to become unicorns. In that race, you're essentially sacrifice, building a long ecosystem play, and that's kind of what we chose to do, and that took a little longer. Now, if you think about the, how do you truly monetize open source, it takes a little longer and is much more difficult sales machine to scale, right, sort of. Our open source business actually is reasonably positive EBITDA business because it makes more money than we spend on it. But trying to teach sales teams, how to sell open source, that's a much, that's a rate limiting step. And that's why we chose and also explaining to the investors, how open source is being invested in as you go closer to the IPO markets, that's where we chose, let's go into license software model and scale that as a regular business. >> So I've said a few times, it's kind of like ironic that, this pandemic is as we're entering a new decade, you know, we've kind of we're exiting the era, I mean, the many, many decades of Moore's law being the source of innovation and now it's a combination of data, applying machine intelligence and being able to scale and with cloud. Well, my question is, what did we expect out of AI this decade if those are sort of the three, the cocktail of innovation, if you will, what should we expect? Is it really just about, I suggest, is it really about automating, you know, businesses, giving them more agility, flexibility, you know, etc. Or should we should we expect more from AI this decade? >> Well, I mean, if you think about the decade of 2010 2011, that was defined by software is eating the world, right? And now you can say software is the world, right? I mean, pretty much almost all conditions are digital. And AI is eating software, right? (mumbling) A lot of cloud transitions are happening and are now happening much faster rate but cloud and AI are kind of the leading, AI is essentially one of the biggest driver for cloud adoption for many of our customers. So in the enterprise world, you're seeing rebuilding of a lot of data, fast data driven applications that use AI, instead of rule based software, you're beginning to see patterned, mission AI based software, and you're seeing that in spades. And, of course, that is just the tip of the iceberg, AI has been with us for 100 years, and it's going to be ahead of us another hundred years, right, sort of. So as you see the discovery rate at which, it is really a fundamentally a math, math movement and in that math movement at the beginning of every century, it leads to 100 years of phenomenal discovery. So AI is essentially making discoveries faster, AI is producing, entertainment, AI is producing music, AI is producing choreographing, you're seeing AI in every walk of life, AI summarization of Zoom meetings, right, you beginning to see a lot of the AI enabled ETF peaking of stocks, right, sort of. You're beginning to see, we repriced 20,000 bonds every 15 seconds using H2O AI, corporate bonds. And so you and one of our customers is on the fastest growing stock, mostly AI is powering a lot of these insights in a fast changing world which is globally connected. No one of us is able to combine all the multiple dimensions that are changing and AI has that incredible opportunity to be a partner for every... (mumbling) For a hospital looking at how the second half will look like for physicians looking at what is the sentiment of... What is the surge to expect? To kind of what is the market demand looking at the sentiment of the customers. AI is the ultimate money ball in business and then I think it's just showing its depth at this point. >> Yeah, I mean, I think you're right on, I mean, basically AI is going to convert every software, every application, or those tools aren't going to have much use, Sri we got to go but thanks so much for coming to theCUBE and the great work you guys are doing. Really appreciate your insights. stay safe, and best of luck to you guys. >> Likewise, thank you so much. >> Welcome, and thank you for watching everybody, this is Dave Vellante for the CXO series on theCUBE. We'll see you next time. All right, we're clear. All right.

(techno music) >> Announcer: Live, from downtown San Francisco, it's theCUBE. Covering IBM Chief Data Officer Strategy Summit 2018. Brought to you by IBM. >> Welcome back to San Francisco everybody, this is theCUBE, the leader in live tech coverage. We're here at the Parc 55 in San Francisco covering the IBM CDO Strategy Summit. I'm here with Joel Horwitz who's the Vice President of Digital Partnerships & Offerings at IBM. Good to see you again Joel. >> Thanks, great to be here, thanks for having me. >> So I was just, you're very welcome- It was just, let's see, was it last month, at Think? >> Yeah, it's hard to keep track, right. >> And we were talking about your new role- >> It's been a busy year. >> the importance of partnerships. One of the things I want to, well let's talk about your role, but I really want to get into, it's innovation. And we talked about this at Think, because it's so critical, in my opinion anyway, that you can attract partnerships, innovation partnerships, startups, established companies, et cetera. >> Joel: Yeah. >> To really help drive that innovation, it takes a team of people, IBM can't do it on its own. >> Yeah, I mean look, IBM is the leader in innovation, as we all know. We're the market leader for patents, that we put out each year, and how you get that technology in the hands of the real innovators, the developers, the longtail ISVs, our partners out there, that's the challenging part at times, and so what we've been up to is really looking at how we make it easier for partners to partner with IBM. How we make it easier for developers to work with IBM. So we have a number of areas that we've been adding, so for example, we've added a whole IBM Code portal, so if you go to developer.ibm.com/code you can actually see hundreds of code patterns that we've created to help really any client, any partner, get started using IBM's technology, and to innovate. >> Yeah, and that's critical, I mean you're right, because to me innovation is a combination of invention, which is what you guys do really, and then it's adoption, which is what your customers are all about. You come from the data science world. We're here at the Chief Data Officer Summit, what's the intersection between data science and CDOs? What are you seeing there? >> Yeah, so when I was here last, it was about two years ago in 2015, actually, maybe three years ago, man, time flies when you're having fun. >> Dave: Yeah, the Spark Summit- >> Yeah Spark Technology Center and the Spark Summit, and we were here, I was here at the Chief Data Officer Summit. And it was great, and at that time, I think a lot of the conversation was really not that different than what I'm seeing today. Which is, how do you manage all of your data assets? I think a big part of doing good data science, which is my kind of background, is really having a good understanding of what your data governance is, what your data catalog is, so, you know we introduced the Watson Studio at Think, and actually, what's nice about that, is it brings a lot of this together. So if you look in the market, in the data market, today, you know we used to segment it by a few things, like data gravity, data movement, data science, and data governance. And those are kind of the four themes that I continue to see. And so outside of IBM, I would contend that those are relatively separate kind of tools that are disconnected, in fact Dinesh Nirmal, who's our engineer on the analytic side, Head of Development there, he wrote a great blog just recently, about how you can have some great machine learning, you have some great data, but if you can't operationalize that, then really you can't put it to use. And so it's funny to me because we've been focused on this challenge, and IBM is making the right steps, in my, I'm obviously biased, but we're making some great strides toward unifying the, this tool chain. Which is data management, to data science, to operationalizing, you know, machine learning. So that's what we're starting to see with Watson Studio. >> Well, I always push Dinesh on this and like okay, you've got a collection of tools, but are you bringing those together? And he flat-out says no, we developed this, a lot of this from scratch. Yes, we bring in the best of the knowledge that we have there, but we're not trying to just cobble together a bunch of disparate tools with a UI layer. >> Right, right. >> It's really a fundamental foundation that you're trying to build. >> Well, what's really interesting about that, that piece, is that yeah, I think a lot of folks have cobbled together a UI layer, so we formed a partnership, coming back to the partnership view, with a company called Lightbend, who's based here in San Francisco, as well as in Europe, and the reason why we did that, wasn't just because of the fact that Reactive development, if you're not familiar with Reactive, it's essentially Scala, Akka, Play, this whole framework, that basically allows developers to write once, and it kind of scales up with demand. In fact, Verizon actually used our platform with Lightbend to launch the iPhone 10. And they show dramatic improvements. Now what's exciting about Lightbend, is the fact that application developers are developing with Reactive, but if you turn around, you'll also now be able to operationalize models with Reactive as well. Because it's basically a single platform to move between these two worlds. So what we've continued to see is data science kind of separate from the application world. Really kind of, AI and cloud as different universes. The reality is that for any enterprise, or any company, to really innovate, you have to find a way to bring those two worlds together, to get the most use out of it. >> Fourier always says "Data is the new development kit". He said this I think five or six years ago, and it's barely becoming true. You guys have tried to make an attempt, and have done a pretty good job, of trying to bring those worlds together in a single platform, what do you call it? The Watson Data Platform? >> Yeah, Watson Data Platform, now Watson Studio, and I think the other, so one side of it is, us trying to, not really trying, but us actually bringing together these disparate systems. I mean we are kind of a systems company, we're IT. But not only that, but bringing our trained algorithms, and our trained models to the developers. So for example, we also did a partnership with Unity, at the end of last year, that's now just reaching some pretty good growth, in terms of bringing the Watson SDK to game developers on the Unity platform. So again, it's this idea of bringing the game developer, the application developer, in closer contact with these trained models, and these trained algorithms. And that's where you're seeing incredible things happen. So for example, Star Trek Bridge Crew, which I don't know how many Trekkies we have here at the CDO Summit. >> A few over here probably. >> Yeah, a couple? They're using our SDK in Unity, to basically allow a gamer to use voice commands through the headset, through a VR headset, to talk to other players in the virtual game. So we're going to see more, I can't really disclose too much what we're doing there, but there's some cool stuff coming out of that partnership. >> Real immersive experience driving a lot of data. Now you're part of the Digital Business Group. I like the term digital business, because we talk about it all the time. Digital business, what's the difference between a digital business and a business? What's the, how they use data. >> Joel: Yeah. >> You're a data person, what does that mean? That you're part of the Digital Business Group? Is that an internal facing thing? An external facing thing? Both? >> It's really both. So our Chief Digital Officer, Bob Lord, he has a presentation that he'll give, where he starts out, and he goes, when I tell people I'm the Chief Digital Officer they usually think I just manage the website. You know, if I tell people I'm a Chief Data Officer, it means I manage our data, in governance over here. The reality is that I think these Chief Digital Officer, Chief Data Officer, they're really responsible for business transformation. And so, if you actually look at what we're doing, I think on both sides is we're using data, we're using marketing technology, martech, like Optimizely, like Segment, like some of these great partners of ours, to really look at how we can quickly A/B test, get user feedback, to look at how we actually test different offerings and market. And so really what we're doing is we're setting up a testing platform, to bring not only our traditional offers to market, like DB2, Mainframe, et cetera, but also bring new offers to market, like blockchain, and quantum, and others, and actually figure out how we get better product-market fit. What actually, one thing, one story that comes to mind, is if you've seen the movie Hidden Figures- >> Oh yeah. >> There's this scene where Kevin Costner, I know this is going to look not great for IBM, but I'm going to say it anyways, which is Kevin Costner has like a sledgehammer, and he's like trying to break down the wall to get the mainframe in the room. That's what it feels like sometimes, 'cause we create the best technology, but we forget sometimes about the last mile. You know like, we got to break down the wall. >> Where am I going to put it? >> You know, to get it in the room! So, honestly I think that's a lot of what we're doing. We're bridging that last mile, between these different audiences. So between developers, between ISVs, between commercial buyers. Like how do we actually make this technology, not just accessible to large enterprise, which are our main clients, but also to the other ecosystems, and other audiences out there. >> Well so that's interesting Joel, because as a potential partner of IBM, they want, obviously your go-to-market, your massive company, and great distribution channel. But at the same time, you want more than that. You know you want to have a closer, IBM always focuses on partnerships that have intrinsic value. So you talked about offerings, you talked about quantum, blockchain, off-camera talking about cloud containers. >> Joel: Yeah. >> I'd say cloud and containers may be a little closer than those others, but those others are going to take a lot of market development. So what are the offerings that you guys are bringing? How do they get into the hands of your partners? >> I mean, the commonality with all of these, all the emerging offerings, if you ask me, is the distributed nature of the offering. So if you look at blockchain, it's a distributed ledger. It's a distributed transaction chain that's secure. If you look at data, really and we can hark back to say, Hadoop, right before object storage, it's distributed storage, so it's not just storing on your hard drive locally, it's storing on a distributed network of servers that are all over the world and data centers. If you look at cloud, and containers, what you're really doing is not running your application on an individual server that can go down. You're using containers because you want to distribute that application over a large network of servers, so that if one server goes down, you're not going to be hosed. And so I think the fundamental shift that you're seeing is this distributed nature, which in essence is cloud. So I think cloud is just kind of a synonym, in my opinion, for distributed nature of our business. >> That's interesting and that brings up, you're right, cloud and Big Data/Hadoop, we don't talk about Hadoop much anymore, but it kind of got it all started, with that notion of leave the data where it is. And it's the same thing with cloud. You can't just stuff your business into the public cloud. You got to bring the cloud to your data. >> Joel: That's right. >> But that brings up a whole new set of challenges, which obviously, you're in a position just to help solve. Performance, latency, physics come into play. >> Physics is a rough one. It's kind of hard to avoid that one. >> I hear your best people are working on it though. Some other partnerships that you want to sort of, elucidate. >> Yeah, no, I mean we have some really great, so I think the key kind of partnership, I would say area, that I would allude to is, one of the things, and you kind of referenced this, is a lot of our partners, big or small, want to work with our top clients. So they want to work with our top banking clients. They want, 'cause these are, if you look at for example, MaRisk and what we're doing with them around blockchain, and frankly, talk about innovation, they're innovating containers for real, not virtual containers- >> And that's a joint venture right? >> Yeah, it is, and so it's exciting because, what we're bringing to market is, I also lead our startup programs, called the Global Entrepreneurship Program, and so what I'm focused on doing, and you'll probably see more to come this quarter, is how do we actually bridge that end-to-end? How do you, if you're startup or a small business, ultimately reach that kind of global business partner level? And so kind of bridging that, that end-to-end. So we're starting to bring out a number of different incentives for partners, like co-marketing, so I'll help startups when they're early, figure out product-market fit. We'll give you free credits to use our innovative technology, and we'll also bring you into a number of clients, to basically help you not burn all of your cash on creating your own marketing channel. God knows I did that when I was at a start-up. So I think we're doing a lot to kind of bridge that end-to-end, and help any partner kind of come in, and then grow with IBM. I think that's where we're headed. >> I think that's a critical part of your job. Because I mean, obviously IBM is known for its Global 2000, big enterprise presence, but startups, again, fuel that innovation fire. So being able to attract them, which you're proving you can, providing whatever it is, access, early access to cloud services, or like you say, these other offerings that you're producing, in addition to that go-to-market, 'cause it's funny, we always talk about how efficient, capital efficient, software is, but then you have these companies raising hundreds of millions of dollars, why? Because they got to do promotion, marketing, sales, you know, go-to-market. >> Yeah, it's really expensive. I mean, you look at most startups, like their biggest ticket item is usually marketing and sales. And building channels, and so yeah, if you're, you know we're talking to a number of partners who want to work with us because of the fact that, it's not just like, the direct kind of channel, it's also, as you kind of mentioned, there's other challenges that you have to overcome when you're working with a larger company. for example, security is a big one, GDPR compliance now, is a big one, and just making sure that things don't fall over, is a big one. And so a lot of partners work with us because ultimately, a number of the decision makers in these larger enterprises are going, well, I trust IBM, and if IBM says you're good, then I believe you. And so that's where we're kind of starting to pull partners in, and pull an ecosystem towards us. Because of the fact that we can take them through that level of certification. So we have a number of free online courses. So if you go to partners, excuse me, ibm.com/partners/learn there's a number of blockchain courses that you can learn today, and will actually give you a digital certificate, that's actually certified on our own blockchain, which we're actually a first of a kind to do that, which I think is pretty slick, and it's accredited at some of the universities. So I think that's where people are looking to IBM, and other leaders in this industry, is to help them become experts in their, in this technology, and especially in this emerging technology. >> I love that blockchain actually, because it's such a growing, and interesting, and innovative field. But it needs players like IBM, that can bring credibility, enterprise-grade, whether it's security, or just, as I say, credibility. 'Cause you know, this is, so much of negative connotations associated with blockchain and crypto, but companies like IBM coming to the table, enterprise companies, and building that ecosystem out is in my view, crucial. >> Yeah, no, it takes a village. I mean, there's a lot of folks, I mean that's a big reason why I came to IBM, three, four years ago, was because when I was in start-up land, I used to work for H20, I worked for Alpine Data Labs, Datameer, back in the Hadoop days, and what I realized was that, it's an opportunity cost. So you can't really drive true global innovation, transformation, in some of these bigger companies because there's only so much that you can really kind of bite off. And so you know at IBM it's been a really rewarding experience because we have done things like for example, we partnered with Girls Who Code, Treehouse, Udacity. So there's a number of early educators that we've partnered with, to bring code to, to bring technology to, that frankly, would never have access to some of this stuff. Some of this technology, if we didn't form these alliances, and if we didn't join these partnerships. So I'm very excited about the future of IBM, and I'm very excited about the future of what our partners are doing with IBM, because, geez, you know the cloud, and everything that we're doing to make this accessible, is bar none, I mean, it's great. >> I can tell you're excited. You know, spring in your step. Always a lot of energy Joel, really appreciate you coming onto theCUBE. >> Joel: My pleasure. >> Great to see you again. >> Yeah, thanks Dave. >> You're welcome. Alright keep it right there, everybody. We'll be back. We're at the IBM CDO Strategy Summit in San Francisco. You're watching theCUBE. (techno music) (touch-tone phone beeps)

(busy music) >> Announcer: Live from Midtown Manhattan, it's the Cube, covering Big Data New York City 2017, brought to you by SiliconANGLE Media and its ecosystem sponsors. >> Hello, and welcome to the special Cube presentation here in New York City for Big Data NYC, in conjunction with all the activity going on with Strata, Hadoop, Strata Data Conference right around the corner. This is the Cube's special annual event in New York City where we highlight all the trends, technology experts, thought leaders, entrepreneurs here inside the Cube. We have our three days of wall to wall coverage, evening event on Wednesday. I'm John Furrier, the co-host of the Cube, with Jim Kobielus, and Peter Burris will be here all week as well. Kicking off day one, Jim, the monster week of Big Data NYC, which now has turned into, essentially, the big data industry is a huge industry. But now, subsumed within a larger industry of AI, IoT, security. A lot of things have just sucked up the big data world that used to be the Hadoop world, and it just kept on disrupting, and creative disruption of the old guard data warehouse market, which now, looks pale in comparison to the disruption going on right now. >> The data warehouse market is very much vibrant and alive, as is the big data market continuing to innovate. But the innovations, John, have moved up the stack to artificial intelligence and deep learning, as you've indicated, driving more of the Edge applications in the new generation of mobile and smart appliances and things that are coming along like smart, self-driving vehicles and so forth. What we see is data professionals and developers are moving towards new frameworks, like TensorFlow and so forth, for development of the truly disruptive applications. But big data is the foundation. >> I mean, the developers are the key, obviously, open source is growing at an enormous rate. We just had the Linux Foundation, we now have the Open Source Summit, they have kind of rebranded that. They're going to see explosion from code from 64 million lines of code to billions of lines of code, exponential growth. But the bigger picture is that it's not just developers, it's the enterprises now who want hybrid cloud, they want cloud technology. I want to get your reaction to a couple of different threads. One is the notion of community based software, which is open source, extending into the enterprise. We're seeing things like blockchain is hot right now, security, two emerging areas that are overlapping in with big data. You obviously have classic data market, and then you've got AI. All these things kind of come in together, kind of just really putting at the center of all that, this core industry around community and software AI, particular. It's not just about machine learning anymore and data, it's a bigger picture. >> Yeah, in terms of a community, development with open source, much of what we see in the AI arena, for example, with the up and coming, they're all open source tools. There's TensorFlow, there's Cafe, there's Theano and so forth. What we're seeing is not just the frameworks for developing AI that are important, but the entire ecosystem of community based development of capabilities to automate the acquisition of training data, which is so critically important for tuning AI, for its designated purpose, be it doing predictions and abstractions. DevOps, what are coming into being are DevOps frameworks to span the entire life cycle of the creation and the training and deployment and iteration of AI. What we're going to see is, like at the last Spark Summit, there was a very interesting discussion from a Stanford researcher, new open source tools that they're developing out in, actually, in Berkeley, I understand, for, related to development of training data in a more automated fashion for these new challenges. The communities are evolving up the stack to address these requirements with fairly bleeding edge capabilities that will come in the next few years into the mainstream. >> I had a chat with a big time CTO last night, he worked at some of the big web scale company, I won't say the name, give it away. But basically, he asked me a question about IoT, how real is it, and obviously, it's hyped up big time, though. But the issue in all this new markets like IoT and AI is the role of security, because a lot of enterprises are looking at the IoT, certainly in the industrial side has the most relevant low hanging fruit, but at the end of the day, the data modeling, as you're pointing out, becomes a critical thing. Connecting IoT devices to, say, an IP network sounds trivial in concept, but at the end of the day, the surface area for security is oak expose, that's causing people to stop what they're doing, not deploying it as fast. You're seeing kind of like people retrenching and replatforming at the core data centers, and then leveraging a lot of cloud, which is why Azure is hot, Microsoft Ignite Event is pretty hot this week. Role of cloud, role of data in IoT. Is IoT kind of stalled in your mind? Or is it bloating? >> I wouldn't say it's stalled or that it's bloating, but IoT is definitely coming along as the new development focus. For the more disruptive applications that can derive more intelligence directly to the end points that can take varying degrees of automated action to achieve results, but also to very much drive decision support in real time to people on their mobiles or in whatever. What I'm getting at is that IoT is definitely a reality in the real world in terms of our lives. It's definitely a reality in terms of the index generation of data applications. But there's a lot of the back end in terms of readying algorithms and in training data for deployment of really high quality IoT applications, Edge applications, that hasn't come together yet in any coherent practice. >> It's emerging, it's emerging. >> It's emerging. >> It's a lot more work to do. OK, we're going to kick off day one, we've got some great guests, we see Rob Bearden in the house, Rob Thomas from IBM. >> Rob Bearden from Hortonworks. >> Rob Bearden from Hortonworks, and Rob Thomas from IBM. I want to bring up, Rob wrote a book just recently. He wrote Big Data Revolution, but he also wrote a new book called, Every Company is a Tech Company. But he mentions, he kind of teases out this concept of a renaissance, so I want to get your thoughts on this. If you look at Strata, Hadoop, Strata Data, the O'Reilly Conference, which has turned into like a marketing machine, right. A lot of hype there. But as the community model grows up, you're starting to see a renaissance of real creative developers, you're starting to see, not just open source, pure, full stack developers doing all the heavy lifting, but real creative competition, in a renaissance, that's really the key. You're seeing a lot more developer action, tons outside of the, what was classically called the data space. The role of data and how it relates to the developer phenomenon that's going on right now. >> Yeah, it's the maker culture. Rob, in fact, about a year or more ago, IBM, at one of their events, they held a very maker oriented event, I think they called it Datapalooza at one point. What it's looking at, what's going on is it's more than just classic software developers are coming to the fore. When you're looking at IoT or Edge applications, it's hardware developers, it's UX developers, it's developers and designers who are trying to change and drive data driven applications into changing the very fabric of how things are done in the real world. What Peter Burris, we had a wiki about him called Programming in the Real World. What that all involves is there's a new set of skill sets that are coming together to develop these applications. It's well beyond just simply software development, it's well beyond simply data scientists. Maker culture. >> Programming in the real world is a great concept, because you need real time, which comes back down to this. I'm looking for this week from the guests we talked to, what their view is of the data market right now. Because if you want to get real time, you've got to move from that batch world to the real time world. I'm not saying batch is over, you've still got to store data, and that's growing at an exponential rate as well. But real time data, how do you use data in real time, how do the modelings work, how do you scale that. How do you take a DevOps culture to the data world is what I'm looking for. What are you looking for this week? >> What I'm looking for this week, I'm looking for DevOps solutions or platforms or environments for teams of data scientists who are building and training and deploying and evaluating, iterating deep learning and machine learning and natural language processing applications in a continuous release pipeline, and productionizing them. At Wikibon, we are going deeper in that whole notion of DevOps for data science. I mean, IBM's called it inside ops, others call it data ops. What we're seeing across the board is that more and more of our customers are focusing on how do we bring it all together, so the maker culture. >> Operationalizing it. >> Operationalizing it, so that the maker cultures that they have inside their value chain can come together and there's a standard pattern workflow of putting this stuff out and productionizing it, AI productionized in the real world. >> Moving in from the proof of concept notion to actually just getting things done, putting it out in the network, and then bringing it to the masses with operational support. >> Right, like the good folks at IBM with Watson data platform, on some levels, is a DevOPs for data science platform, but it's a collaborative environment. That's what I'm looking to see, and there's a lot of other solution providers who are going down that road. >> I mean, to me, if people have the community traction, that is the new benchmark, in my opinion. You heard it here on the Cube. Community continues to scale, you can start seeing it moving out of open source, you're seeing things like blockchain, you're seeing a decentralized Internet now happening everywhere, not just distributed but decentralized. When you have decentralization, community and software really shine. It's the Cube here in New York City all week. Stay with us for wall to wall coverage through Thursday here in New York City for Big Data NYC, in conjunction with Strata Data, this is the Cube, we'll be back with more coverage after this short break. (busy music) (serious electronic music) (peaceful music) >> Hi, I'm John Furrier, the Co-founder of SiliconANGLE Media, and Co-host of the Cube. I've been in the tech business since I was 19, first programming on mini computers in a large enterprise, and then worked at IBM and Hewlett Packard, a total of nine years in the enterprise, various jobs from programming, training, consulting, and ultimately, as an executive sales person, and then started my first company in 1997, and moved to Silicon Valley in 1999. I've been here ever since. I've always loved technology, and I love covering emerging technology. I was trained as a software developer and love business. I love the impact of software and technology to business. To me, creating technology that starts a company and creates value and jobs is probably one of the most rewarding things I've ever been involved in. I bring that energy to the Cube, because the Cube is where all the ideas are, and where the experts are, where the people are. I think what's most exciting about the Cube is that we get to talk to people who are making things happen, entrepreneurs, CEO of companies, venture capitalists, people who are really, on a day in and day out basis, building great companies. In the technology business, there's just not a lot real time live TV coverage, and the Cube is a non-linear TV operation. We do everything that the TV guys on cable don't do. We do longer interviews, we ask tougher questions. We ask, sometimes, some light questions. We talk about the person and what they feel about. It's not prompted and scripted, it's a conversation, it's authentic. For shows that have the Cube coverage, it makes the show buzz, it creates excitement. More importantly, it creates great content, great digital assets that can be shared instantaneously to the world. Over 31 million people have viewed the Cube, and that is the result of great content, great conversations. I'm so proud to be part of the Cube with a great team. Hi, I'm John Furrier, thanks for watching the Cube. >> Announcer: Coming up on the Cube, Tekan Sundar, CTO of Wine Disco. Live Cube coverage from Big Data NYC 2017 continues in a moment. >> Announcer: Coming up on the Cube, Donna Prlich, Chief Product Officer at Pentaho. Live Cube coverage from Big Data New York City 2017 continues in a moment. >> Announcer: Coming up on the Cube, Amit Walia, Executive Vice President and Chief Product Officer at Informatica. Live Cube coverage from Big Data New York City continues in a moment. >> Announcer: Coming up on the Cube, Prakash Nodili, Co-founder and CEO of Pexif. Live Cube coverage from Big Data New York City continues in a moment. (serious electronic music)

>> Announcer: Live from midtown Manhattan, it's theCUBE covering BigData New York City 2017. Brought to you by SiliconANGLE Media, and its ecosystem sponsors. >> Hello everyone, welcome back to our day one, at Big Data NYC, of three days of wall to wall coverage. This is theCUBE. I'm John Furrier, with my co-hosts Jim Kobielus and Peter Burris. We do this event every year, this is theCUBE's BigData NYC. It's our event that we run in New York City. We have a lot of great content, we have theCUBE going live, we don't go to Strata anymore. We do our own event in conjunction, they have their own event. You can go pay over there and get the booth space, but we do our media event and attract all the influencers, the VIPs, the executives, the entrepreneurs, we've been doing it for five years, we're super excited, and thank our sponsors for allowing us to get here and really appreciate the community for continuing to support theCUBE. We're here to wrap up day one what's going on in New York, certainly we've had a chance to check out the Strata situations, Strata Data, which is Cloudera, and O'Reilly, mainly O'Reilly media, they run that, kind of old school event, guys. Let's kind of discuss the impact of the event in context to the massive growth that's going outside of their event. And their event is a walled garden, you got to pay to get in, they're very strict. They don't really let a lot of people in, but, okay. Outside of that the event it going global, the activity around big data is going global. It's more than Hadoop, we certainly thought about that's old news, but what's the big trend this year? As the horizontally scalable cloud enters the equation. >> I think the big trend, John, is the, and we've talked about in our research, is that we have finally moved away from big data, being associated with a new type of infrastructure. The emergence of AI, deep learning, machine learning, cognitive, all these different names for relatively common things, are an indications that we're starting to move up into people thinking about applications, people thinking about services they can use to get access, or they can get access to build their applications. There's not enough skills. So I think that's probably the biggest thing is that the days of failure being measured by whether or not you can scale your cluster up, are finally behind us. We're using the cloud, other resources, we have enough expertise, the technologies are becoming simpler and more straightforward to do that. And now we're thinking about how we're going to create value out of all of this, which is how we're going to use the data to learn something new about what we're doing in the organization, combine it with advanced software technologies that actually dramatically reduce the amount of work that's necessary to make a decision. >> And the other trend I would say, on top of that, just to kind of put a little cherry on top of that, kind of the business focus which is again, not the speeds and feeds, although under the hood, lot of great innovation going on from deep learning, and there's a ton of stuff. However, the conversation is the business value, how it's transforming work and, but the one thing that nobody's talking about is, this is why I'm not bullish on these one shows, one show meets all kind of thing like O'Reilly Media does, because there's multiple personas in a company now in the ecosystem. There are now a variety of buyers of some products. At least in the old days, you'd go talk to the IT CIO and you're in. Not anymore. You have an analytics person, a Chief Data Officer, you might have an IT person, you might have a cloud person. So you're seeing a completely broader set of potential buyers that are driving the change. We heard Paxata talk about that. And this is a dynamic. >> Yeah, definitely. We see a fair amount of, what I'm sensing about Strata, how it's evolving these big top shows around data, it's evolving around addressing a broader, what we call maker culture. It's more than software developers. It's business analysts, it's the people who build the hardware for the internet of things into which AI and machine learning models are being containerized and embedded. I've, you know, one of the takeaways from today so far, and the keynotes are tomorrow at Strata, but I've been walking the atrium at the Javits Center having some interesting conversations, in addition, of course, to the ones we've been having here at theCUBE. And what I'm notic-- >> John: What are those hallway conversations that you're having? >> Yeah. >> What's going on over there? >> Yeah, what I've, the conversations I've had today have been focused on, the chief trend that I'm starting to sense here is that the productionization of the machine learning development process or pipeline, is super hot. It spans multiple data platforms, of course. You've got a bit of Hadoop in the refinery layer, you've got a bit of in-memory columnar databases, like the Act In discussed at their own, but the more important, not more important, but just as important is that what users are looking at is how can we build these DevOps pipelines for continuous management of releases of machine learning models for productionization, but also for ongoing evaluation and scoring and iteration and redeployment into business applications. You know there's, I had conversations with Mapbar, I had conversations with IBM, I mean, these were atrium conversations about things that they are doing. IBM had an announcement today on the wires and so forth with some relevance to that. And so I'm seeing a fair, I'm hearing, I'm sensing a fair amount of It's The Apps, it's more than just Hadoop. But it's very much the flow of these, these are the core pieces, like AI, core pieces of intellectual property in the most disruptive applications that are being developed these days in all manner, in business and industry in the consumer space. >> So I did not go over to the show floor yet, I've not been over to the Atrium. But, I'll bet you dollars to donuts this is indicative of something that always happens in a complex technology environment. And again, this is something we've thought about particularly talked about here on theCUBE, in fact we talked to Paxata about it a little bit as well. And that is, as an organization gains experience, it starts to specialize. But there's always moments, there' always inflection points in the process of gaining that experience. And by that, or one of the indications of that is that you end up with some people starting to specialize, but not quite sure what they're specializing in yet. And I think that's one of the things that's happening right now is that the skills gap is significant. At the same time that the skills gap is being significant, we're seeing people start to declare their specializations that they don't have skills, necessarily, to perform yet. And the tools aren't catching up. So there's still this tension model, open source, not necessarily focusing on the core problem. Skills looking for tools, and explosion in the number of tools out there, not focused on how you simplify, streamline, and put into operation. How all these things work together. It's going to be an interesting couple of years, but the good news, ultimately, is that we are starting to see for the first time, even on theCUBE interviews today, the emergence of a common language about how we think about the characteristics of the problem. And I think that that heralds a new round of experience and a new round of thinking about what is all the business analysts, the data scientists, the developer, the infrastructure person, business person. >> You know, you bring up that comment, those comments, about the specialists and the skills. We talked, Jim and I talked on the segment this morning about tool shed. We're talking about there are so many tools out there, and everyone loves a good tool, a hammer. But the old expression is if you're a hammer, everything looks like a nail, that's cliche. But what's happened is there are a plethora of tools, right, and tools are good. Platforms are better. As people start to replatformize everything they could have too many tools. So we asked the C Chief Data Officer, he goes yeah, I try to manage the tool tsunami, but his biggest issue was he buys a hammer, and it turns into a lawnmower. That's a vendor mentality of-- >> What a truck. Well, but that's a classic example of what I'm talking about. >> Or someone's trying to use a hammer to mow the lawn right? Again, so this is what you're getting at. >> Yeah! >> The companies out there are groping for relevance, and that's how you can see the pretenders from the winners. >> Well, a tool, fundamentally, is pedagogical. A tool describes the way work is going to be performed, and that's been a lot of what's been happening over the course of the past few years. Now, businesses that get more experience, they're describing their own way of thinking throughout a problem. And they're still not clear on how to bring the tools together because the tools are being generated, put into the marketplace by an expanding array of folks and companies, and they're now starting to shuffle for position. But I think ultimately, what we're going to see happen over the next year and I think this is an inflection point, going back to this big tent notion, is the idea that ultimately we are going to see greater specialization over the next few years. My guess is that this year will probably, should get better, or should get bigger, I'm not certain it will because it's focused on the problems that we already solved and not moving into the problems that we need to focus on. >> Yeah, I mean, a lot of the problems I have with the O'Reilly show is that they try to throw default leadership out there, and there's some smart people that go to that, but the problem is is that it's too monetization, they try to make too much money from the event when this action's happening. And this is where the tool becomes, the hammer becomes a lawnmower, because what's happening is that the vendor's trying to stay alive. And you mentioned this earlier, to your point, the customers that are buyers of the technology don't want to have something that's not going to be a fit, that's going to be agile from us. They don't want the hammer that they bought to turn into something that they didn't buy it for. And sometimes, teams can't make that leap, skillset-wise, to literally pivot overnight. Especially as a startup. So this is where the selection of the companies makes a big difference. And a lot of the clients, a lot of customers that we're serving on the end user side are reaching the conclusion that the tools themselves, while important, are clearly not where the value is. The value is in how they put them together for their business. And that's something that's going to have to, again, that's a maturation process, roles, responsibilities, the chief data officer, they're going to have a role in that or not, but ultimately, they're going to have to start finding their pipelines, their process for ingestion out to analysis. >> Let me get your reaction, you guys, your reactions to this tape. Because one of the things that I heard today, and I think this validates a bigger trend as we talk about the landscape of the markup from the event to how people are behaving and promoting and building products and companies. The pattern that I'm hearing, we said it multiple times on theCUBE today and one from the guy who's basically reading the script, is, in his interview, explaining 'cause it's so factual, I asked him the straight-up question, how do you deal with suppliers? What's happening is the trend is don't show me sizzle. I want to see the steak. Don't sell me hype, I got too many business things to work on right now, I need to nail down some core things. I got application development, I got security to build out big time, and then I got all those data channels that I need, I don't have time for you to sell me a hammer that might not be a hammer in the future! So I need real results, I need real performance that's going to have a business impact. That is the theme, and that trumps the hype. I see that becoming a huge thing right now. Your thoughts, reactions, guys-- >> Well I'll start-- >> What's your reaction then? True or false on the trend? Be-- >> Peter: True! >> Get down to business. >> I'll say that much, true, but go ahead. >> I'll say true as well, but let me just add some context. I think a show like O'Reilly Strata is good up to a point, especially to catalyze an industry, a growing industry like big data's own understanding of it, of the value that all these piece parts, Hadoop and Spark and so forth, can add, can provide when deployed in a unit according to some emerging patterns, whatever. But at a certain point where a space like this becomes well-established, it just becomes a pure marketing event. And customers, at a certain point say, you know, I come here for ideas about things that I can do in my environ, my business, that could actually many ways help me to do new things. You know, you can't get that at a marketing-oriented, you can get that, as a user, more at a research-oriented show. When it's an emerging market, like let's say Spark has been, like the Spark Summit was in the beginning, those are kind of like, when industries go through the phase those are sort of in the beginning, sort of research-focused shows where industry, the people who are doing the development of this new architecture, they talk ideas. Now I think in 2017, where we're at now, is what the idea is everybody's trying to get their heads around, they're all around AI, what the heck that is. For a show like an O'Reilly Ready show to have relevance in a market that's in this much ferment of really innovation around AI and deep learning, there needs to be a core research focus that you don't get at this point in the lifecycle of Strata, for example. So that's my take on what's going on. >> So, my take is this. And first of all, I agree with everything you said, so it's not in opposition to anything. Many years ago I had this thought that I think still is very true. And that is the value of industry, the value of infrastructure is inversely correlated with the degree to which anybody knows anything about it. So if I know a lot about my infrastructure, it's not creating a lot of business value. In fact, more often than not, it's not working, which is why people end up knowing more about it. But the problem is, the way that technology has always been sold is as a differentiated, some sort of value-add thing. So you end up with this tension. And this is an application domain, a very, very complex application domain like big data. The tension is, my tool is so great that, and it's differentiating all those other stuff, yeah but it becomes valuable to me if and only if nobody knows it exists. So I think, and one of the reasons why I bring this up, John, is many of the companies that are in the big data space today that are most successful are companies that are positioning themselves as a service. There's a lot of interesting SaaS applications for big data analysis, pipeline management, all the other things you can talk about, that are actually being rendered as a service, and not as a product. So that all you need to know is what the tool does. You don't need to know the tool. And I don't know that that's necessarily going to last, but I think it's very, very interesting that a lot of the more successful companies that we're talking to are themselves mere infrastructure SaaS companies. >> Because-- >> AtScale is interesting, though. They came in as a service. But their service has an interesting value proposition. They can allow you to essentially virtualize the data to play with it, so people can actually sandbox data. And if it gets traction, they can then double-down on it. So to me that's a freebie. To me, I'm a customer, I got to love that kind of environment because you're essentially giving almost a developer-like environment-- >> Peter: Value without necessarily-- >> Yeah, the cost, and the guy gets the signal from the marketplace, his customer, of what data resolves. To me that's a very cool scene. I don't, you saying that's bad, or? >> No, no, I think it's interesting. I think it's-- >> So you're saying service is-- >> So what I'm saying is, what I'm saying is, that the value of infrastructure is inversely proportional to the degree to which anybody knows anything about it. But you've got a bunch of companies who are selling, effectively, infrastructure software, so it's a value-add thing, and that creates a problem. And a lot of other companies not only have the ability to sell something as a service as opposed to a product, they can put the service froward, and people are using the service and getting what they need out of it without knowing anything about the tool. >> I like that. Let me just maybe possibly restate what you just said. When a market goes toward a SaaS go-to-market delivery model for solutions, the user, the buyer's focus is shifted away from what the solution can do, I mean, how it works under the cover. >> Peter: Quote, value-add-- >> To what it can do potentially for you. >> The business, that's right. >> But you're not going to, don't get distracted by the implementation details. You have then as a user become laser-focused on, wow, there's a bunch of things that this can do for me. I don't care how it works, really. You SaaS provider, you worry about that stuff. I can worry now about somehow extracting the value. I'm not distracted. >> This show, or this domain, is one of the domains where SaaS has moved, just as we're thinking about moving up the stack, the SaaS business model is moving down the stack in the big data world. >> All right, so, in summary, the stack is changing. Predictions for the next few days. What are we going to see come out of Strata Data, and our BigData NYC? 'Cause remember, this show was always a big hit, but it's very clear from the data on our dashboards, we're seeing all the social data. Microsoft Ignite is going on, and Microsoft Azure, just in the past few years, has burst on the scene. Cloud is sucking the oxygen out of the big data event. Or is it? >> I doubt it was sucking it out of the event, but you know, theCUBE is in, theCUBE is not at Ignite. Where's theCUBE right now? >> John: BigData NYC. >> No, it's here, but it's also at the Splunk show. >> John: That's true. >> And isn't it interesting-- >> John: We're sucking the data out of two events. >> Did a lot of people coming in, exactly. A lot of people coming-- >> We're live streaming in a streaming data kind of-- >> John just said we suck, there's that record saying that. >> We're sucking all the data. >> So we are-- >> We're sharing data. These videos are data-driven. >> Yeah, absolutely, but the point is, ultimately, is that, is that Splunk is an example of a company that's putting forward a service about how you do this and not necessarily a product focus. And a lot of the folks that are coming on theCUBE here are also going on to theCUBE down in Washington D.C., which is where the Splunk show's at. And so I think one of the things, one of the predictions I'll make, is that we're going to hear over the next couple of days more companies talk about their SaaS trash. >> Yeah, I mean I just think, I agree with you, but I also agree with the comments about the technology coming together. And here's one thing I want to throw on the table. I've gotten the sense a few times about connecting the dots on it, we'll put it out publicly for comment right now. The role that communities will play outside of developer, is going to be astronomical. I think we're seeing signals, certainly open-source communities have been around for a long time. They continue to grow shoulders of giants before them. Even these events like O'Reilly, which are a small community that they rely on is now not the only game in town. We're seeing the notion of a community strategy in things like Blockchain, you're seeing it in business, you're seeing people rolling out their recruitment to say, data scientists. You're seeing a community model developing in business, yes or no? >> Yes, but I would say, I would put it this way, John. That it's always been there. The difference is that we're now getting enough experience with things that have occurred, for example, collaboration, communal, communal collaboration in open-source software that people are now saying, and they've developed a bunch of social networking techniques where they can actually analyze how those communities work together, but now they're saying, hmm, I've figured out how to do an assessment analysis understanding that community. I'm going to see if I can take that same concept and apply it over here to how sales works, or how B-to-B engagement works, or how marketing gets conducted, or how sales and marketing work together. And they're discovering that the same way of thinking is actually very fruitful over there. So I totally agree, 100%. >> So they don't rely on other people's version of a community, they can essentially construct their own. >> They are, they are-- >> John: Or enabling their own. >> That's right, they are bringing that approach to thinking about a community-driven business and they're applying it to a lot of new ways, and that's very exciting. >> As the world gets connected with mobile and internet of things as we're seeing, it's one big online community. We're seeing things, I'm writing a post right now, what you could, what B-to-B markets should learn from the fake news problem. And that is content and infrastructure are now contextually tied together. >> Peter: Totally. >> And related. The payload of the fake news is also related to the gamification of the network effect, hence the targeting, hence the weaponization. >> Hey, we wrote the three Cs, we wrote a piece on the three Cs of strategy a year and a half ago. Content, community, context. And at the end of the day, the most important thing to what you're saying about, is that there is, you know, right now people talk about social networking. Social media, you think Facebook. Facebook is a community with a single context, stay in touch with your friends. >> Connections. >> Connections. But what you're really saying is that for the first time we're now going to see an enormous amount of technology being applied to the fullness of all the communities. We're going to see a lot more communities being created with the software, each driven by what content does, creates value, against the context of how it works, where the community's defined in terms of what do we do? >> Let me focus on the fact that bringing, using community as a framework for understanding how the software world is evolving. The software world is evolving towards, I've said this many times in my work about a resurge, the data scientists or data people, data science skills are the core developers in this new era. Now, what is data science all about at its heart? Machine learning, building, and training machine learning models. And so training machine learning models is everything towards making sure that they are fit for their predicted purpose of classification. Training data, where you get all the training data from to feed all, to train all these models? Where do you get all the human resources to label, to do the labeling of the data sets, and so forth, that you need communities, crowdsourcing and whatnot, and you need sustainable communities that can supply the data and the labeling services, and so forth, to be able to sustain the AI and machine learning revolution. So content, creating data and so forth, really rules in this new era, like-- >> The interest in machine learning is at an all-time high, I guess. >> Jim: Yeah, oh yeah, very much so. >> Got it, I agree. I think the social grab, interest grab, value grab is emerging. I think communities, content, context, communities are relevant. I think a lot of things are going to change, and that the scuttlebutt that I'm hearing in this area now is it's not about the big event anymore. It's about the digital component. I think you're seeing people recognize that, but they still want to do the face-to-face. >> You know what, that's right. That's right, they still want, let's put it this way. That there are, that the whole point of community is we do things together. And there are some things that are still easier to do together if we get together. >> But B-to-B marketing, you just can't say, we're not going to do events when there's a whole machinery behind events. Legion batch marketing, we call it. There's a lot of stuff that goes on in that funnel. You can't just say hey, we're going to do a blog post. >> People still need to connect. >> So it's good, but there's some online tools that are happening, so of course. You wanted to say something? >> Yeah, I just want to say one thing. Face to face validates the source of expertise. I don't really fully trust an expert, I can't in my heart engage with them, 'til I actually meet them and figure out in person whether they really do have the goods, or whether they're repurposing some thinking that they got from elsewhere and they gussy it up. So face, there's no substitute for face-to-face to validate the expertise. The expertise that you value enough to want to engage in your solution, or whatever it might be. >> Awesome, I agree. Online activities, the content, we're streaming the data, theCUBE, this is our annual event in New York City. We've got three days of coverage, Tuesday, Wednesday, Thursday, here, theCUBE in Manhattan, right around the corner from Strata Hadoop, the Javits Center of influencers. We're here with the VIPs, with the entrepreneurs, with the CEOs and all the top analysts from WikiBon and around the community. Be there tomorrow all day, day one wrap up is done. Thanks for watching, see you tomorrow. (rippling music)

>> Presenter: From the Fairmont Hotel, in the heart of Silicon Valley, it's theCUBE. Covering When IoT Met AI The Intelligence of Things. Brought to you by Western Digital. >> Hey welcome back everybody, Jeff Frick here with theCUBE. We're in downtown San Jose at the Fairmont Hotel, at an event called When IoT Met AI The Intelligence of Things. You've heard about the internet of things, and on the intelligence of things, it's IoT, it's AI, it's AR, all this stuff is really coming to play, it's very interesting space, still a lot of start-up activity, still a lot of big companies making plays in this space. So we're excited to be here, and really joined by our host, big thanks to Western Digital for hosting this event with WDLabs' Dave Tang. Got newly promoted since last we spoke. The SVP of corporate marketing and communications, for Western Digital, Dave great to see you as usual. >> Well, great to be here, thanks. >> So I don't think the need for more storage is going down anytime soon, that's kind of my takeaway. >> No, no, yeah. If this wall of data just keeps growing. >> Yeah, I think the term we had yesterday at the Ag event that we were at, also sponsored by you, is really the flood of data using an agricultural term. But it's pretty fascinating, as more, and more, and more data is not only coming off the sensors, but coming off the people, and used in so many more ways. >> That's right, yeah we see it as a virtual cycle, you create more data, you find more uses for that data to harness the power and unleash the promise of that data, and then you create even more data. So, when that virtual cycle of creating more, and finding more uses of it, and yeah one of the things that we find interesting, that's related to this event with IoT and AI, is this notion that data is falling into two general categories. There's big data, and there's fast data. So, big data I think everyone is quite familiar with by this time, these large aggregated likes of data that you can extract information out of. Look for insights and connections between data, predict the future, and create more prescriptive recommendations, right? >> Right. >> And through all of that you can gain algorithms that help to make predictions, or can help machines run based on that data. So we've gone through this phase where we focused a lot on how we harness big data, but now we're taking these algorithms that we've gleaned from that, and we're able to put them in real time applications, and that's sort of been the birth of fast data, it's been really-- >> Right, the streaming data. We cover Spark Summit, we cover Flink, and New, a new kind of open source project that came out of Berlin. That some people would say the next generation of Spark, and the other thing, you know, good for you guys, is that it used to be, not only was it old data, but it was a sampling of old data. Now on this new data, and the data stream that's all of the data. And I would actually challenge, I wonder if that separation as you describe, will stay, because I got to tell you, the last little drive I bought, just last week, was an SSD drive, you know, one terabyte. I needed some storage, and I had a choice between spinning disc and not, and I went with the flat. I mean, 'cause what's fascinating to me, is the second order benefits that we keep hearing time, and time, and time again, once people become a data-driven enterprise, are way more than just that kind of top-level thing that they thought. >> Exactly, and that's sort of that virtual cycle, you got to taste, and you learn how to use it, and then you want more. >> Jeff: Right, right. >> And that's the great thing about the breadth of technologies and products that Western Digital has, is from the solid state products, the higher performance flash products that we have, to the higher capacity helium-filled drive technologies, as well as devices going on up into systems, we cover this whole spectrum of fast data and big data. >> Right, right. >> I'll give an example. So credit card fraud detection is an interesting area. Billions of dollars potentially being lost there. Well to learn how to predict when transactions are fraudulent, you have to study massive amounts of data. Billions of transactions, so that's the big data side of it, and then as soon as you do that, you can take those algorithms and run them in real time. So as transactions come in for authorization, those algorithms can determine, before they're approved, that one's fraudulent, and that one's not. Save a lot of time and processing for fraud claims. So that's a great example of once you learn something from big data, you apply it to the real-time realm, and it's quite dire right? And then that spawned you to collect even more data, because you want to find new applications and new uses. >> Right, and too kind of this wave of computing back and forth from the shared services computer, then the desktop computer, now it's back to the cloud, and then now it's-- >> Dave: Out with the edge. >> IoT, it's all about the edge. >> Yeah, right. >> And at the end of the day, it's going to be application-specific. What needs to be processed locally, what needs to be processed back at the computer, and then all the different platforms. We were again at a navigation for autonomous vehicles show, who knew there was such a thing that small? And even the attributes of the storage required in the ecosystem of a car, right? And the environmental conditions-- >> That's right. >> Is the word I'm looking for. Completely different, new opportunity, kind of new class of hardware required to operate in that environment, and again that still combines cloud and Edge, sensors and maps. So just the, I don't think that the man's going down David. >> Yeah, absolutely >> I think you're in a good spot. (Jeff laughing) >> You're absolutely right, and even though we try to simplify into fast data, and big data, and Core and Edge, what we're finding is that applications are increasingly specialized, and have specialized needs in terms of the type of data. Is it large amounts of data, is it streaming? You know, what are the performance characteristics, and how is it being transformed, what's the compute aspect of it? And what we're finding, is that the days of general-purpose compute and storage, and memory platforms, are fading, and we're getting into environments with increasingly specialized architectures, across all those elements. Compute, memory and storage. So that's what's really exciting to be in our spot in the industry, is that we're looking at creating the future by developing new technologies that continue to fuel that growth even further, and fuel the uses of data even further. >> And fascinating just the ongoing case of Moore's law, which I know is not, you know you're not making microprocessors, but I think it's so powerful. Moore's law really is a philosophy, as opposed to an architectural spec. Just this relentless pace of innovation, and you guys just continue to push the envelope. So what are your kind of priorities? I can't believe we're halfway through 2017 already, but for kind of the balance of the year kind of, what are some of your top-of-mind things? I know it's exciting times, you're going through the merger, you know, the company is in a great space. What are your kind of top priorities for the next several months? >> Well, so, I think as a company that has gone through serial acquisitions and integrations, of course we're continuing to drive the transformation of the overall business. >> But the fun stuff right? It's not to increase your staff (Jeff laughing). >> Right, yeah, that is the hardware. >> Stitching together the European systems. >> But yeah, the fun stuff includes pushing the limits even further with solid state technologies, with our 3D NAND technologies. You know, we're leading the industry in 64 layer 3D NAND, and just yesterday we announced a 96 layer 3D NAND. So pushing those limits even further, so that we can provide higher capacities in smaller footprints, lower power, in mobile devices and out on the Edge, to drive all these exciting opportunities in IoT an AI. >> It's crazy, it's crazy. >> Yeah it is, yeah. >> You know, terabyte SD cards, terabyte Micro SD cards, I mean the amount of power that you guys pack into these smaller and smaller packages, it's magical. I mean it's absolutely magic. >> Yeah, and the same goes on the other end of the spectrum, with high-capacity devices. Our helium-filled drives are getting higher and higher capacity, 10, 12, 14 terabyte high-capacity devices for that big data core, that all the data has to end up with at some point. So we're trying to keep a balance of pushing the limits on both ends. >> Alright, well Dave, thanks for taking a few minutes out of your busy day, and congratulations on all your success. >> Great, good to be here. >> Alright, he's Dave Tang from Western Digital, he's changing your world, my world, and everyone else's. We're here in San Jose, you're watching theCUBE, thanks for watching.

>> (Announcer) Live from San Jose, in the heart of Silicon Valley, it's theCUBE covering DataWorks Summit 2017, brought to you by Hortonworks. >> Hi everybody, this is George Gilbert, Big Data and Analytics Analyst with Wikibon. We are wrapping up our show on theCUBE today at DataWorks 2017 in San Jose. It has been a very interesting day, and we have a special guest to help us do a survey of the wrap-up, George Chow from Simba. We used to call him Chief Technology Officer, now he's Technology Fellow, but when we was explaining the different in titles to me, I thought he said Technology Felon. (George Chow laughs) But he's since corrected me. >> Yes, very much so >> So George and I have been, we've been looking at both Spark Summit last week and DataWorks this week. What are some of the big advances that really caught your attention? >> What's caught my attention actually is how much manufacturing has really, I think, caught into the streaming data. I think last week was very notable that both Volkswagon and Audi actually had case studies for how they're using streaming data. And I think just before the break now, there was also a similar session from Ford, showcasing what they are doing around streaming data. >> And are they using the streaming analytics capabilities for autonomous driving, or is it other telemetry that they're analyzing? >> The, what is it, I think the Volkswagon study was production, because I still have to review the notes, but the one for Audi was actually quite interesting because it was for managing paint defect. >> (George Gilbert) For paint-- >> Paint defect. >> (George Gilbert) Oh. >> So what they were doing, they were essentially recording the environmental condition that they were painting the cars in, basically the entire pipeline-- >> To predict when there would be imperfections. >> (George Chow) Yes. >> Because paint is an extremely high-value sort of step in the assembly process. >> Yes, what they are trying to do is to essentially make a connection between downstream defect, like future defect, and somewhat trying to pinpoint the causes upstream. So the idea is that if they record all the environmental conditions early on, they could turn around and hopefully figure it out later on. >> Okay, this sounds really, really concrete. So what are some of the surprising environmental variables that they're tracking, and then what's the technology that they're using to build model and then anticipate if there's a problem? >> I think the surprising finding they said were actually, I think it was a humidity or fan speed, if I recall, at the time when the paint was being applied, because essentially, paint has to be... Paint is very sensitive to the condition that is being applied to the body. So my recollection is that one of the finding was that it was a narrow window during which the paint were, like, ideal, in terms of having the least amount of defect. >> So, had they built a digital twin style model, where it's like a digital replica of some aspects of the car, or was it more of a predictive model that had telemetry coming at it, and when it's an outside a certain bounds they know they're going to have defects downstream? >> I think they're still working on the predictive model, or actually the model is still being built, because they are essentially trying to build that model to figure out how they should be tuning the production pipeline. >> Got it, so this is sort of still in the development phase? >> (George Chow) Yeah, yeah >> And can you tell us, did they talk about the technologies that they're using? >> I remember the... It's a little hazy now because after a couple weeks of conference, so I don't remember the specifics because I was counting on the recordings to come out in a couples weeks' time. So I'll definitely share that. It's a case study to keep an eye on. >> So tell us, were there other ones where this use of real-time or near real-time data had some applications that we couldn't do before because we now can do things with very low latency? >> I think that's the one that I was looking forward to with Ford. That was the session just earlier, I think about an hour ago. The session actually consisted of a demo that was being done live, you know. It was being streamed to us where they were showcasing the data that was coming off a car that's been rigged up. >> So what data were they tracking and what were they trying to anticipate here? >> They didn't give enough detail, but it was basically data coming off of the CAN bus of the car, so if anybody is familiar with the-- >> Oh that's right, you're a car guru, and you and I compare, well our latest favorite is the Porche Macan >> Yes, yes. >> SUV, okay. >> But yeah, they were looking at streaming the performance data of the car as well as the location data. >> Okay, and... Oh, this sounds more like a test case, like can we get telemetry data that might be good for insurance or for... >> Well they've built out the system enough using the Lambda Architecture with Kafka, so they were actually consuming the data in real-time, and the demo was actually exactly seeing the data being ingested and being acted on. So in the case they were doing a simplistic visualization of just placing the car on the Google Map so you can basically follow the car around. >> Okay so, what was the technical components in the car, and then, how much data were they sending to some, or where was the data being sent to, or how much of the data? >> The data was actually sent, streamed, all the way into Ford's own data centers. So they were using NiFi with all the right proxy-- >> (George Gilbert) NiFi being from Hortonworks there. >> Yeah, yeah >> The Hortonworks data flow, okay >> Yeah, with all the appropriate proxys and firewall to bring it all the way into a secure environment. >> Wow >> So it was quite impressive from the point of view of, it was life data coming off of the 4G modem, well actually being uploaded through the 4G modem in the car. >> Wow, okay, did they say how much compute and storage they needed in the device, in this case the car? >> I think they were using a very lightweight platform. They were streaming apparently from the Raspberry Pi. >> (George Gilbert) Oh, interesting. >> But they were very guarded about what was inside the data center because, you know, for competitive reasons, they couldn't share much about how big or how large a scale they could operate at. >> Okay, so Simba has been doing ODBC and JDBC drivers to standard APIs, to databases for a long time. That was all about, that was an era where either it was interactive or batch. So, how is streaming, sort of big picture, going to change the way applications are built? >> Well, one way to think about streaming is that if you look at many of these APIs, into these systems, like Spark is a good example, where they're trying to harmonize streaming and batch, or rather, to take away the need to deal with it as a streaming system as opposed to a batch system, because it's obviously much easier to think about and reason about your system when it is traditional, like in the traditional batch model. So, the way that I see it also happening is that streaming systems will, you could say will adapt, will actually become easier to build, and everyone is trying to make it easier to build, so that you don't have to think about and reason about it as a streaming system. >> Okay, so this is really important. But they have to make a trade-off if they do it that way. So there's the desire for leveraging skill sets, which were all batch-oriented, and then, presumably SQL, which is a data manipulation everyone's comfortable with, but then, if you're doing it batch-oriented, you have a portion of time where you're not sure you have the final answer. And I assume if you were in a streaming-first solution, you would explicitly know whether you have all the data or don't, as opposed to late arriving stuff, that might come later. >> Yes, but what I'm referring to is actually the programming model. All I'm saying is that more and more people will want streaming applications, but more and more people need to develop it quickly, without having to build it in a very specialized fashion. So when you look at, let's say the example of Spark, when they focus on structured streaming, the whole idea is to make it possible for you to develop the app without having to write it from scratch. And the comment about SQL is actually exactly on point, because the idea is that you want to work with the data, you can say, not mindful, not with a lot of work to account for the fact that it is actually streaming data that could arrive out of order even, so the whole idea is that if you can build applications in a more consistent way, irrespective whether it's batch or streaming, you're better off. >> So, last week even though we didn't have a major release of Spark, we had like a point release, or a discussion about the 2.2 release, and that's of course very relevant for our big data ecosystem since Spark has become the compute engine for it. Explain the significance where the reaction time, the latency for Spark, went down from several hundred milliseconds to one millisecond or below. What are the implications for the programming model and for the applications you can build with it. >> Actually, hitting that new threshold, the millisecond, is actually a very important milestone because when you look at a typical scenario, let's say with AdTech where you're serving ads, you really only have, maybe, on the order about 100 or maybe 200 millisecond max to actually turn around. >> And that max includes a bunch of things, not just the calculation. >> Yeah, and that, let's say 100 milliseconds, includes transfer time, which means that in your real budget, you only have allowances for maybe, under 10 to 20 milliseconds to compute and do any work. So being able to actually have a system that delivers millisecond-level performance actually gives you ability to use Spark right now in that scenario. >> Okay, so in other words, now they can claim, even if it's not per event processing, they can claim that they can react so fast that it's as good as per event processing, is that fair to say? >> Yes, yes that's very fair. >> Okay, that's significant. So, what type... How would you see applications changing? We've only got another minute or two, but how do you see applications changing now that, Spark has been designed for people that have traditional, batch-oriented skills, but who can now learn how to do streaming, real-time applications without learning anything really new. How will that change what we see next year? >> Well I think we should be careful to not pigeonhole Spark as something built for batch, because I think the idea is that, you could say, the originators, of Spark know that it's all about the ease of development, and it's the ease of reasoning about your system. It's not the fact that the technology is built for batch, so the fact that you could use your knowledge and experience and an API that actually is familiar, should leverage it for something that you can build for streaming. That's the power, you could say. That's the strength of what the Spark project has taken on. >> Okay, we're going to have to end it on that note. There's so much more to go through. George, you will be back as a favorite guest on the show. There will be many more interviews to come. >> Thank you. >> With that, this is George Gilbert. We are DataWorks 2017 in San Jose. We had a great day today. We learned a lot from Rob Bearden and Rob Thomas up front about the IBM deal. We had Scott Gnau, CTO of Hortonworks on several times, and we've come away with an appreciation for a partnership now between IBM and Hortonworks that can take the two of them into a set of use cases that neither one on its own could really handle before. So today was a significant day. Tune in tomorrow, we have another great set of guests. Keynotes start at nine, and our guests will be on starting at 11. So with that, this is George Gilbert, signing out. Have a good night. (energetic, echoing chord and drum beat)

>> Announcer: Live from San Francisco, it's The Cube. Covering Spark Summit 2017. Brought to you by Databricks. >> The Cube is live once again from Spark Summit 2017, I'm David Goad, your host, here with George Gilbert, and we are interviewing many of the speakers that we saw on stage this morning at the keynote. Happy to introduce our next guest on the show, his name is Matt Fryer, Matt, how're you doing? >> Matt: Very well. >> You're the chief, Chief Data Science Officer, I don't see many CDSOs out there, is that a common-- >> I think to say, it's a newer title, and it's coming, I think, where companies that feel the use of data, data science and algorithms, are fundamental to their, their futures. They're creating both the mix of commercial, technical, and algorithmic skill sets, this one team, and to execute together, and that's where the title came from. There's more coming, there's a number of-- Facebook have a few, that's one for example, but it's a newer title, I think it's going to become larger and larger, as time goes on. >> David: So, the CDSO for Hotels.com, something else we learned about you that you may not want me to reveal, but I heard you were the inspiration for Captain Obvious, is that true? >> Uh, that's not true. (laughter) I think Captain Obvious is only an expression of my brand, so there's an awesome brand team, at our office in Dallas. (crosstalk) We all love the captain, he has some good humorous moments, and he keeps us all kind of happy. >> Oh, yeah, he states the obvious, we're going to talk about some of the obvious, and maybe some of the not-so obvious here in this interview. So let's talk a little bit about company culture, because you talked a lot on the stage this morning about customer-first kind of approach, rather than a, "Ooh, look what I can do with the technology." Talk a little bit more about the culture at Hotels.com. >> And that's important, and I think, we're a very data-driven culture, I think most tech companies, and travel, technology companies have that kind of ethos. But fundamentally, the focus and the reason we exist is for the customer. So we want to bring, and actually-- in even better ways than that, I think it's the people. So whether it's the focus on the customer, if we did the right thing by the customer, we fundamentally want you to use our platform time and time again. Whatever need you have, booking, lodging and travel, please use our platform. That's the crucial win. So, to do that, we have to always delight you in every experience you have with us. And equally about people, it's about the team, so we have an internal concept called being supportive. So the whole part of our team culture, is that everybody helps everybody else out, we don't single things out, we're all part of the same team, and we all win if all of us pull together. That makes it a great place, a fun place to work, we're going to play with some new technologies, tech is important to us, but actually the people are even more important to us. >> In part why you love the Spark Summit then, huh? Same kind of spirit here, right? >> It's great, I think it's my third Spark Summit, my second time over in San Francisco, and the size of it is very impressive now. I just love meeting other people learning about some of the things they're up to, how we can apply those back to our business, and hopefully sharing a little bit of what we're up to. >> David: Let's dive into how you're applying it to your business, you talked about this evolution toward becoming an algorithm business, what does that mean and what part does Spark play in that? >> Matt: I think what it is, is about how do you, if you think about a bit of the journey, historically, a lot of the opportunity came in building new features, constantly building it, it's almost like a semi arms race, about how to build more and more features. The crucial thing I think going forward, and particularly with mobile devices now, we have over half our traffic, comes from people using smartphones, on both the app and mobile web. That bringing together means that, be more targeted, in understanding your journey, and people are, last on to time, speed is much more important, people expect things to be right there when they need it, relevance is much more important to people, so we need to bring all those things together to offer a much more targeted experience, and a much more real-time experience. People expect you to have understood what they did milliseconds ago, and respond to that. The only way you can do that is using data science and algorithms. You balance out on a business operation side, just how do you scale? The analogy I use with, say, anomaly detection, which is a crucial feature for enterprises. Used to have a large business intelligence, lots of reports, pages of paper, now people have things like Tablo, Power BI, those are great and you need those to start with, but really as a business leader, you want to know, "Tell me what's broken, tell me what's changed, "because if it's changed something caused the change, "tell me why it's slowly moving, and most importantly, "tell me where the opportunity is." And that transforms the conversation where algorithms can really surface that to users, and it's about organic intelligence, it's not about artificial intelligence, it's about how would you bring together the people, and the advance in technology to really do a great job for customers. >> David: Well, you mentioned AI, you made a big bold claim about AI, I'm going to ask George to weigh in on this in just a moment, you said AI was going to be the next big thing in the travel industry, can you explain? >> One of the next big things, I think. Yeah, I think it's already happening, in fact, our chairman, Mr. Diller made that statement very recently, also backed up by both the CEO and the brand president, where it's... If you think about 20 years ago, one of the things both Expedia and Hotels.com, and travel online space did, were democratize price information, and made it transparent to users. So previously, the power was with the travel agents, that power moved to the user, they had the information. And that's evolved over time, and what we feel with artificial intelligence, particularly organic intelligence, enablers like mobile, messaging and having conversations, have a machine learning how to make this happen, that you can turn the screen around and actually empower users always with the second revolution. They actually have the advice, and the benefits you had a number of years ago from travel agents: A, they had the price transparency, they have the other part now, which is the content, advice, and what's the most relevant to help them. And you can listen to what they're saying to you, as a customer, and actually we can now replay the perfect information back to them, or increasingly perfect as time goes on. (crosstalk) >> That is fascinating, 'cause in the way you broke that out, with--it wasn't actually only travel, but over the last couple decades, price transparency became an issue for many industries, but what you're saying now is, by giving the content to surprise and delight the customer, as long as you're collecting the data breadcrumbs to help you do that, you're not giving up control, you're actually creating stickiness. >> Matt: We're empowering, is the language I use. And if you empower the user, the more likely to come back to use your service in the future, and that's really what we want, we want happy customers. >> George: Tell us a little bit, at the risk of dropping a little in the wait, tell us a little bit about how you empower, in other words, how do you know what type of content to serve up, and how do you measure how they engage with it? >> It's a great question, and I think it's quite embryonic, part of the world right now. I don't think anybody's-- have we made some great developments? I said it was a long journey we have, but it's a lot about how do you, and this is true across data science machine learning, great data science is fundamental to having great feedback loops. So, there's lots of different techniques and tactics around how you might discover those feedback loops, and customers demand that you use their data to help them. So, we need to get faster, and streaming is one way, that's becoming feasible, and the advances in streaming and it's great Databricks are working on that, but the advances in streaming allows it to feed that loop, to take that much--those real-time signals, as well as previous signals, to really help figure out what you're trying to do today, what content-- interesting thing is, Netflix and Amazon were some pioneers in this space, where if you use Netflix service, often you go, "How the hell did they know "this video was going to be right for me?" And, some of the comments, and you can say, well, what they're actually doing is they're looking at microsegments, so previously everyone talked about custom segments as these very large groups, and they have their place, but increasing machine learning allows you to build microsegments. What I can start to do is actually discover from the behavior of others, things you likely-- very relevant things that you're going to be very interested in, and actually help inspire you and discover things you didn't even know existed. And by filling that gap and using those microsegments as well as put truly personal, personalization, I can bring that together to offer you a much more enhanced service. >> George: And so, help make that concrete in terms of, what would I as a potential--I want to plan a vacation for the summer, I have my five and a half inch or, five-seven iPhone, and that's my primary device. And in banking, it's moved from tying everything to the checking account, to tying every interaction to your mobile device. So what would you show me on my mobile device, that would get me really engaged about going to some location? >> So I think a lot of it is about where you are in that journey. So, you think, there's so many different routes customers can take, through that buying decision. And depends on the trip type, whether it's a leisure trip, seeing your family and friends, how much knowledge you may have about them, have you been there before? We look for all those signals, to try and help inspire. So a great example might be, if you stayed in a hotel on our site before, and you liked that hotel, and you come back and do a search again, we try and make it easy to continue by putting that hotel at the top. Trying to make it easy to task-complete. We have a trip planner capability you'll see on the home screen, which allows you to record and play back some of your previous searches, so you can quickly see and compare where you've been, and what's interesting for you. But on top of that, we can then use the signals, and increasingly, we have a very advanced filter list, and that's a key, and we're looking in stuff, how we do conversations in the chatbox, is this sort of future, how to have a conversation to say, "Hey, here's a list of hotels, which we used a mix of your, "the types of preferences understood about you, "and the wider thing, where you are in the world, "what's going on, what time of day." We take hundreds of different signals to try and figure out what the right list is for you, and from that list, the great thing is most people interact with that list and give us more signals, exactly what you wanted. We can hone and hone and hone, and repeat, 'cause I said at the start, for example, those majority of customers will do multiple searches. They want to understand what the market is, they may not be interested in one particular place, they may have a sweeter place there instead. Even now, where we've moved further up the funnel, investing behind, how can you figure out what destination you're interested in? So you may not even know what destination you're interested in, or there might be other destinations that you didn't know--with a very relevant for your use case, particularly if you're going on vacation, we can help inspire you to find that hidden gem, that hidden great prize, you may not even know it existed. Being the much better job, but to show you how busy the market is, to how fast you should be looking to book there, if it's a very compressed, busy market, you need to get in there quick to lock your price in, and we're now providing that information to help you make a better decision. And we can mine all that data, to empower you to make smart decisions with smart data. >> I want to clarify something I saw in your demonstration this morning, you were talking about detecting the differences between photos and user-generated content, so do you have users actually posting their own photos of the hotel, right next to the photoshopped pictures of the hotel? >> Matt: We do, yeah. >> David: What are the ramifications of that? >> So it's an interesting advancement we've made, so we've... In the last of the year, we now offer and asking users to submit their photos, to help other users. I think one of the crucial things is about how to be authentic. Over the years, we've had tens of millions of testimonial reviews, text reviews, and we can see they're really, crucially important to users, and their buying decisions. >> David: It scares the hotel owners to death though, doesn't it? >> Matt: Well, I think it does, but I think the testimony of the customer, could be one of the key things we call them, as we have verified reviews, so to leave a review on our site, you've had to stay in that hotel. We think that's a crucial step in really helping to say, "These are your customers." In recent times, we've taken that product further, to now when you actually arrive at the hotel within a few hours, We'll ask you what your first impressions were. We would ask if you want to share that with the hotel owner. To get the hotel owner a chance to actually rectify any early challenges, so you can have a great stay. And one of the crucial things we have is that, what's really, really important, is that users and customers have a great stay, that reflects on our Net Promoter score, and their view of us, and we need to fill that cycle and make sure we have happy users. So that real-time review is super crucial, in basing how can hotels--if they want happy users and customers as well, it helps them to cut a course correct, if there's an issue, and we can step in as well to help the user if it's a really deep issue. And then with the photos, the key to think is how to navigate and understand what the photo is, so the user helps us by tagging that, which is great, but how we-- >> David: Possibly mistagging it. >> Possibly mistagging it on occasion, that's something we've, we've built in some skill as you've heard, on how to tackle that, but the crucial thing is how to bring these together, if you're on a mobile device, you've got to scan through each photo, and in places around the world have limited bandwidth, a limited time to go through them, so what we're now working on is how to assess the quality of those photos, to try and make sure we authentically--what we want to do, is get the customer the most lively experience they will have. As I said before, we're on the customer's kind of focus, we want to make sure they get the best photos, the most realistic of what's going to happen, and doing the most diverse. You want to see three photos, exactly the same, and we're working on the moment, you can swipe left and swipe right, we're working on how that display evolves over time, but it's exciting. >> David: Very exciting, fascinating stuff. Sorry that we're up against a hard break, coming here in just a moment, but I wanted to give you just 30 seconds to kind of sum up, maybe the next big technical challenge you're looking at that involves Spark, and we'll close with that. >> Cool, it's a great question. I think I talked a little about that in the keynote, totally caught the kind of out challenge. How to scale a mountain, which has been-- there's been great advance on how to stream data into platforms, Spark is a core part of that, and the platforms that we've been building, both internally, and partnering with Databricks and using their platform, has really given us a large boost going forwards, but how you turn those algorithms and that competitive algorithmic advantage, into a live production environment, whether it's marketplaces, Adtech marketplaces or websites, or in call centers, or in social media, wherever the platform needs to go, that's a hard problem right now. Or, I think it's too hard a problem right now. And I'd love to see--and we're going to invest behind that, a transformation, that hopefully this time next year, that is no longer a problem, and is actually an asset. >> David: Well I hope I'm not Captain Obvious to say, I know you're up to the challenge. Thank you so much, Matt Fryer, we appreciate you being on the show, thank you for sharing what's going on at Hotels.com. And thank you all for watching The Cube, we'll be back in a few moments with our next guest, here at Spark Summit 2017. (electronic music) (wind blowing)

>> Announcer: Live from San Francisco, it's theCUBE covering Spark Summit 2017. Brought to you by Databricks. >> Getting close to the end of the day here at Spark Summit, but we saved the best for last I think. I'm pretty sure about that. I'm David Goad, your host here on theCUBE and we now have data scientists from Riot Games, yes, Riot Games. His name is Wesley Kerr. Wesley, thanks for joining us. >> Thanks for having me. >> What's the best money-making game at Riot Games? >> Well we only have one game. We're known for League of Legends. It came out in 2009, it has been growing and well-received by our fans since then. >> And what's your role there? It says data scientist, but what do you really do? >> So we build models to look at things like in game behavior. We build models to actually help players engage with our store and buy our content. We look at different ways we can, just, improve our player experience. >> Alright well let's talk about a little more under the hood, here. How are you deploying Spark in the game? >> So we relied on Databricks for all of our deployment. We do many different clusters. We have about 14 data scientists that work with us, each one is sort of able to manage their own clusters: spin 'em up, tear 'em down, find their data that way and work with it through Databricks. >> So what else will you cover? You had a keynote session this morning, right? >> Yep. >> Give a recap for theCUBE audience of what you talked about. >> So we talked about our efforts in player behavior where we build models and deploy models that are watching chat between players so we evaluate whether or not players are being unsportsmanlike and come up with ways to, sort of, help them curb that behavior and be more sportsmanlike in our game. >> Oh wow, unsportsmanlike. How do you define that? It's if people are being abusive? >> So what we saw was there are about one or two percent of our games that is some form of serious abuse and that comes in term of hate speech, racism, sexism, things that have no place in the game and so we want them to realize that that language is bad and they shouldn't be using it. >> It's all key word driven or are there other behaviors or things that can indicate? >> So right now it's purely based on things said in chat, but we're currently investigating other, sort of, other ways of measuring that behavior and how it occurs in game and how it could influence what people are saying. >> Maybe like tweets coming from The White House? (laughing) >> Okay, so George. >> We should be able to measure that as well. >> So how about those warriors? (laughing) >> No, George did you want to talk a little bit more >> Sure. >> David: about the technical achievements here? When you look at like trying to measure engagement and sort of maybe it sounds like converting high engagement to store purchases, tell us a little more maybe how that works. >> So we look at, we want. Our game is completely free to play. Players can download, play it all the way through and we really try to create a very engaging game that they want to come back and they want to play and then everything they can buy in the store is actually just cosmetics. So we really hope to build content that our players love and are happy to spend money on. As far as... We just really want engagement to be from around players coming back and playing and having a good time and it's less about how to get that high engagement conversion into monetization as we've seen that players who are happy and loving the game are happy to spend their money. >> So tell us more about how you build some of these models like, you know, turning it into not turning it into Spark code, but how do you analyze it and, sort of, what's the database mechanism for, you know, 'cause the storage layer in Spark, you know, is just like the file system? >> Sure, yeah absolutely. So we are a world-wide game. We're played by over 100 million players around the world >> David: Wow. >> And so that data comes flowing in from all around the world into our centralized data warehouse. That data warehouse has gameplay data so we know how you did in game. It also has time series events, so things that occurred in each game. And our game is really session based so players can come play for an hour, that's one game, and then they leave and come back and play again. And so what we're able to do is then, sort of, look at those models and how they did. And I'll give you an example around our content recommendations. So we look at the champions that you've been playing recently to predict which champions you are likely to play next. And that we can actually just query the database, start building our collaborative filtering models on top of it, and then recommend champions that you may not play now, you may be interested in playing, or we may decide to give you a special discount on a champion if we think it'll resonate well with you. >> And in this case, just to be clear, the champions you're talking about are other players, not models? >> It's actually the in-game avatar. So it's the champion that they play. So we have 130 unique champions and each game you choose which champion you want to play and so then that plays out for like. It's much more like a sport than it is like a game. So it's five v five, online competitive. So there are different objectives on the map. You work with your team to complete those objectives and beat the other team. So we like to think of it like basketball, but with magic and in a virtual world. >> And the teams stay together? Or are they constantly recombining? >> They can disband, yeah. Your next game may find nine other people. If you're playing with your friends then you can just keep queuing up with them as well. So the champions that they control there happen to be who you're playing in that game. >> And when you are trying to anticipate champions that someone might play in the future, what are the variables that you're trying to guess and how long did it take you to build those models? >> Yeah, it's a good question. Right now we are able to sort of leverage the power of our user, our players, so we have 100 million. And so what we do and we have in our game there are roles so, for instance, like there's a center in basketball, we have a bot lane. So we have bottom lane support and bottom lane ADC. So a support character is there to make sure that your ADC is able to defeat the other team. And if you play a lot of support, odds are there are other players in the world who play a lot of support too so we find similar players. We find that if they engaged on the same sorts of champions that you play. For instance, I'm a Leona main and so I play her a lot. And if I were to look at what other people played in addition to Leona it could be things like Braum and so then we would recommend Braum as a champion that you should try out that you've maybe not played yet. >> David: Okay. >> So and then what's the data warehouse that you guys use for the ultimate repository of all this? >> All the data flows into a Hive data warehouse, stored in S3. We have two different ways of interacting with it. One, we can run queries against Hive. It tends to be a bit slower for our use cases. And then our data scientists tend to access that all that data through Databricks and Spark. And it runs much quicker for our use cases. >> Do you take what's in S3 and put it into a parquet format to accelerate? >> Sometimes, so we do some of those rewrites. We do a lot of our secondary ETLs where we're just joining across multiple tables and writing back out. We'll optimize those for our Spark use cases and there's writing back, sort of, read from S3, do some transformations, write back to S3. >> And how latency-sensitive is this? Are you guys trying to make decisions as the player moves along in his level or? >> So historically we've been batch. We do- our recommendations are updated weekly so we haven't needed a much higher cadence. But we're moving to a point where I want to see us be able to actually make recommendations on the client and do it immediately after you've finished a game with, say, Leona, here's an offer for Braum. Go check it out, give it a try in your next game. >> So Wesley what would you like to see developed that hasn't been developed yet that would really help in your business specifically? >> So one thing that's really exciting for gaming right now is procedural generation and artificial intelligence. So here there are a lot of opportunities, you've seen some collaborations between Deep Mind and Blizzard where they're learning to play Starcraft. For me, I think there's a similar world where we have a game that has different sorts of mechanics. So we have a large social piece to our game and teamwork is required. And so understanding how we can leverage that and help influence the future of artificial intelligence is something that I want to see us be able to do. >> Did you talk with anybody here at the Spark Summit about that? >> Anyone who would listen. (laughing) So we chatted some with the teams up at Blizzard and Twitch about some of the things they're doing for natural language as well. >> Alright so what was the most useful conversation you had here at the summit? >> The most useful one that I had, I think, was with the Databricks team. So at the end of my keynote, It was kind of serendipitous, I was talking about some work we had done with deep learning and sort of doing hyper parameter searches over our worker nodes, so actually being able to quickly try out many different models. And in the announcement that morning before my keynote, Tim talked about how they actually have deep learning pipelines now and it was based on a conversation we had had so I was very excited to see it come to fruition and now is open source and we can leverage it. >> Awesome, well, we're up against a hard break here. >> Wesley: Okay. >> We're almost at the end of the day. Wesley, it's been a riot talking to you. We really appreciate it and thank you for coming on the show and sharing your knowledge. >> Wesley: You bet, thanks for having me. >> Alright and that's it, we're going to wrap it up today. We have a wrap-up coming up, as a matter of fact, in just a few minutes. My name is David Goad. You're watching theCUBE at Spark Summit. (upbeat music)

>> Announcer: Live from San Francisco, it's theCUBE. covering Spark Summit 2017, brought to you by Databricks. (techno music) >> Welcome to theCUBE, at Spark Summit here at San Francisco, at the Moscone Center West, and we're going to be competing with all the excitement happening behind us. They're going to be going off with raffles, and I don't know what all. But we'll just have to talk above them, right? >> Clarke: Well at least we didn't get to win. >> Our next guest here on the show is Clarke Patterson from Confluent. You're the Senior Director of Product Marketing, is that correct? >> Yeah, you got it. >> All right, well it's exciting -- >> Clarke: Pleasure to be here >> To have you on the show. >> Clarke: It's my first time here. >> David: First time on theCUBE? >> I feel like one of those radio people, first time caller, here I am. Yup, first time on theCUBE. >> Well, long time listener too, I hope. >> Clarke: Yes, I am. >> And so, have you announced anything new that you want to talk about from Confluent? >> Yeah, I mean not particularly at this show per se, but most recently, we've done a lot of stuff to enable customers to adopt Confluent in the Cloud. So we came up with a Confluent Cloud offering, which is a managed service of our Confluent platform a couple weeks ago, at our event around Kafka. So we're really excited about that. It really fits that need where Cloud First or operation-starved organizations are really wanting to do things with storing platforms based on Kafka, but they just don't have the means to make it happen. And so, we're now standing this up as a managed service center that allows them to get their hands on this great set of capabilities with us as the back stop to do things with it. >> And you said, Kafka is not just a publish and subscribe engine, right? >> Yeah, I'm glad that you asked that. So, that one of the big misconceptions, I think, of Kafka. You know, it's made its way into a lot of organizations from the early use case of publish and subscribe for data. But, over the last 12 to 18 months, in particular, there's been a lot of interesting advancements. Two things in particular: One is the ability to connect, which is called a Connect API in Kafka. And it essentially simplifies how you integrate large amounts of producers and consumers of data as information flows through. So, a modernization of ETL, if you will. The second thing is stream processing. So there's a Kafka streams API that's built-in now as well that allows you to do the lightweight transformations of data as it flows from point A to point B, and you could publish out new topics if you need to manipulate things. And it expands the overall capabilities of what Kafka can do. >> Okay, and I'm going to ask George here to dive in, if you could. >> And I was just going to ask you. >> David: I can feel it. (laughing) >> So, this is interesting. But if we want to frame this in terms of what people understand from, I don't want to say prehistoric eras, but earlier approaches to similar problems. So, let's say, in days gone by, you had an ETL solution. >> Clarke: Yup. >> So now, let's put Connect together with stream processing, and how does that change the whole architecture of integrating your systems? >> Yeah, I mean I think the easiest way to think about this is if you think about some of the different market segments that have existed over the last 10 to 20 years. So data integration was all about how do I get a lot of different systems to integrate a bunch of data and transform it in some manner, and ship it off to some other place in my business. And it was really good at building these end-to-end workflows, moving big quantities of data. But it was generally kind of batch-oriented. And so we've been fixated on, how do we make this process faster? To some degree, and the other segment is application integration which said, hey, you know when I want applications to talk to one another, it doesn't have the scale of information exchange, but it needs to happen a whole lot faster. So these real-time integration systems, ESBs, and things like that came along and it was able to serve that particular need. But as we move forward into this world that we're in now, where there's just all sorts of information, companies want to become advanced-centric. You need to be able to get the best of both of those worlds. And this is really where Kafka is starting to sit. It's saying, hey let's take massive amounts of data producers that need to connect to massive amounts of data consumers, be able to ship a super-granular level of information, transform it as you need, and do that in real-time so that everything can get served out very, very fast. >> But now that you, I mean that's a wonderful and kind of pithy kind of way to distill it. But now that we have this new way of thinking of app integration, data integration, best of both worlds, that has sort of second order consequences in terms of how we build applications and connect them. So what does that look like? What do applications look like in the old world and now what enables them to be sort of re-factored? Or for new apps, how do you build them differently? >> Yeah, I mean we see a lot of people that are going into microservices oriented architecture. So moving away from one big monolithic app that takes this inordinate amount of effort to change in some capacity. And quite frankly, it happens very, very slow. And so they look to microservices to be able to split those up into very small, functional components that they can integrate a whole lot faster, decouple engineering teams so we're not dependent on one another, and just make things happen a whole lot quicker than we could before. But obviously when you do that, you need something that can connect all those pieces, and Kafka's a great thing to sit in there as a way to exchange state across all these things. So that's a massive use case for us and for Kafka specifically in terms of what we're seeing people do. >> You've said something in there at the end that I want to key off, which is, "To exchange state." So in the old world, we used a massive shared database to share state for a monolithic app or sometimes between monolithic apps. So what sort of state-of-the-art way that that's done now with microservices, if there's more than one, how does that work? >> Yeah, I mean so this is kind of rooted in the way we do stream processing. So there's this concept of topics, which effectively could align to individual microservices. And you're able to make sure that the most recent state of any particular one is stored in the central repository of Kafka. But then given that we take an API approach to stream processing, it's easy to embed those types of capabilities in any of the end-points. And so some of the activity can happen on that particular front, then it all gets synchronized down into the centralized hub. >> Okay, let me unpack that a little bit. Because you take an API approach, that means that if you're manipulating a topic, you're processing a microservice and that has state in it? Is that the right way to think about it? >> I think that's the easiest way to think about it, yeah. >> Okay. So where are we? Is this a 10 year migration, or is it a, some certain class of apps will lend themselves well to microservices, legacy apps will stay monolithic, and some new apps, some new Greenfield apps, will still be database-centric? How do you, or how should customers think about that mix? >> Yeah that's a great question. I don't know that I have the answer to it. The best gauge I can have is just the amount of interest and conversations that we have on this particular topic. I will say that from one of the topics that we do engage with, it's easily one of the most popular that people are interested in. So if that's a data point, it's definitely a lot of interested people trying to figure out how to do this stuff very, very fast. >> How to do the microservices? >> Yeah and I think if you look at some of the more notable tech companies of late, they're architected this way from the start. And so everyone's kind of looking at the Netflix of the world, and the Ubers of the world saying, I want to be like those guys, how do I do that? And it's driving them down this path. So competitive pressure, I think, will help force people's hands. The more that your competitors are getting in front of you and are able to deliver a better customer experience through some sort of mobile app or something like that, then it's going to force people to have to make these changes quicker. But how long that takes it'll be interesting to see. >> Great! Great stuff. Switch gears just a little bit. Talk about maybe why you're using Databricks and what some of the key value you've gotten out of that. >> Yeah, so I wouldn't say that we're using Databricks per se, but we integrate directly with Spark. So if you look at a lot of the use cases that people use Spark for, they need to obviously get data to where it is. And some of the principles that I said before about Kafka generally, it's a very flexible, very dynamic mechanism for taking lots of sources of information, culling all that down into one centralized place and then distributing it to places such as Spark. So we see a lot of people using the technologies together to get the data from point A to point B, do some transformation as they so need, and then obviously do some amazing computing horsepower and whatnot in Spark itself. >> David: All right. >> I'm processing this, and it's tough because you can go in so many different directions, especially like the question about Spark. I guess, give us some of the scenarios where Spark would fit. Would it be like doing microservices that require more advanced analytics, and then they feed other topics, or feed consumers? And then where might you stick with a shared database that a couple services might communicate with, rather than maintaining the state within the microservice? >> I think, let me see if I can kind of unpack that myself a little bit. >> George: I know it was packed pretty hard. (laughing) >> Got a lot packed in there. When folks want to do things like, I guess when you think about it like an overall business process. If you think about something like an order to cash business process these days, it has a whole bunch of different systems that hang off it. It's got your order processing. You've got an inventory management. Maybe you've got some real-time pricing. You've got some shipments. Things, like that all just kind of hang off of the flow of data across there. Now with any given system that you use for addressing any answers to each of those problems could be vastly different. It could be Spark. It could be a relational database. It could be a whole bunch of different things. Where the centralization of data comes in for us is to be able to just kind of make sure that all those components can be communicating with each other based on the last thing that happened within each of them individually. And so their ability to embed transformation, data transformations and data processing in themselves and then publish back out any change that they had into the shared cluster subsequently makes that state available to everybody else. So that if necessary, they can react to it. So in a lot of ways, we're kind of agnostic to the type of processing that happens on the end-points. It's more just the free movement of all the data to all those things. And then if they have any relevant updates that need to make it back to any of the other components hanging on that process flow, they should have the ability to publish that back down it. >> And so one thing that Jay Kreps, Founder and CEO, talks about is that Kafka may ultimately, or in his language, will ultimately grow into something that rivals the relational database. Tell us what that world would look like. >> It would be controversial (laughing). >> George: That's okay. >> You want me to be the bad guy? So it's interesting because we did Kafka Summit about a month ago, and there's a lot of people, a lot of companies I should say, that are actually using and calling Kafka an enterprise data hub, a central hub for data, a data distribution network. And they are literally storing all sorts (raffle announcements beginning on loudspeaker) of different links of data. So one interesting example was the New York Times. So they used Kafka and literally stored every piece of content that has ever been generated at that publisher since the beginning of time in Kafka. So all the way back to 1851, they've obviously digitized everything. And it sits in there, and then they disposition that back out to various forms of the business. So that's -- >> They replay it, they pull it. They replay and pull, wow, okay. >> So that has some very interesting implications. So you can replay data. If you run some analytics on something and you didn't get the result that you wanted, and you wanted to redo it, it makes it really easy and really fast to be able to do that. If you want to bring on a new system that has some new functionality, you can do that really quickly because you have the full pedigree of everything that sits in there. And then imagine this world where you could actually start to ask questions on it directly. That's where it starts to get very, very profound, and it will be interesting to see where that goes. >> Two things then: First, it sounds, like a database takes updates, so you don't have a perfect historical record. You have a snapshot of current values. Like whereas in a log, like Kafka, or log-structured data structure you have every event that ever happened. >> Clarke: Correct. >> Now, what's the impact on performance if you want to pull, you know -- >> Clarke: That much data? >> Yeah. >> Yeah, I mean so it all comes down to managing the environment on which you run it. So obviously the more data you're going to store in here, and the more type of things you're going to try to connect to it, you're going to have to take that into account. >> And you mentioned just a moment ago about directly asking about the data contained in the hub, in the data hub. >> Clarke: Correct. >> How would that work? >> Not quite sure today, to be honest with you. And I think this is where that question, I think, is a pretty provocative one. Like what does it mean to have this entire view of all granular event streams, not in some aggregated form over time? I think the key will be some mechanism to come onto an environment like this to make it more consumable for more business types users. And that's probably one of the areas we'll want to watch to see how that's (background noise drowns out speaker). >> Okay, only one unanswered question. But you answered all the other ones really well. So we're going to wrap it up here. We're up against a loud break right now. I want to think Clarke Patterson from Confluent for joining us. Thank you so much for being on the show. >> Clarke: Thank you for having me. >> Appreciate it so much. And thank you for watching theCUBE. We'll be back after the raffle in just a few minutes. We have one more guest. Stay with us, thank you. (techno music)

>> Narrator: Live from San Francisco, it's the Cube. Covering Sparks Summit 2017. Brought to you by Databricks. (upbeat music) >> Welcome back to the Cube, day two at Sparks Summit. It's very exciting. I can't wait to talk to this gentleman. We have the CEO from Databricks, Ali Ghodsi, joining us. Ali, welcome to the show. >> Thank you so much. >> David: Well we sat here and watched the keynote this morning with Databricks and you delivered. Some big announcements. Before we get into some of that, I want to ask you, it's been about a year and a half since you transitioned from VP Products and Engineering into a CEO role. What's the most fun part of that and maybe what's the toughest part? >> Oh, I see. That's a good question and that's a tough question too. Most fun part is... You know, you touch many more facets of the business. So in engineering, it's all the tech and you're dealing only with engineers, mostly. Customers are one hop away, there's a product management layer between you and the customers. So you're very inwards focused. As a CEO you're dealing with marketing, finance, sales, these different functions. And then, externally with media, with stakeholders, a lot of customer calls. There's many, many more facets of the business that you're seeing. And it also gives you a preview and it also gives you a perspective that you couldn't have before. You see how the pieces fit together so you actually can have a better perspective and see further out than you could before. Before, I was more in my own pick situation where I was seeing sort of just the things relating to engineering so that's the best part. >> You're obviously working close with customers. You introduced a few customers this morning up on stage. But after the keynote, did you hear any reactions from people? What are they saying? >> Yes the keynote was recently so on my way here I've had multiple people sort of... A couple people that high-fived just before I got up on stage here. On several softwaring, people are really excited about that. Less devops, less configuration, let them focus on the innovation, they want that. So that's something that's celebrated. Yesterday-- >> Recap that real quickly for our audience here, what the server-less operating is. >> Absolutely, so it's very simple. We want lots of data scientists to be able to do machine learning without have to worry about the infrastructure underneath it. So we have something called server-less pools and server-less pools you can just have lots of data scientists use it. Under the hood, this pool of resources shrinks and expands automatically. It adds storage, if needed. And you don't have to worry about the configuration of it. And it also makes sure that it's isolating the different data scientists. So if one data scientist happened to run something that takes much more resources, it won't effect the other data scientists that are sharing that. So the short story of it is you cut costs significantly, you can now have 3000 people share the same resources and it enables them to move faster because they don't have to worry about all the devops that they otherwise have to do. >> George, is that a really big deal? >> Well we know whenever there's infrastructure that gets between a developer, data science, and their outcomes, that's friction. I'd be curious to say let's put that into a bigger perspective, which is if you go back several years, what were the class of apps that Spark was being used for, and in conjunction with what other technologies. Then bring us forward to today and then maybe look out three years. >> Ali: Yeah, that's a great question. So from the very beginning, data is key for any of these predictive analytics that we are doing. So that was always a key thing. But back then we saw more Hadoop data lakes. There more data lakes, data reservoirs, data marks that people were building out. We saw also a lot of traditional data warehousing. These days, we see more and more things moving to cloud. The Hadoop data lake received, often times at enterprises, being transformed into a cloud blob storage. That's cheaper, it's dual-up replicated, it's on many continents. That's something that we've seen happen. And we work across any of these, frankly. We, from the very beginning, Spark, one of its strengths is it integrates really well wherever your data is. And there's a huge community of developers around it, over 1000 people now that have contributed to it. Many of these people are in other organizations, they're employed by other companies and their job is to make sure that Databricks or Spark works really, really well with, say, Cassandra or with S3. That's a shift that we're seeing. In terms of applications people are building it's moving more into production. Four years ago much more of it was interactive exploratory. Now we're seeing production use cases. The fraud analytics use case that I mentioned, that's running continuously and the requirements there are different. You can't go down for ten minutes on a Saturday morning at 4 a.m. when you're doing credit card fraud because that's a lot of fraud and that affects the business of, say, Capital One. So that's much more crucial for them. >> So what would be the surrounding infrastructure and applications to make that whole solution work? Would you plug into a traditional system of record at the sales order entry kind of process point? Are you working off sort of semi-real-time or near real-time data? And did you train the models on the data lake? How did the pieces fit together? >> Unfortunately the answers depends on the particular architecture that the customer has. Every enterprise is slightly different. But it's not uncommon that the data is coming in. They're using Spark structured streaming in Databricks to get it into S3, so that's one piece of the puzzle. Then when it ends up there, from then on it funnels out to many different use cases. It could be a data warehousing use case, where they're just using interactive sequel on it. So that's the traditional interactive use case, but it could be a real-time use case, where it's actually taking those data that it's processed and it's detecting anomalies and putting triggers in other systems and then those systems downstream will react to those triggers for anomalies. But it could also be that it's periodically training models and storing the models somewhere. Often times it might be in a Cassandra, or in a Redis, or something of that sort. It will store the model there and then some web application can then take it from there, do point queries to it and say okay, I have a particular user that came in here George now, quickly look up what is his feature vector, figure out what the product recommendations we should show to this person and then it takes it from there. >> So in those cases, Cassandra or Redis, they're playing the serving layer. But generating the prediction model is coming from you and they're just doing the inferencing, the prediction itself. So if you look out several years, without asking you the roadmap, which you can feel free to answer, how do you see that scope of apps expanding or the share of an existing app like that? >> Yeah, I think two interesting trends that I believe in, I'll be foolish enough to make predictions. One is that I think that data warehousing, as we know it today, will continue to exist. However, it will be transformed and all the data warehousing solutions that we have today will add predictive capabilities or it will disappear. So let me motivate that. If you have a data warehouse with customer data in it and a fact table, you have all your transactions there, you have all your products there. Today, you can plug in BI tools and on top of that you can see what's my business health today and yesterday. But you can't ask it: tell me about tomorrow. Why not? The data is there, why can I not ask it this customer data, you tell me which of these customers are going to turn, or which one of them should I reach out to because I can possibly upsell these? Why wouldn't I want to do that? I think everyone would want to do that and everyday a warehousing solution in ten years will have these capabilities. Now with Spark sequel you can do that and the announcement yesterday showed you also how you can bake models, machinery models, and export them so a sequel analyst can just act system directly with no machine learning experience. It's just a simple function call and it just works. So that's one prediction I'll make. The second prediction I'll make is that we're going to see lots of revolutions in different industries, beyond the traditional 'get people to click on ads' and understand social behavior. We're going to go beyond that. So for those use cases it will be closer to the things I mentioned like Shell and what you need to do there is involve these domain experts. The domain experts will come in, the doctors, or the machine specialists, you have to involve them in the loop. And they'll be able to transform, maybe much less exotic applications, it's not the super high-tech Silicon Valley stuff, but it's nevertheless extremely important to every enterprise, to every protocol, on the planet. That's, I think, the exciting part of where predictions will go in the next decade or two. >> If I were to try and pick out the most man-bytes dug kind of observation in there, you know, it's supposed to be the unexpected thing, I would say where you said all data warehouses are going to become predictive services. Because what we've been hearing, it's sort of the other side of that coin which is all the operational databases will get all the predictive capabilities. But you said something very different. I guess my question is are you seeing the advanced analytics going to the data warehouse because the repository of data is going to be bigger there and so you can either build better models or because it's not burdened with transaction SLAs that you can serve up predictions quicker? >> The data warehousing has been about basic statistics. It's been a sequel that the language that is used is to get descriptive statistics. Tables with averages and medians, that's statistics. Why wouldn't you want to have advanced statistics which now does predictions on it. It just so happens that sequel is not the right interface for that. So it's going to be very natural that people who are already asking statistical questions for the last 30 years from their customer data, these massive throes of data that they have stored. Why wouldn't they want to also say, 'okay now give me more advanced statistics?' I'm not an expert on advanced statistics but you the system. Tell me what I should watch out for. Which of these customers do I talk to? Which of the products are in trouble? Which of the products are not, or which parts of my business are not doing well now? Predict the future for me. >> George: When you're doing that though, you're now doing it on data that has a fair amount of latency built into it. Because that's how it got into the data warehouse. Where if it's in the operational database, it's really low latency, typically low latency stuff. Where and why do you see that distinction? >> I do think also that we'll see more and more real-time engines take over. If you do things in real-time you can do it for a fraction of the cost. So we'll also see those capabilities come in. So you don't have to... Your question is, why would you want to once a week batch everything into a central warehouse and I agree with that. It will be streaming in live and then you can on that, do predictions, you can do basic analytics. I think basically the lines will blur between all these technologies that we're seeing. In some sense, Spark actually was the precursor to all that. So Spark already was unifying machine learning, sequel, ETL, real-time, and you're going to see that everywhere appear. >> You mentioned Shell as an example, one of your customers, you also had HP, Capital One, and you developed this unified analytics platform, that's solving some of their common problems. Now that you're in the mood to make predictions, what do you think are going to be the most compelling use cases or industries where you're going to see Databricks going in the future? >> That's a hard one. Right now, I think healthcare. There's a lot of data sets, there's a lot of gene sequencing data. They want to be able to use machine learning. In fact, I think those industries being transformed slowly from using classical statistics into machine learning. We've actually helped some of these companies do that. We've set up workshops and they've gotten people trained. And now they're hiring machine learning experts that are coming in. So that's one I think in the healthcare industry, whether it's for drug-testing, clinical-trials, even diagnosis, that's a big one, I do think industrial IT. These are big companies with lots of equipment, they have tons of sensor data, massive data sets. There's a lot of predictions that they can do on that. So that's a second one I would say. Financial industry, they've always been about predictions, so it makes a lot of sense that they continue doing that. Those are the biggest ones for Databricks. But I think now also as slowly, other verticals are moving into the cloud. We'll see more of other use cases as well. But those are the biggest ones I see right now. It's hard to say where it will be ten years from now, or 15. Things are going so fast that it's hard to even predict six months. >> David: Do you believe IOT is going to be a big business driver? >> Yes, absolutely. >> I want to circle back where you said that we've got different types of databases but we're going to unify the capabilities. Without saying, it's not like one wins, one loses. >> Ali: Yes, I didn't want to do that. >> So describe maybe the characteristics of what a database that compliments Sparks really well might look like. >> That's hard for me to say. The capabilities of Spark, I think, are here to stay. The ability to be able to ETL variety of data that doesn't have structure, so Structured Query Language, SQL, is not fit for it, that is really important and it's going to become more important since data is the new oil, as they said. Well, then it's going to be very important to be able to work with all kinds of data and getting that into the systems. There's more things everyday being created. Devices, IOT, whatever it is that are spewing out this data in different forms and shapes. So being able to work with that variety, that's going to be an important property. So they'll have to do that. That's the ETL portion or the ELT portion. The real-time portion, not having to do this in a batch manner once a week because now time is a competitive advantage. So if I'm one week behind you that means I'm going to lose out. So doing that in real-time, or near human-time or human real-time, that's going to be really important. So that's going to come as well, I think, and people will demand that. That's going to be a competitive advantage. Wherever you can add that secret sauce it's going to add value to the customers. And then finally the predictive stuff, adding the predictive stuff. But I think people will want to continue to also do all the old stuff they've been doing. I don't think that's going to go away. Those bring value to customers, they want to do all those traditional use cases as well. >> So what about now where customers expect to have some, not clear how much, un-Primmed application platform like Spark. Some in the cloud that now that you've totally reordered the TCO equation. But then also at the edge for IOT-type use cases, do you have to slim down Spark to work at the edge? If you have server-less working in the cloud, does that mean you have to change the management paradigm on Prim. What does that mix look like? How does someone, you know how does a Fortune 200 company, get their arms around that? >> Ali: Yeah, this is a surprising thing, most surprising thing for me in the last year, is how many of those Fortune 200's that I was talking to three years ago and they were saying 'no way, we're not going into the cloud. You don't understand the regulations that we are facing or the amount of data that we have.' Or 'we can do it better,' or 'the security requirements that we have, no one can match that.' To now, those very same companies are saying 'absolutely, we're going.' It's not about if, it's about when. Now I would be hard-pressed to find any enterprise that says 'no, we're not going to go, ever.' And some companies we've even seen go from the cloud to on Prim, and then now back. Because the prices are getting more competitive in the cloud. Because now there's three, at least, major players that are competing and they're well-funded companies. In some sense, you have ad money and office money and retail money being thrown at this problem. Prices are getting competitive. Very soon, most IT folks will realize, there's no way we can do this faster, or better, or more reliable secure ourselves. >> David: We've got just a minute to go here before the break so we're going to kind of wrap it up here. And we got over 3000 people here at Spark Summit so it's the Spark community. I want you to talk to them for a moment. What problems do you want them to work on the most? And what are we going to be talking about a year from now at this table? >> The second one is harder. So I think the Spark community is doing a phenomenal job. I'm not going to tell them what to do. They should continue doing what they are doing already which is integrating Spark in the ecosystem, adding more and more integrations with the greatest technologies that are happening out there. Continue the innovation and we're super happy to have them here. We'll continue it as well, we'll continue to host this event and look forward to also having a Spark Summit in Europe, and also the East Coast soon. >> David: Okay, so I'm not going to ask you to make any more predictions. >> Alright, excellent. >> David: Ali this is great stuffy today. Thank you so much for taking some time and giving us more insight after the keynote this morning. Good luck with the rest of the show. >> Thank you. >> Thanks, Ali. And thank you for watching. That's Ali Ghodsi CEO from Databricks. We are Spark Summit 2017 here, on the Cube. Thanks for watching, stay with us. (upbeat mustic)

>> Announcer: Live from San Francisco, it's theCube, covering Spark Summit 2017, brought to you by Databricks. >> Welcome back to theCube at Spark Summit 2017. I don't know about you, George, I'm having a great time learning from all of our attendees. >> We've been absorbing now for almost two days. >> Yeah, well, and we're about to absorb a little bit more here, too, because the next guest, I looking forward to, I saw his name on the schedule, all right, that's the guy who talks about herding cats, it's John Cavanaugh, Master Architect from HP. John, welcome to the show. >> Great, thanks for being here. >> Well, I did see, I don't know if it's about cats in the Internet, but either cats or self-driving cars, one of the two in analogies. But talk to us about your session. Why did you call it Herding Cats, and is that related to maybe the organization at HP? >> Yeah, there's a lot of organizational dynamics as part of our migration at Spark. HP is a very distributed organization, and it has had a lot of distributed autonomy, so, you know, trying to get centralized activity is often a little challenging. You guys have often heard, you know, I am from the government, I'm here to help. That's often the kind of shields-up response you will get from folks, so we got a lot of dynamics in terms of trying to bring these distributed organizations on board to a new common platform, and a allay many of the fears that they had with making any kind of a change. >> So, are you centered at a specific division? >> So, yes, I'm the print platforms and future technology group. You know, there's two large business segments with HP. There's our personal systems group that produces everything from phones to business PCs to high-end gaming. But I'm in the printing group, and while many people are very familiar with your standard desktop printer, you know, the printers we sell really vary from a very small product we call Sprocket, it fits in your hand, battery-operated, to literally a web press that's bigger than your house and prints at hundreds of feet per minute. So, it's a very wide product line, and it has a lot of data collection. >> David: Do you have 3D printing as well? >> We do have 3D printing as well. That's an emergent area for us. I'm not super familiar with that. I'm mostly on the 2D side, but that's a very exciting space as well. >> So tell me about what kind of projects that you're working on that do require that kind of cross-team or cross-departmental cooperation. >> So, you know, in my talk, I talked about the Wild West Era of Big Data, and that was prior to 2015, and we had a lot of groups that were standing up all kinds of different big data infrastructures. And part of this stems from the fact that we were part of HP at the time, and we could buy servers and racks of servers at cost. Storage was cheap, all these things, so they sprouted up everywhere. And, around 2015, everybody started realizing, oh my God, this is completely fragmented. How do we pull things back together? And that's when a lot of groups started trying to develop platformish types of activities, and that's where we knew we needed to go, but there was even some disagreement from different groups, how do we move forward. So, there's been a lot of good work within HP in terms of creating a virtual community, and Spark really kind of caught on pretty quickly. Many people were really tired of kind of Hadoop. There were a lot of very opinionated models in Hadoop, where Spark opens up a lot more into the data science community. So, that went really well, and we made a big push into AWS for much of our cloud activities, and we really ended up then pretty quickly with Databricks as an enterprise partner for us. >> And so, George, you've done a lot of research. I'm sure you talked to enterprise companies along the way. Is this a common issue with big enterprises? >> Well, for most big data projects they've started, the ones we hear a lot about is there's a mandate from the CIO, we need a big data strategy, and so some of those, in the past, stand up five or 10-node Hadoop cluster and run some sort of pilot and say, this is our strategy. But is sounds like you herded a lot of cats... >> We had dozens of those small Hadoop clusters all around the company. (laughter) >> So, how did you go about converting that energy, that excess energy towards something more harmonized around Databricks? >> Well, a lot of people started recognizing we had a problem, and this really wasn't going to scale, and we really needed to come up with a broader way to share things across the organization. So, the timing was really right, and a lot of people were beginning to understand that. And, you know, we said for us, probably about five different kind of key decisions we ended up making. And part of the whole strategy was to empower the businesses. As I have mentioned, we are a very distributed organization, so, you can't really dictate the businesses. The businesses really need the owners' success. And one of the decisions that was made, it might be kind of controversial for many CIOs, is that we've made a big push on cloud-hosted and business-owned, not IT-owned. And one of the real big reasons for that is we were no longer viewing data and big data as kind of a business-intelligence activity or a standardized reporting activity. We really knew that, to be successful moving forward, is needed to be built into our products and services, and those products and services are managed by the businesses. So, it can't be something that would be tossed off to an IT organization. >> So that the IT organization, then, evolved into being more of an innovative entity versus a reactive or supportive entity for all those different distributing groups. >> Well, in our regard, we've ended up with AWS as part of our activity, and, really, much of our big data activities are driven by the businesses. The connections we have with IT are more related to CRM and product data master sheets and selling in channels and all that information. >> But if you take a bunch of business-led projects and then try and centralize some aspect of them, wouldn't IT typically become the sort of shared infrastructure architecture advisor for that, and then the businesses now have a harmonized platform on which they can build shared data sets? >> Actually, in our case, that's what we did. We had a lot of our businesses that already had significant services hosted in AWS. And those were very much part of the high-data generators. So, it became a very natural evolution to continue with some of our AWS relationships and continue on to Databricks. So, as an organization today, we have three kind of main buckets for our Databricks, but, you know, any business, they can get their accounts. We try and encourage everything to get into a data link, and that's three, and Parquet formats, one of the decisions that was adapted. And then, from there, people can begin to move. You know, you can get notebooks, you can share notebooks, you can look at those things. You know, the beauty of Databricks and AWS is instant on. If I want to play around with something with a half a dozen nodes, it's great. If I need a thousand for a workload, boom, I've got it! I know, kind of others, then, with this cost and the value returned, there's really no need for permissions or coordination with other entities, and that's kind of what we wanted the businesses to have that autonomy to drive their business success. >> But, does there not to be some central value added in the way of, say, data curation through a catalog or something like that? >> Yes, so, this is not necessarily a model where all the businesses are doing all kinds of crazy things. One of the things that we shepherded by one of our CTOs and the other functions, we ended up creating a virtual community within HP. This kind of started off with a lot of "tribal elders" or "tribal leaders." With this virtual community, today we get together every two weeks, and we have presentations and discussions on all things from data science into machine learning, and that's where a lot of this activity around how do we get better at sharing. And this is fostered, kind of splinters off for additional activity. So we have one on data telemetry within our organization. We're trying to standardize more data formats and schemas for those so we can have more broader sharing. So, these things have been occurring more organically as part of a developer enablement kind of moving up rather than more of kind of dictates moving down. >> That's interesting. Potentially, really important, when you say, you're trying to standardize some of the telemetry, what are you instrumenting. Is it just all the infrastructure or is it some of the products that HP makes? >> It's definitely the products and the software. You know, like I said, we manage a huge spectrum of print products, and my apologies if I'm focusing on it, but that is what I know the best. You know, we've actually been doing telemetry and analysis since the late 90s. You know, we wanted to understand use of supplies and usage so we could do our own forecasting, and that's really, really grown over the years. You know, now, we have parts of our services organization management services, where they're offering big data analytics as part of the package, and we provide information about predictive failure of parts. And that's going to be really valuable for some of our business partners that allows them. We have all kinds fancy algorithms that we work on. The customers have specific routes that they go for servicing, and we may be able to tell them, hey, in a certain time period, we think these devices in your field so you can coordinate your route to hit those on an efficient route rather than having to make a single truck roll for one repair, and do that before a customer experiences a problem. So, it's been kind of a great example of different ways that big data can impact the business. >> You know, I think Ali mentioned in the keynote this morning about the example of a customer getting a notification that their ink's going to run out, and the chance that you get to touch that customer and get them to respond and buy, you could make millions of dollar difference, right? Let's talk about some of the business outcomes and the impact that some of your workers have done, and what it means, really, to the business. >> Right now, we're trying to migrate a lot of legacy stuff, and you know, that's kind of boring. (laughs) It's just a lot of work, but there are things that need to happen. But there's really the power of the big data platform has been really great with Databricks. I know, John Landry, one of our CTOs, he's in the personal systems group. He had a great example on some problems they had with batteries and laptops, and, you know, they have a whole bunch of analytics. They've been monitoring batteries, and they found a collection of batteries that experienced very early failure rates. I happen to be able to narrow it down to specific lots from a specific supplier, and they were able to reach out to customers to get those batteries replaced before they died. >> So, a mini-recall instead of a massive PR failure. (laughs) >> You know, it was really focused on, you know, customers didn't even know they were going to have a problem with these batteries, that they were going to die early. You know, you got to them ahead of time, told them we knew this was going to be a problem and try to help them. I mean, what a great experience for a customer. (laughs) That's just great. >> So, once you had this telemetry, and it sounds like a bunch of shared repositories, not one intergalactic one. What were some of the other use cases like, you know, like the battery predictive failure type scenarios. >> So, you know, we have some very large gaps, or not gaps, with different categories. We have clearly consumer products. You know, you sell millions and millions of those, and we have little bit of telemetry with those. I think we want to understand failures and ink levels and some of these other things. But, on our commercial web presses, these very large devices, these are very sensitive. These things are down, they have a big problem. So, these things are generating all kinds of data. All right, we have systems on a premise with customers that are alerting them to potential failures, and there's more and more activity going on there to understand predictive failure and predictive kind of tolerance slippages. I'm not super familiar with that business, but I know some guys that they've started introducing more sensors into products, specifically so they can get more data, to understand things. You know, slight variations in tensioning and paper, you know, these things that are running hundreds of feet per minute can have a large impact. So, I think that's really where we see more and more of the value coming from is being able to return that value back to the customer, not just help us make better decisions, but to get that back to the customer. You know, we're talking about expanding more customer-facing analytics in these cases, or we'll expose to customers some of the raw data, and they can build their own dashboards. Some of these industries have traditionally been very analog, so this move to digital web process and this mountain of data is a little new for them, but HP can bring a lot to the table in terms of our experience in computing and big data to help them with their businesses. >> All right, great stuff. And we just got a minute to go before we're done. I have two questions for you, the first is an easy yes/no question. >> John: Okay. >> Is Purdue going to repeat as Big 10 champ in basketball? >> Oh, you know, I don't know. (laughs) I hope so! >> We both went to Purdue. >> I'm more focused on the Warriors winning. (laughter) >> All right, go Warriors! And, the real question is, what surprised you the most? This is your first Spark Summit. What surprised you the most about the event? >> So, you know, you see a lot of Internet-born companies, and it's amazing how many people have just gone fully native with Spark all over the place, and it's a beautiful thing to see. You know, in larger enterprises, that transition doesn't happen like that. I'm kind of jealous. (laughter) We have a lot more things slug through, but the excitement here and all the things that people are working on, you know, you can only see so many tracks. I'm going to have to spend two days when I get back, just watching the videos on all of the tracks I couldn't attend. >> All right, Internet-born companies versus the big enterprise. Good luck herding those cats, and thank you for sharing your story with us today and talking a little bit about the culture there at HP. >> John: Thank you very much. >> And thank you all for watching this segment of theCube. Stay with us, we're still covering Spark Summit 2017. This is Day Two, and we're not done yet. We'll see you in a few minutes. (theCube jingle)

>> Announcer: Live from San Francisco, it's the Cube covering Spark Summit 2017, brought to you by databricks. >> Hi, we're back here where the Cube is live, and I didn't even know it Welcome, we're at Spark Summit 2017. Having so much fun talking to our guests I didn't know the camera was on. We are doing a talk with Cloudera, a couple of experts that we have here. First is Mark Grover, who's a software engineer and an author. He wrote the book, "Dupe Application Architectures." Mark, welcome to the show. >> Mark: Thank you very much. Glad to be here. And just to his left we also have Jennifer Wu, and Jennifer's director of product management at Cloudera. Did I get that right? >> That's right. I'm happy to be here, too. >> Alright, great to have you. Why don't we get started talking a little bit more about what Cloudera is maybe introducing new at the show? I saw a booth over here. Mark, do you want to get started? >> Mark: Yeah, there are two exciting things that we've launched at least recently. There Cloudera Altus, which is for transient work loads and being able to do ETL-Like workloads, and Jennifer will be happy to talk more about that. And then there's Cloudera data science workbench, which is this tool that allows folks to use data science at scale. So, get away from doing data science in silos on your personal laptops, and do it in a secure environment on cloud. >> Alright, well, let's jump into Data Science Workbench first. Tell me a little bit more about that, and you mentioned it's for exploratory data science. So give us a little more detail on what it does. >> Yeah, absolutely. So, there was private beta for Cloudera Data Science Workbench earlier in the year and then it was GA a few months ago. And it's like you said, an exploratory data science tool that brings data science to the masses within an enterprise. Previously people used to have, it was this dichotomy, right? As a data scientist, I want to have the latest and greatest tools. I want to use the latest version of Python, the latest notebook kernel, and I want to be able to use R and Python to be able to crunch this data and run my models in machine learning. However, on the other side of this dichotomy are the IT organization of the organization, where if they want to make sure that all tools are compliant and that your clusters are secure, and your data is not going into places that are not secured by state of the art security solutions, like Kerberos for example, right? And of course if the data scientists are putting the data on their laptops and taking the laptop around to wherever they go, that's not really a solution. So, that was one problem. And the other one was if you were to bring them all together in the same solution, data scientists have different requirements. One may want to use Python 2.6. Another one maybe want to use 3.2, right? And so Cloudera Data Science Workbench is a new product that allows data scientists to visualize and do machine learning through this very nice notebook-like interface, share their work with the rest of their colleagues in the organization, but also allows you to keep your clusters secure. So it allows you to run against a Kerberized cluster, allows single sign on to your web interface to Data Science Workbench, and provides a really nice developer experience in the sense that My workflow and my tools and my version of Python does not conflict with Jennifer's version of Python. We all have our own docker and Kubernetes-based infrastructure that makes sure that we use the packages that we need, and they don't interfere with each other. We're going to go to Jennifer on Altus in just a few minutes, but George first give you a chance to maybe dig in on Data Science workshop. >> Two questions on the data science side: some of the really toughest nuts to crack have been Sort of a common environment for the collaborators, but also the ability to operationalize the models once you've sort of agreed on them, and manage the lifecycle across teams, you know? Like, challenger champion, promote something, or even before that doing the ab testing, and then sort of what's in production is typically in a different language from what, you know, it was designed in and sort of integrating it with the apps. Where is that on the road map? Cause no one really has a good answer for that. >> Yeah, that's an excellent question. In general I think it's the problem to crack these days. How do you productionalize something that was written by a data scientist in a notebook-like system onto the production cluster, right? And I think the part where the data scientist works in a different language than the language that's in production, I think that problem, the best I can say right now is to actually have someone rewrite that. Have someone rewrite that in the language you're going to make in production, right? I don't see that to be the more common part. I think the more widespread problem is even when the language is production, how do you go making the part that the data scientist wrote, the model or whatever that would be, into a prodution cluster? And so, Data Science Workbench in particular runs on the same cluster that is being managed by Cloudera manager, right? So this is a tool that you install, but that is available to you as a web server, as a web interface, and so that allows you to move your development machine learning algorithms from your data science workbench to production much more easier, because it's all running on the same hardware and same systems. There's no separate Cloudera managers that you have to use to manage the workbench compared to your actual cluster. >> Okay. A tangential question, but one of the, the difficulties of doing machine learning is finding all the training data and, and sort of data science expertise to sit with the domain expert to, you know, figure out proper model of features, things like that. One of the things we've seen so far from the cloud vendors is they take their huge datasets in terms of voice, you know, images. They do the natural language understanding, speech or rather text to speech, you know, facial recognition. Cause they have such huge datasets they can train on. We're hearing noises that they'd going to take that down to the more mundane statistical kind of machine learning algorithms, so that you wouldn't be, like, here's a algorithm to do churn, you know, go to town, but that they might have something that's already kind of pre-populated that you would just customize. Is that something that you guys would tackle, too? >> I can't speak for the road map in that sense, but I think some of that problem needs to be tackled by projects like Spark for example. So I think as the stack matures, it's going to raise the level of abstraction as time goes on. And I think whatever benefits Spark ecosystem will have will come directly to distributions like Cloudera. >> George: That's interesting. >> Yeah >> Okay >> Alright, well let's go to Jennifer now and talk about Altus a little bit. Now you've been on the Cube show before, right? >> I have not. >> Okay, well, familiar with your work. Tell us again, you're the product manager for Altus. What does it do, and what was the motivation to build it? >> Yeah, we're really excited about Cloudera Altus. So, we released Cloudera Altus in its first GA form in April, and we launched Cloudera Altus in a public environment in Strata London about two weeks ago, so we're really excited about this and we are very excited to now open this up to all of the customer base. And what it is is a platform as a service offering designed to leverage, basically, the agility and the scale of cloud, and make a very easy to use type of experience to expose Cloudera capacity for, in particular for data engineering type of workloads. So the end user will be able to very easily, in a very agile manner, get data engineering capacity on Cloudera in the cloud, and they'll be able to do things like ETL and large scale data processing, productionized machine learning workflows in the cloud with this new data engineering as a service experience. And we wanted to abstract away the cloud, and cluster operations, and make the end user a really, the end user experience very easy. So, jobs and workloads as first class objects. You can do things like submit jobs, clone jobs, terminate jobs, troubleshoot jobs. We wanted to make this very, very easy for the data engineering end user. >> It does sound like you've sort of abstracted away a lot of the infrastructure that you would associate with on-prem, and sort of almost make it, like, programmable and invisible. But, um, I guess my, one of my questions is when you put it in a cloud environment, when you're on-prem you have a certain set of competitors which is kind of restrictive, because you are the standalone platform. But when you go on the cloud, someone might say, "I want to use red shift on Amazon," or Snowflake, you know, as the MPP sequel database at the end of a pipeline. And it's not just, I'm using those as examples. There's, you know, dozens, hundreds, thousands of other services to choose from. >> Yes. >> What happens to the integrity of that platform if someone carves off one piece? >> Right. So, interoperability and a unified data pipeline is very important to us, so we want to make sure that we can still service the entire data pipeline all the way from ingest and data processing to analytics. So our team has 24 different open source components that we deliver in the CDH distribution, and we have committers across the entire stack. We know the application, and we want to make sure that everything's interoperable, no matter how you deploy the cluster. So if you deploy data engineering clusters through Cloudera Altus, but you deployed Impala clusters for data marks in the cloud through Cloudera Director or through any other format, we want all these clusters to be interoperable, and we've taken great pains in order to make everything work together well. >> George: Okay. So how do Altus and Sata Science Workbench interoperate with Spark? Maybe start with >> You want to go first with Altus? >> Sure, so, we, in terms of interoperability we focus on things like making sure there are no data silos so that the data that you use for your entire data lake can be consumed by the different components in our system, the different compute engines and different tools, and so if you're processing data you can also look at this data and visualize this data through Data Science Workbench. So after you do data ingestion and data processing, you can use any of the other analytic tools and then, and this includes Data Science Workbench. >> Right, and for Data Science Workbench runs, for example, with the latest version of Spark you could pick, the currently latest released version of Spark, Spark 2.1, Spark 2.2 is being boarded of course, and that will soon be integrated after its release. For example you could use Data Science Workbench with your flavor of Spark two's version and you can run PySpark or Scala jobs on this notebook-like interface, be able to share your work, and because you're using Spark Underneath the hood it uses yarn for resource management, the Data Science Workbench itself uses Docker for configuration management, and Kubernetes for resource managing these Docker containers. >> What would be, if you had to describe sort of the edge conditions and the sweet spot of the application, I mean you talked about data engineering. One thing, we were talking to Matei Zaharia and Ronald Chin about was, and Ali Ghodsi as well was if you put Spark on a database, or at least a, you know, sophisticated storage manager, like Kudu, all of a sudden there're a whole new class of jobs or applications that open up. Have you guys thought about what that might look like in the future, and what new applications you would tackle? >> I think a lot of that benefit, for example, could be coming from the underlying storage engine. So let's take Spark on Kudu, for example. The inherent characteristics of Kudu today allow you to do updates without having to either deal with the complexity of something like Hbase, or the crappy performance of dealing HDFS compactions, right? So the sweet spot comes from Kudu's capabilities. Of course it doesn't support transactions or anything like that today, but imagine putting something like Spark and being able to use the machine learning libraries and, we have been limited so far in the machine learning algorithms that we have implemented in Spark by the storage system sometimes, and, for example new machine learning algorithms or the existing ones could rewritten to make use of the update features for example, in Kudu. >> And so, it sounds like it makes it, the machine learning pipeline might get richer, but I'm not hearing that, and maybe this isn't sort of in the near term sort of roadmap, the idea that you would build sort of operational apps that have these sophisticated analytics built in, you know, where the analytics, um, you've done the training but at run time, you know, the inferencing influences a transaction, influences a decision. Is that something that you would foresee? >> I think that's totally possible. Again, at the core of it is the part that now you have one storage system that can do scans really well, and it can also do random reads and writes any place, right? So as your, and so that allows applications which were previously siloed because one appication that ran off of HDFS, another application that ran out of Hbase, and then so you had to correlate them to just being one single application that can use to train and then also use their trained data to then make decisions on the new transactions that come in. >> So that's very much within the sort of scope of imagination, or scope. That's part of sort of the ultimate plan? >> Mark: I think it's definitely conceivable now, yeah. >> Okay. >> We're up against a hard break coming up in just a minute, so you each get a 30-second answer here, so it's the same question. You've been here for a day and a half now. What's the most surprising thing you've learned that you thing should be shared more broadly with the Spark community? Let's start with you. >> I think one of the great things that's happening in Spark today is people have been complaining about latency for a long time. So if you saw the keynote yesterday, you would see that Spark is making forays into reducing that latency. And if you are interested in Spark, using Spark, it's very exciting news. You should keep tabs on it. We hope to deliver lower latency as a community sooner. >> How long is one millisecond? (Mark laughs) >> Yeah, I'm largely focused on cloud infrastructure and I found here at the conference that, like, many many people are very much prepared to actually start taking more, you know, more POCs and more interest in cloud and the response in terms of all of this in Altus has been very encouraging. >> Great. Well, Jennifer, Mark, thank you so much for spending some time here on the Cube with us today. We're going to come by your booth and chat a little bit more later. It's some interesting stuff. And thank you all for watching the Cube today here at Spark Summit 2017, and thanks to Cloudera for bringing us these two experts. And thank you for watching. We'll see you again in just a few minutes with our next interview.

>> Narrator: Live from San Francisco, it's theCUBE, covering Spark Summit 2017. Brought to you by Databricks. >> Welcome back we're here at theCube at Spark Summit 2017. I'm David Goad here with George Gilbert, George. >> Good to be here. >> Thanks for hanging with us. Well here's the other man of the hour here. We just talked with Ali, the CEO at Databricks and now we have the Chief Architect and co-founder at Databricks, Reynold Xin. Reynold, how are you? >> I'm good. How are you doing? >> David: Awesome. Enjoying yourself here at the show? >> Absolutely, it's fantastic. It's the largest Summit. It's a lot interesting things, a lot of interesting people with who I meet. >> Well I know you're a really humble guy but I had to ask Ali what should I ask Reynold when he gets up here. Reynold is one of the biggest contributors to Spark. And you've been with us for a long time right? >> Yes, I've been contributing for Spark for about five or six years and that's probably the most number of commits to the project and lately more I'm working with other people to help design the roadmap for both Spark and Databricks with them. >> Well let's get started talking about some of the new developments that you want maybe our audience at theCUBE hasn't heard here in the keynote this morning. What are some of the most exciting new developments? >> So, I think in general if we look at Spark, there are three directions I would say we doubling down. One the first direction is the deep learning. Deep learning is extremely hot and it's very capable but as we alluded to earlier in a blog post, deep learning has reached sort of a mass produced point in which it shows tremendous potential but the tools are very difficult to use. And we are hoping to democratize deep learning and do what Spark did to big data, to deep learning with this new library called deep learning pipelines. What it does, it integrates different deep learning libraries directly in Spark and can actually expose models in sequel. So, even the business analysts are capable of leveraging that. So, that one area, deep learning. The second area is streaming. Streaming, again, I think that a lot of customers have aspirations to actually shorten the latency and increase the throughput in streaming. So, the structured streaming effort is going to be generally available and last month alone on Databricks platform, I think out customers processed three trillion records, last month alone using structured streaming. And we also have a new effort to actually push down the latency all the way to some millisecond range. So, you can really do blazingly fast streaming analytics. And last but not least is the SEQUEL Data Warehousing area, Data warehousing I think that it's a very mature area from the outset of big data point of view, but from a big data one it's still pretty new and there's a lot of use cases that's popping up there. And Spark with approaches like the CBO and also impact here in the database runtime with DBIO, we're actually substantially improving the performance and the capabilities of data warehousing futures. >> We're going to dig in to some of those technologies here in just a second with George. But have you heard anything here so far from anyone that's changed your mind maybe about what to focus on next? So, one thing I've heard from a few customers is actually visibility and debugability of the big data jobs. So many of them are fairly technical engineers and some of them are less sophisticated engineers and they have written jobs and sometimes the job runs slow. And so the performance engineer in me would think so how do I make the job run fast? The different way to actually solve that problem is how can we expose the right information so the customer can actually understand and figure it out themselves. This is why my job is slow and this how I can tweak it to make it faster. Rather than giving people the fish, you actually give them the tools to fish. >> If you can call that bugability. >> Reynold: Yeah, Debugability. >> Debugability. >> Reynold: And visibility, yeah. >> Alright, awesome, George. >> So, let's go back and unpack some of those kind of juicy areas that you identified, on deep learning you were able to distribute, if I understand things right, the predictions. You could put models out on a cluster but the really hard part, the compute intensive stuff, was training across a cluster. And so Deep Learning, 4J and I think Intel's BigDL, they were written for Spark to do that. But with all the excitement over some of the new frameworks, are they now at the point where they are as good citizens on Spark as they are on their native environments? >> Yeah so, this is a very interesting question, obviously a lot of other frameworks are becoming more and more popular, such as TensorFlow, MXNet, Theano, Keras and Office. What the Deep Learning Pipeline library does, is actually exposes all these single note Deep Learning tools as highly optimized for say even GPUs or CPUs, to be available as a estimator or like a module in a pipeline of the machine learning pipeline library in spark. So, now users can actually leverage Spark's capability to, for example, do hyper parameter churning. So, when you're building a machine learning model, it's fairly rare that you just run something once and you're good with it. Usually have to fiddle with a lot of the parameters. For example, you might run over a hundred experiments to actually figure out what is the best model I can get. This is where actually Spark really shines. When you combine Spark with some deep learning library be it BigDL or be it MXNet, be it TensorFlow, you could be using Spark to distribute that training and then do cross validation on it. So you can actually find the best model very quickly. And Spark takes care of all the job scheduling, all the tolerance properties and how do you read data in from different data sources. >> And without my dropping too much in the weeds, there was a version of that where Spark wouldn't take care of all the communications. It would maybe distribute the models and then do some of the averaging of what was done out on the cluster. Are you saying that all that now can be managed by Spark? >> In that library, Spark will be able to actually take care of picking the best model out of it. And there are different ways you an design how do you define the best. The best could be some average of some different models. The best could be just pick one out of this. The best could be maybe there's a tree of models that you classify it on. >> George: And that's a hyper parameter configuration choice? >> So that is actually building functionality in Sparks machine learning pipeline. And now what we're doing is now you can actually plug all those deep learning libraries directly into that as part of the pipeline to be used. Another maybe just to add, >> Yeah, yeah, >> Another really cool functionality of the deep learning pipeline is transfer learning. So as you said, deep learning takes a very long time, it's very computationally demanding. And it takes a lot of resources, expertise to train. But with transfer learning what we allow the customers to do is they can take an existing deep learning model as well train in a different domain and they we'd retrain it on a very small amount of data very quickly and they can adapt it to a different domain. That's how sort of the demo on the James Bond car. So there is a general image classifier that we train it on probably just a few thousand images. And now we can actually detect whether a car is James Bond's car or not. >> Oh, and the implications there are huge, which is you don't have to have huge training data sets for modifying a model of a similar situation. I want to, in the time we have, there's always been this debate about whether Sparks should manage state, whether it's database, key value store. Tell us how the thinking about that has evolved and then how the integration interfaces for achieving that have evolved. >> One of the, I would say, advantages of Spark is that it's unbiased and works with a variety of storage systems, be it Cassandra, be it Edgebase, be it HDFS, be is S3. There is a metadata management functionality in Spark which is the catalog of tables that customers can define. But the actual storage sits somewhere else. And I don't think that will change in the near future because we do see that the storage systems have matured significantly in the last few years and I just wrote blog post last week about the advantage of S3 over HDFS for example. The storage price is being driven down by almost a factor of 10X when you go to the cloud. I just don't think it makes sense at this point to be building storage systems for analytics. That said, I think there's a lot of building on top of existing storage system. There's actually a lot of opportunities for optimization on how you can leverage the specific properties of the underlying storage system to get to maximum performance. For example, how are you doing intelligent caching, how do you start thinking about building indexes actually against the data that's stored for scanned workloads. >> With Tungsten's, you take advantage of the latest hardware and where we get more memory intensive systems and now that the Catalyst Optimizer has a cost based optimizer or will be, and large memory. Can you change how you go about knowing what data you're managing in the underlying system and therefore, achieve a tremendous acceleration in performance? >> This is actually one area we invested in the DBIO module as part of Databricks Runtime, and what DBIO does, a lot of this are still in progress, but for example, we're adding some form of indexing capability to add to the system so we can quickly skip and prune out all the irrelevant data when the user is doing simple point look-ups. Or if the user is doing a scan heavy workload with some predicates. That actually has to do with how we think about the underlying data structure. The storage system is still the same storage system, like S3, but were adding actually indexing functionalities on top of it as part of DBIO. >> And so what would be the application profiles? Is it just for the analytic queries or can you do the point look-ups and updates in that sort of scenario too? >> So it's interesting you're talking about updates. Updates is another thing that we've got a lot of future requests on. We're actively thinking about how we will support update workload. Now, that said, I just want to emphasize for both use case of doing point look-ups and updates, we're still talking about in the context of analytic environment. So we would be talking about for example maybe bulk updates or low throughput updates rather than doing transactional updates in which every time you swipe a credit card, some record gets updated. That's probably more belongs on the transactional databases like Oracle or my SEQUEL even. >> What about when you think about people who are going to run, they started out with Spark on prem, they realize they're going to put much more of their resources in the cloud, but with IIOT, industrial IOT type applications they're going to have Spark maybe in a gateway server on the edge? What do you think that configuration looks like? >> Really interesting, it's kind of two questions maybe. The first is the hybrid on prem, cloud solution. Again, so one of the nice advantage of Spark is the couple of storage and compute. So when you want to move for example, workloads from one prem to the cloud, the one you care the most about is probably actually the data 'cause the compute, it doesn't really matter that much where you run it but data's the one that's hard to move. We do have customers that's leveraging Databricks in the cloud but actually reading data directly from on prem the reliance of the caching solution we have that minimize the data transfer over time. And is one route I would say it's pretty popular. Another on is, with Amazon you can literally give them just a show ball of functionality. You give them hard drive with trucks, the trucks will ship your data directly put in a three. With IOT, a common pattern we see is a lot of the edge devices, would be actually pushing the data directly into some some fire hose like Kinesis or Kafka or, I'm sure Google and Microsoft both have their own variance of that. And then you use Spark to directly subscribe to those topics and process them in real time with structured streaming. >> And so would Spark be down, let's say at the site level. if it's not on the device itself? >> It's a interesting thought and maybe one thing we should actually consider more in the future is how do we push Spark to the edges. Right now it's more of a centralized model in which the devices push data into Spark which is centralized somewhere. I've seen for example, I don't remember exact the use case but it has to do with some scientific experiment in the North Pole. And of course there you don't have a great uplink of all the data connecting transferring back to some national lab and rather they would do a smart parsing there and then ship the aggregated result back. There's another one but it's less common. >> Alright well just one minute now before the break so I'm going to give you a chance to address the Spark community. What's the next big technical challenge you hope people will work on for the benefit of everybody? >> In general Spark came along with two focuses. One is performance, the other one's ease of use. And I still think big data tools are too difficult to use. Deep learning tools, even harder. The barrier to entry is very high for office tools. I would say, we might have already addressed performance to a degree that I think it's actually pretty usable. The systems are fast enough. Now, we should work on actually make (mumbles) even easier to use. It's what also we focus a lot on at Databricks here. >> David: Democratizing access right? >> Absolutely. >> Alright well Reynold, I wish we could talk to you all day. This is great. We are out of time now. Want to appreciate you coming by theCUBE and sharing your insights and good luck with the rest of the show. >> Thank you very much David and George. >> Thank you all for watching here were at theCUBE at Sparks Summit 2017. Stay tuned, lots of other great guests coming up today. We'll see you in a few minutes.

>> Announcer: Live from San Francisco it's theCUBE Covering Spark Summit 2017, brought to you by Databricks. >> Welcome back to theCUBE. You are watching coverage of Spark Summit 2017. It's day two, we've got so many new guests to talk to today. We already learned a lot, right George? >> Yeah, I mean we had some, I guess, pretty high bandwidth conversations. >> Yes, well I expect we're going to have another one here too, because the person we have is the founder of Predictive Analytics World, it's Eric Siegel, Eric welcome to the show. >> Hey thanks Dave, thanks George. You go by Dave or David? >> Dave: Oh you can call me sir, and that would be. >> I was calling you, should I, can I bow? >> Oh no we are bowing to you, you're the author of the book, Predictive Analytics, I love the subtitle, the Power to Predict Who Will Click, Buy, Lie or Die. >> And that sums up the industry right? >> Right, so if people are new to the industry, that's sort of an informal definition of predictive analytics, basically also known as machine learning. Where you're trying to make predictions for each individual, whether it's a customer for marketing, a suspect for fraud or law enforcement, a voter for political campaigning, a patient for healthcare. So, in general it's on that level, it's a prediction for each individual. So how does data help make those predictions? And then you can only imagine just how many ways in which predicting on that level helps organizations improve all their activities. >> Well we know you were on the keynote stage this morning. Could you maybe summarize for the CUBE audience, what a couple of the top themes that you were talking about? >> Yeah, I covered two advanced topics. I wanted to make sure this pretty technical audience was aware of because a lot of people aren't and one is called uplift modeling, so that's optimizing for persuasion for things like marketing and also for healthcare, actually. And for political campaigning. So when you do predictive analytics for targeting marketing normally sort of the traditional approach is, let's predict will this person buy if I contact them because when well its okay maybe its a good idea to spend the two dollars to send them a brochure its marketing treatment, right. But there is actually a little bit different question that would make even driving them better decisions Which is not will this person buy but would contacting them, sending them the brochure, influence them to buy, will it increase the chance that we get that positive outcome. That's a different question, and it doesn't correspond with standard predictive modeling or machine learning methods So uplift modeling, also known as net lift modeling, persuasion modeling its a way to actually create a predictive model like any other except that it's target is, is it a good idea to contact this person because it will increase the chances that they are going to have a positive outcome. So that's the first of the two. And I cram this all in 20 minutes. The other one was a little more commonly known But I think people would like to visit it and it's called P-Hacking or vast search. Where you can be fooled by randomness and data relatively easily in the era of Big Data there is this all to common pitfall where you find a predictive insight in the data and it turns out it was actually just a random perturbation. How do you know the difference? >> Dave: Fake news right? >> Okay fake news, except that in this case, it was generated by a computer, right? And then there is a statistical test that makes it look like its actually statistically significant and we should have credibility to it, on it or about it. So you can avert it, you have compensate for the fact that you are trying lots, that you are evaluating many different predictive insights or hypotheses whatever you want to call it and make sure that the one that you are believing you sort of checked for the ability that it wasn't just random luck, that's known as p-hacking. >> Alright, so uplift modeling and p-hacking. George do you want to drill on those a little bit. >> Yeah, I want to start from maybe the vocabulary of our audience where they say sort of like uplift modeling goes beyond prediction. Actually even for the second one with p-hacking is that where you're essentially playing with the parameters of the model to find the difference between correlation and causation and going from prediction to prescription? >> It's not about causation, its actually so correlation is what you get when you get a predictive insight or some component of a predictive model where you see these things connected therefore one is predictive of the other. Now the fact that does not entail causation is a really good point to remind people of as such. But even before you address that question, the first question is this correlation actually legit? Is there really a correlation between this things? Is this an actual finding? Or is it just happened to be the case in this particular sample of limited sample data that I have access to at the moment, right? So is it a real link or correlation in the first place before you even start asking any question about causality and it does have, it does related to what you alluded to with regard to tuning parameters because its closely related to this issue of overfitting. People who do predictive modeling are very familiar with overfitting. The standard practice all tools implementations of machine learning and predictive modeling do this, which is they hold the side evaluation set called test set. So you don't get to cheat, creates a predict model. It learns from the data, does the number crunching, its mostly automated, right. And it comes out with this beautiful model that does well predicting and then you evaluate, you assess it over this held aside. Oh my thing's falling off here. >> Dave: Just second on your. >> See then you evaluate it on this held aside set it was quarantine so you didn't get to cheat. You didn't get to look at it when you are creating the model. So it serves as an objective performance measure. The problem is and here is the huge irony, the things that we get from data, the predictive insights, there was one famous one that was broadcasted too loudly because its not nearly as credible as they first thought. Is that an orange used car is a better one to buy because its less likely to be a lemon. That's what it looked like in this one data set. The problem is, that when you have a single insight where its relatively simple, just talking about the car, the color to make the prediction. A predictive model is much more complex and deals with lots of other attributes not just the color, for example, make, year, model everything on that individual car, individual person, you can imagine all the attributes that's the point of the modeling process, the learning process, how do you consider multiple things. If its just a really simple thing with just based on the car color, then many of even the most advanced data science practitioners kind of forget that there is still potential to effectively overfit, that you might have found something that doesn't apply in general, only applies over this particular set of data. So that's where the trap falls and they don't necessarily hold themselves a high standard of having this held aside test set. So its kind of ironic thing, the things that most likely to make the headlines like orange cars are simpler, easier to understand, but are less well understood that they could be wrong. >> You know keying off that, that's really interesting, because we've been hearing for years that what's made, especially deep learning relevant over the last few years is huge compute up in the cloud and huge data sets. >> Yeah. >> But we're also starting to hear about methods of generating a sort of synthetic data so that if you don't have, I don't know what the term is, organic training data, and then test data, we're getting to the point where we can do high quality models with less. >> Yes, less of that training data. And did you. >> Tell us. >> Did you interview with the keynote speaker from Stanford about that? >> No, I only saw part of his. >> Yeah his speech yesterday. That's an area that I'm relatively new to but it sounds extremely important because that is the bottleneck. He called it, if data's the new oil, he's calling it the new-new oil. Which is more specific than data, it's training data. So all of the machine learning or predictive modeling methods of which we speak, are, in most cases, what's called supervised learning. So the thing that makes it supervised is you have a bunch of examples where you already know the answer. So you're trying to figure out is this picture of a cat or of a dog, that means you need to have a whole bunch of data from which to learn, the training data, where you've already got it labeled. You already know the correct answer. In many business applications just because of history you know who did or didn't respond to your marketing, you know who did or did not turn out to be fraudulent. History is experience in which to learn, it's in the data, so you do have that labeled, yes, no, like you already know the answer, you don't need to predict on them, it's in the past but you use that as training data. So we have that in many cases. But for something like classifying an image, and we're trying to figure out does this have a picture of a cat somewhere in the image, or whatever all these big image classification problems, you do need, often, a manual effort to label the data. Have the positive and negative examples, that's what's called training data, the learning data. It's actually called training data. There's definitely a bottleneck so anything that can be done to avert that bottleneck decrease the amount that we need, or find ways to make, sort of, rough training data that may serve as a building block for the modeling process this kind of thing. That's not my area of expertise, sounds really intriguing though. >> What about, and this may be further out on the horizon but one thing we are hearing about is the extreme shortage of data scientists who need to be teamed up with domain experts to figure out the knobs, the variables to create these elaborate models. We're told that even if you're doing the traditional, statistical, machine learning models, that eventually deep learning can help us identify the features or the variables just the way they sort of identify you know ears and whiskers and a nose and then figure out from that the cat. That's something that's in the near term, the medium term in terms of helping to augment what the data scientist does? >> It's in the near term and that's why everyone's excited about deep learning right now is that, basically the reason we built these machines called computers is because they automate stuff. Pretty much anything that you can think of and define well, you can program. Then you've got a machine that does it. Of course one of the things we wanted to learn, to do actually, is to learn from data. Now, it's literally really very analogous to what it means for a human to learn. You've got a limited number of examples that you're trying to draw generalizations from those. When you go to bigger scale problems where the thing you're classifying isn't just like a customer, and all the things you know about the customer, are they likely to commit fraud, yes or no. But it become a level more complex when it's an image right, image is worth a thousand words. And maybe literally more than a thousand words where it says of data if it's a high resolution. So how do you process that? Well there's all sorts of research like well we can define the thing that tries to find arcs, and circles and edges and this kind of thing, or, we can try to, once again, let that be automatic. Let the computer do that. So deep learning is a way to allow, spark is a way to make it operate quickly but there's another level of scale other than speed. The level of scale is just like how complex of a task can you leave up to the automaton, to go by itself. That's what deep learning does is it scales in that respect it has the ability to automate more layers of that complexity as far as finding those kinds of what might me domain specific features and images. >> Okay, but I'm thinking not just the, help me figure out speech to text and natural language understanding or classify. >> Anything with a signal where it's a high bandwidth amount of data coming in that you want to classify. >> OK, so could that, does that extend to I'm building a very elaborate predictive model not on, is there a cat in the video or in the picture so much as I guess you called it, is there an uplift potential and how big is that potential, in a context of making a sale on an eCommerce site. >> So what you just tapped into was when you go to marketing and many other business applications, you don't actually need to have high accuracy what you have to do is have a prediction that's better than guessing. So for example, if I get a 1% response rate to my marketing campaign, but I can find a pocket that's got 3% response rate, it may be very much rocket science to define and learn from the data how to define that specifically defined sub-segment that has a higher response rate, or whatever it is. But the 3% isn't like, I have high confidence this person's definitely going to buy, it's still just 3%, but that difference can make a huge difference and can improve the bottom line marketing by a factor of five and that kind of thing. It's not necessarily about accuracy. If you've got an image and you need to know is there a picture of a car, or is this traffic light green or red, somewhere in this image, then there's certain application areas, self driving cars what have you, it does need to be accurate right. But maybe there's more potential for it to be accurate because there's more predictability inherent to that problem. Like I can predict that there's a traffic light that has a green light somewhere in an image because there is enough label data and the nature of the problem is more tractable because it's not as challenging to find where the traffic light is, and then which color it is. You need it to scale, to reach that level of classification performance in terms of accuracy or whatever measure you use for certain applications. >> Are you seeing like new methodologies like reinforcement learning or deep learning where the models are adversarial where they make big advances in terms of what they can learn without a lot of supervision? Like the ones where. >> It's more self learning and unsupervised. >> Sort of glue yourself onto this video game screen we'll give you control of the steering wheel and you figure out how to win. >> Having less required supervision, more self-learning, anomaly detection or clustering, these are some of the unsupervised ones. When it comes to vision there are part of the process that can be unsupervised in the sense that you don't need labels on your target like is there a car in the picture. But it can still learn the feature detection in a way that doesn't have that supervised data. Although that image classification in general, on that level deep learning, is not my area of expertise. That's a very up and coming part of machine learning but it's only needed when you have these high bandwidth inputs like an entire image, high resolution, or a video, or a high bandwidth audio. So it's signal processing type problems where you start to need that kind of deep learning. >> Great discussion Eric, just a couple of minutes to go in this segment here. I want to make sure I give a chance to talk about Predictive Analytics World and what's your affiliation with that ad what do you want theCUBE audience to know? >> Oh sure, Predictive Analytics World I'm the founder it's the leading cross-vendor event focused on commercial deployment of predictive analytics and machine learning. Our main event a few times a year is a broad scope business focused event but we also have industry vertical focused specialized events just for financial services, healthcare, workforce, manufacturing and government applications of predictive analytics and machine learning. So there's a number a year, and two weeks from now in Chicago, October in New York and you can see the full agendas at PredictiveAnalyticsWorld.com. >> Alright great short commercial there. 30 seconds. >> It's the elevator pitch. >> Answered the toughest question in 30 seconds what the toughest question you got after your keynote this morning? Maybe a hallway conversation or. >> What's the toughest question I got after my keynote? >> Dave: From one of the attendees. >> Oh, the question that always comes up is how do you get this level of complexity across to non-technical people or your boss or your colleagues or your friends and family. By the way that's something I worked really hard on with the book which is meant for all readers although the last few chapters have. >> How do you get executive sponsors to get what you're doing? >> Well, as I say, give them the book. Because the point of the book is it's pop science it's accessible, it's analytically driven, it's entertaining it keeps it relevant but it does address advanced topics at the end of the book. So it sort of ends, industry overview kind of thing. The bottom line there, in general, is that you want to focus on the business impact. What I mentioned briefly a second ago if we can improve target marketing this much it will increase profit by a factor five something like that. So you start with that and then answer any questions they have about, well how does it work, what makes it credible that it really has that much potential in the bottom line. When you're a techie, you're inclined to go forward you start with the technology that you're excited about. That's my background, so that's sort of the definition of being a geek, that you're ore enamored with the technology than the value it produces. Because it's amazing that it works, and it's exciting, it's interesting, it's scientifically challenging. But, when you're talking to the decision makers you have to start with the eventual carrot at the end of the stick, which is the value. >> The business outcome. >> Yeah. >> Great, well that's going to be the last word. That might even make it onto our CUBE Gems segment, great sound bites. George thanks again, great questions and Eric the author of Predictive Analytics, the Power to Predict Who Will Click, Buy, Lie or Die. Thank you for being on the show we appreciate your time. >> Eric: Sure, yeah thank you, great to meet you. >> Thank you for watching theCUBE we'll be back in just a few minutes with our next guest here at Spark Summit 2017.