>> Narrator: From San Jose, California, in the heart of Silicon Valley, it's theCUBE. Covering Hadoop Summit 2016. Brought to you by Hortonworks. Here's your host, John Furrier. >> Welcome back everyone. We are here live in Silicon Valley at Hadoop Summit 2016, actually San Jose. This is theCUBE, our flagship program. We go out to the events and extract the signal to the noise. Our next guest, David Richards, CEO of WANdisco. And Joel Horowitz, strategy and business development, IBM analyst. Guys, welcome back to theCUBE. Good to see you guys. >> Thank you for having us. >> It's great to be here, John. >> Give us the update on WANdisco. What's the relationship with IBM and WANdisco? 'Cause, you know. I can just almost see it, but I'm not going to predict. Just tell us. >> Okay, so, I think the last time we were on theCUBE, I was sitting with Re-ti-co who works very closely with Joe. And we began to talk about how our partnership was evolving. And of course, we were negotiating an OEM deal back then, so we really couldn't talk about it very much. But this week, I'm delighted to say that we announced, I think it's called IBM Big Replicate? >> Joel: Big Replicate, yeah. We have a big everything and Replicate's the latest edition. >> So it's going really well. It's OEM'd into IBM's analytics, big data products, and cloud products. >> Yeah, I'm smiling and smirking because we've had so many conversations, David, on theCUBE with you on and following your business through the bumpy road or the wild seas of big data. And it's been a really interesting tossing and turning of the industry. I mean, Joel, we've talked about it too. The innovation around Hadoop and then the massive slowdown and realization that cloud is now on top of it. The consumerization of the enterprise created a little shift in the value proposition, and then a massive rush to build enterprise grade, right? And you guys had that enterprise grade piece of it. IBM, certainly you're enterprise grade. You have enterprise everywhere. But the ecosystem had to evolve really fast. What happened? Share with the audience this shift. >> So, it's classic product adoption lifecycle and the buying audience has changed over that time continuum. In the very early days when we first started talking more at these events, when we were talking about Hadoop, we all really cared about whether it was Pig and Hive. >> You once had a distribution. That's a throwback. Today's Thursday, we'll do that tomorrow. >> And the buying audience has changed, and consequently, the companies involved in the ecosystem have changed. So where we once used to really care about all of those different components, we don't really care about the machinations below the application layer anymore. Some people do, yes, but by and large, we don't. And that's why cloud for example is so successful because you press a button, and it's there. And that, I think, is where the market is going to very, very quickly. So, it makes perfect sense for a company like WANdisco who've got 20, 30, 40, 50 sales people to move to a company like IBM that have 4 or 5,000 people selling our analytics products. >> Yeah, and so this is an OEM deal. Let's just get that news on the table. So, you're an OEM. IBM's going to OEM their product and brand it IBM, Big Replication? >> Yeah, it's part of our Big Insights Portfolio. We've done a great job at growing this product line over the last few years, with last year talking about how we decoupled all the value-as from the core distribution. So I'm happy to say that we're both part of the ODPI. It's an ODPI-certified distribution. That is Hadoop that we offer today for free. But then we've been adding not just in terms of the data management capabilities, but the partnership here that we're announcing with WANdisco and how we branded it as Big Replicate is squarely aimed at the data management market today. But where we're headed, as David points out, is really much bigger, right? We're talking about support for not only distributed storage and data, but we're also talking about a hybrid offering that will get you to the cloud faster. So not only does Big Replicate work with HDFS, it also works with the Swift objects store, which as you know, kind of the underlying storage for our cloud offering. So what we're hoping to see from this great partnership is as you see around you, Hadoop is a great market. But there's a lot more here when you talk about managing data that you need to consider. And I think hybrid is becoming a lot larger of a story than simply distributing your processing and your storage. It's becoming a lot more about okay, how do you offset different regions? How do you think through that there are multiple, I think there's this idea that there's one Hadoop cluster in an enterprise. I think that's factually wrong. I think what we're observing is that there's actually people who are spinning up, you know, multiple Hadoop distributions at the line of business for maybe a campaign or for maybe doing fraud detection, or maybe doing log file, whatever. And managing all those clusters, and they'll have Cloud Arrow. They'll have Hortonworks. They'll have IBM. They'll have all of these different distributions that they're having to deal with. And what we're offering is sanity. It's like give me sanity for how I can actually replicate that data. >> I love the name Big Replicate, fantastic. Big Insights, Big Replicate. And so go to market, you guys are going to have bigger sales force. It's a nice pop for you guys. I mean, it's good deal. >> We were just talking before we came on air about sort of a deal flow coming through. It's coming through, this potential deal flow coming through, which has been off the charts. I mean, obviously when you turn on the tap, and then suddenly you enable thousands and thousands of sales people to start selling your products. I mean, IBM, are doing a great job. And I think IBM are in a unique position where they own both cloud and on-prem. There are very few companies that own both the on-prem-- >> They're going to need to have that connection for the companies that are going hybrid. So hybrid cloud becomes interesting right now. >> Well, actually, it's, there's a theory that says okay, so, and we were just discussing this, the value of data lies in analytics, not in the data itself. It lies in you've been able to pull out information from that data. Most CIOs-- >> If you can get the data. >> If you can get the data. Let's assume that you've got the data. So then it becomes a question of, >> That's a big assumption. Yes, it is. (laughs) I just had Nancy Handling on about metadata. No, that's an issue. People have data they store they can't do anything with it. >> Exactly. And that's part of the problem because what you actually have to have is CPU slash processing power for an unknown amount of data any one moment in time. Now, that sounds like an elastic use case, and you can't do elastic on-prem. You can only do elastic in cloud. That means that virtually every distribution will have to be a hybrid distribution. IBM realized this years ago and began to build this hybrid infrastructure. We're going to help them to move data, completely consistent data, between on-prem and cloud, so when you query things in the cloud, it's exactly the same results and the correct results you get. >> And also the stability too on that. There's so many potential, as we've discussed in the past, that sounds simple and logical. To do an enterprise grade is pretty complex. And so it just gives a nice, stable enterprise grade component. >> I mean, the volumes of data that we're talking about here are just off the charts. >> Give me a use case of a customer that you guys are working with, or has there been any go-to-market activity or an ideal scenario that you guys see as a use case for this partnership? >> We're already seeing a whole bunch of things come through. >> What's the number one pattern that bubbles up to the top? Use case-wise. >> As Joel pointed out, that he doesn't believe that any one company just has one version of Hadoop behind their firewall. They have multiple vendors. >> 100% agree with that. >> So how do you create one, single cluster from all of those? >> John: That's one problem you solved. >> That's of course a very large problem. Second problem that we're seeing in spades is I have to move data to cloud to run analytics applications against it. That's huge. That required completely guaranteed consistent data between on-prem and cloud. And I think those two use cases alone account for pretty much every single company. >> I think there's even a third here. I think the third is actually, I think frankly there's a lot of inefficiencies in managing just HDFS and how many times you have to actually copy data. If I looked across, I think the standard right now is having like three copies. And actually, working with Big Replicate and WANdisco, you can actually have more assurances and actually have to make less copies across the cluster and actually across multiple clusters. If you think about that, you have three copies of the data sitting in this cluster. Likely, an analysts have a dragged a bunch of the same data in other clusters, so that's another multiple of three. So there's amount of waste in terms of the same data living across your enterprise. That I think there's a huge cost-savings component to this as well. >> Does this involve anything with Project Atlas at all? You guys are working with, >> Not yet, no. >> That project? It's interesting. We're seeing a lot of opening up the data, but all they're doing is creating versions of it. And so then it becomes version control of the data. You see a master or a centralization of data? Actually, not centralize, pull all the data in one spot, but why replicate it? Do you see that going on? I guess I'm not following the trend here. I can't see the mega trend going on. >> It's cloud. >> What's the big trend? >> The big trend is I need an elastic infrastructure. I can't build an elastic infrastructure on-premise. It doesn't make economic sense to build massive redundancy maybe three or four times the infrastructure I need on premise when I'm only going to use it maybe 10, 20% of the time. So the mega trend is cloud provides me with a completely economic, elastic infrastructure. In order to take advantage of that, I have to be able to move data, transactional data, data that changes all the time, into that cloud infrastructure and query it. That's the mega trend. It's as simple as that. >> So moving data around at the right time? >> And that's transaction. Anybody can say okay, press pause. Move the data, press play. >> So if I understand this correctly, and just, sorry, I'm a little slow. End of the day today. So instead of staging the data, you're moving data via the analytics engines. Is that what you're getting at? >> You use data that's being transformed. >> I think you're accessing data differently. I think today with Hadoop, you're accessing it maybe through like Flume or through Oozy, where you're building all these data pipelines that you have to manage. And I think that's obnoxious. I think really what you want is to use something like Apache Spark. Obviously, we've made a large investment in that earlier, actually, last year. To me, what I think I'm seeing is people who have very specific use cases. So, they want to do analysis for a particular campaign, and so they may just pull a bunch of data into memory from across their data environment. And that may be on the cloud. It may be from a third-party. It may be from a transactional system. It may be from anywhere. And that may be done in Hadoop. It may not, frankly. >> Yeah, this is the great point, and again, one of the themes on the show is, this is a question that's kind of been talked about in the hallways. And I'd love to hear your thoughts on this. Is there are some people saying that there's really no traction for Hadoop in the cloud. And that customers are saying, you know, it's not about just Hadoop in the cloud. I'm going to put in S3 or object store. >> You're right. I think-- >> Yeah, I'm right as in what? >> Every single-- >> There's no traction for Hadoop in the cloud? >> I'll tell you what customers tell us. Customers look at what they actually need from storage, and they compare whatever it is, Hadoop or any on-premise proprietor storage array and then look at what S3 and Swift and so on offer to them. And if you do a side-by-side comparison, there isn't really a difference between those two things. So I would argue that it's a fact that functionally, storage in cloud gives you all the functionality that any customer would need. And therefore, the relevance of Hadoop in cloud probably isn't there. >> I would add to that. So it really depends on how you define Hadoop. If you define Hadoop by the storage layer, then I would say for sure. Like HDFS versus an objects store, that's going to be a difficult one to find some sort of benefit there. But if you look at Hadoop, like I was talking to my friend Blake from Netflix, and I was asking him so I hear you guys are kind of like replatforming on Spark now. And he was basically telling me, well, sort of. I mean, they've invested a lot in Pig and Hive. So if you think it now about Hadoop as this broader ecosystem which you brought up Atlas, we talk about Ranger and Knox and all the stuff that keeps coming out, there's a lot of people who are still invested in the peripheral ecosystem around Hadoop as that central point. My argument would be that I think there's still going to be a place for distributed computing kind of projects. And now whether those will continue to interface through Yarn via and then down to HDFS, or whether that'll be Yarn on say an objects store or something and those projects will persist on their own. To me that's kind of more of how I think about the larger discussion around Hadoop. I think people have made a lot of investments in terms of that ecosystem around Hadoop, and that's something that they're going to have to think through. >> Yeah. And Hadoop wasn't really designed for cloud. It was designed for commodity servers, deployment with ease and at low cost. It wasn't designed for cloud-based applications. Storage in cloud was designed for storage in cloud. Right, that's with S3. That's what Swift and so on were designed specifically to do, and they fulfill most of those functions. But Joel's right, there will be companies that continue to use-- >> What's my whole argument? My whole argument is that why would you want to use Hadoop in the cloud when you can just do that? >> Correct. >> There's object store out. There's plenty of great storage opportunities in the cloud. They're mostly shoe-horning Hadoop, and I think that's, anyway. >> There are two classes of customers. There were customers that were born in the cloud, and they're not going to suddenly say, oh you know what, we need to build our own server infrastructure behind our own firewall 'cause they were born in the cloud. >> I'm going to ask you guys this question. You can choose to answer or not. Joel may not want to answer it 'cause he's from IBM and gets his wrist slapped. This is a question I got on DM. Hadoop ecosystem consolidation question. People are mailing in the questions. Now, keep sending me your questions if you don't want your name on it. Hold on, Hadoop system ecosystem. When will this start to happen? What is holding back the M and A? >> So, that's a great question. First of all, consolidation happens when you sort of reach that tipping point or leveling off, that inflection point where the market levels off, and we've reached market saturation. So there's no more market to go after. And the big guys like IBM and so on come in-- >> Or there was never a market to begin with. (laughs) >> I don't think that's the case, but yes, I see the point. Now, what's stopping that from happening today, and you're a naughty boy by the way for asking this question, is a lot of these companies are still very well funded. So while they still have cash on the balance sheet, of course, it's very, very hard for that to take place. >> You picked up my next question. But that's a good point. The VCs held back in 2009 after the crash of 2008. Sequoia's memo, you know, the good times role, or RIP good times. They stopped funding companies. Companies are getting funded, continually getting funding. Joel. >> So I don't think you can look at this market as like an isolated market like there's the Hadoop market and then there's a Spark market. And then even there's like an AI or cognitive market. I actually think this is all the same market. Machine learning would not be possible if you didn't have Hadoop, right? I wouldn't say it. It wouldn't have a resurgence that it has had. Mahout was one of the first machine learning languages that caught fire from Ted Dunning and others. And that kind of brought it back to life. And then Spark, I mean if you talk to-- >> John: I wouldn't say it creates it. Incubated. >> Incubated, right. >> And created that Renaissance-like experience. >> Yeah, deep learning, Some of those machine learning algorithms require you to have a distributed kind of framework to work in. And so I would argue that it's less of a consolidation, but it's more of an evolution of people going okay, there's distributed computing. Do I need to do that on-premise in this Hadoop ecosystem, or can I do that in the cloud, or in a growing Spark ecosystem? But I would argue there's other things happening. >> I would agree with you. I love both areas. My snarky comment there was never a market to begin with, what I'm saying there is that the monetization of commanding the hill that everyone's fighting for was just one of many hills in a bigger field of hills. And so, you could be in a cul-de-sac of being your own champion of no paying customers. >> What you have-- >> John: Or a free open-source product. >> Unlike the dotcom era where most of those companies were in the public markets, and you could actually see proper valuations, most of the companies, the unicorns now, most are not public. So the valuations are really difficult to, and the valuation metrics are hard to come by. There are only few of those companies that are in the public market. >> The cash story's right on. I think to Joel' point, it's easy to pivot in a market that's big and growing. Just 'cause you're in the wrong corner of the market pivoting or vectoring into the value is easier now than it was 10 years ago. Because, one, if you have a unicorn situation, you have cash on the bank. So they have a good flush cash. Your runway's so far out, you can still do your thing. If you're a startup, you can get time to value pretty quickly with the cloud. So again, I still think it's very healthy. In my opinion, I kind of think you guys have good analysis on that point. >> I think we're going to see some really cool stuff happen working together, and especially from what I'm seeing from IBM, in the fact that in the IT crowd, there is a behavioral change that's happening that Hadoop opened the door to. That we're starting to see more and more It professionals walk through. In the sense that, Hadoop has opened the door to not thinking of data as a liability, but actually thinking about data differently as an asset. And I think this is where this market does have an opportunity to continue to grow as long as we don't get carried away with trying to solve all of the old problems that we solved for on-premise data management. Like if we do that, then we're just, then there will be a consolidation. >> Metadata is a huge issue. I think that's going to be a big deal. And on the M and A, my feeling on the M and A is that, you got to buy something of value, so you either have revenue, which means customers, and or initial property. So, in a market of open source, it comes back down to the valuation question. If you're IBM or Oracle or HP, they can pivot too. And they can be agile. Now slower agile, but you know, they can literally throw some engineers at it. So if there's no customers in I and P, they can replicate, >> Exactly. >> That product. >> And we're seeing IBM do that. >> They don't know what they're buying. My whole point is if there's nothing to buy. >> I think it depends on, ultimately it depends on where we see people deriving value, and clearly in WANdisco, there's a huge amount of value that we're seeing our customers derive. So I think it comes down to that, and there is a lot of IP there, and there's a lot of IP in a lot of these companies. I think it's just a matter of widening their view, and I think WANdisco is probably the earliest to do this frankly. Was to recognize that for them to succeed, it couldn't just be about Hadoop. It actually had to expand to talk about cloud and talk about other data environments, right? >> Well, congratulations on the OEM deal. IBM, great name, Big Replicate. Love it, fantastic name. >> We're excited. >> It's a great product, and we've been following you guys for a long time, David. Great product, great energy. So I'm sure there's going to be a lot more deals coming on your. Good strategy is OEM strategy thing, huh? >> Oh yeah. >> It reduces sales cost. >> Gives us tremendous operational leverage. Getting 4,000, 5,000-- >> You get a great partner in IBM. They know the enterprise, great stuff. This is theCUBE bringing all the action here at Hadoop. IBM OEM deal with WANdisco all happening right here on theCUBE. Be back with more live coverage after this short break.

Zay California in the heart of Silicon Valley. It's the cube covering Hadoop summit 2016 brought to you by Hortonworks. Here's your host, John furrier. >>Okay, we'll come back here and we are here live in Silicon Valley for the cube. This is our flagship program. We go out to the events and extract the cylinders. Of course. We're here at the big data event. Hadoop summit 2016 have a special guest celebrity now, author of the bestselling book magical at Virginia Heffernan magic and loss rising on the bestseller lists. Welcome to the cube. Thanks in our show, you are my internet friend and now you're my real life friend. You're my favorite Facebook friend that I just now met for the first time. Great to meet you. We had never met and now we, but we know each other of course intimately through the interwebs. So I've been following your writing your time. Send you do some stuff on medium and then you, you kind of advertise. You're doing this book. I saw you do the Google glasses experiment in. >>It was Brooklyn and it might, it was so into Google glass and I will admit it, I fought for everything. I fell for VR and all its incarnations and um, and the Google last year, it was like that thing that was supposed to put the internet all voice activated, just put the internet always in front of your face. So I started to wear it around in Brooklyn, my prototype. I thought everyone would stop me and say how cool it was. In fact they didn't think it was pull it off new Yorkers. That's how you would, how they really feel. Got a problem with that. Um, your book magic and loss is fantastic and I think it really is good because uh, Dan Lyons wrote, disrupted, loved, which was fantastic. Dan lies big fan of him and his work, but it really, it wasn't a parody of civil rights for Silicon Valley. >>The show that's kinda taken that culture and made it mainstream. I had people call me up and say, Hey, you live in Callow Alto. My God, do you live near the house? Something like it's on Newell, which is one of my cross streets. But the point is tech culture now is kind of in a native, my youngest is 13 and you know, we're in an iPad generation for the youth and we're from the generation where there was no cell phones. And Mike, I remember when pages were the big innovation and internet. But I think, I think when I'm telling you, I think, I know I'm talking to a fellow traveler when I say that there was digital culture before the advent of the worldwide web in the early nineties you know, I, I'm sure you did too. Got electronic games like crazy. I would get any Merlin or Simon or whatever that they, they introduced. >>And then I also dialed into a mainframe in the late seventies and the early eighties to play the computer as we call it. We didn't even call it the internet. And the thing about the culture too was email was very, you know, monochrome screens, but again, clunky but still connected. Right? So we were that generation of, you know, putting that first training wheels on and now exposed to you. So in the book, your premise is, um, there's magical things happening in the internet and art countering the whole trolling. Uh, yeah, the Internet's bad. And we know recently someone asked me, how can the internet be art when Twitter is so angry? What do you think art is? You know, this is an art. Art is emotional. Artists know powerful >>emotions represented in tranquility and this is, you know, what you see on the internet all the time. Of course the aid of course are human. It needs a place to live and call it Twitter. For now it used to be YouTube comments. So, but we are always taking the measure of something we've lost. Um, I get the word loss from lossy compression, you know, the engineering term that, how does, how MP3 takes that big broad music signal and flattens it out. And something about listening to music on MP3, at least for me, made me feel a sense that I was grieving for something. It was missing something from my analog life. On the other hand, more than counterbalanced by the magic that I think we all experienced on the internet. We wouldn't have a friendship if it weren't for social media and all kinds of other things. And strange serendipity happens not to mention artistic expression in the form of photography, film, design of poetry and music, which are the five chapters of the book. >>So the book is fantastic. The convergence and connection of people, concepts, life with the internet digitally is interesting, right? So there's some laws with the MP3. Great example, but have you found post book new examples? I'm sure the internet culture, geese like Mia, like wow, this is so awesome. There's a cultural aspect of it is the digital experience and we see it on dating sites. Obviously you see, you know Snapchat, you know, dating sites like Tinder and other hookups apps and the real estate, everything being Uberized. What's the new things that you've, that's coming out and you must have some >>well this may be controversial, but one thing I see happening is anti digital culture. Partly as an epi phenomenon of side effect of digitization. We have a whole world of people who really want to immerse themselves in things like live music maker culture, things made by hand, vinyl records, vinyl records, which are selling more than ever in the days of the rolling stones. Gimme shelter less they sold less than than they do now. The rolling stones makes $1 billion touring a year. Would we ever have thought that in the, in the, you know, at the Genesis of the iPod when it seemed like, you know, recorded music represented music in that MP3 thing that floated through our, our phones was all we needed. No, we want to look in the faces of the rolling stones, get as close as we can to the way the music is actually made and you know, almost defiantly, and this is how the culture works. This is how youth culture works. Um, reject, create experiences that cannot be digitized. >>This is really more of a counter culture movement on the overt saturation of digital. >>Yes. Yes. You see the first people to scale down from, you know, high powered iPhones, um, when we're youth going to flip phones. You know, it's like the greatest like greatest punk, punk, punk tech. Exactly. It's like, yeah, I'm going to use these instruments, but like if I break a string, who cares on a PDs? The simplest one, right? >>My mom made me use my iPhone. Are we going to, how are we going to have that? it'd >>be like, Oh, look at you with your basic iPhone over there. And I've got my just like hack down, downscale, whatever. And you know what, I don't spend the weekends, don't pick up my phone on the weekends. But you know, there are interesting markets there. And interesting. I mean, for instance, the, you know, the live phenomenon, I know that, you know, there's this new company by one of the founders of Netflix movie pass, which um, for a $30 subscription you've seen movies in the theater as much as you want and the theaters are beautiful. And what instead of Netflix and chill, you know, the, the, the contemporary, you know, standard date, it's dinner and movie. You're out again. You're eating food, which can't be digitized with in-company, which can't be digitized. And then sitting in a theater, you know, a public experience, which is, um, a pretty extraordinary way that the culture and business pushes back on digital. >>Remember I was a comma on my undergraduate days in computer science in the 80s. And before when it was nerdy and eh, and there was a sociology class at Hubba computers and social change. And the big thing was we're going to lose social interactions because of email. And if you think about what you're talking about here is that the face to face presence, commitment of being with somebody right now is a scarce resource. You have an abundance of connections. >>I mean, take the fact what has happened is digital culture has jacked up the value of undigital culture. So for instance, you know, I've, I've met on Facebook, we talk on Facebook messenger, we notice that we're, you know, like kindred spirits in a certain way and we like each other's posts and so forth. Then we have an, a more extensive talk in messenger when we meet in person for the first time. Both of us are East coast people, but we hugged tele because it's like, Oh wow, like you in the flesh. You know it's something exciting. >>Connection virtually. That's right. There's a synchronous connection presence, but we're not really, we haven't met face to face. >>Yeah, there's this great as a great little experiment going on, set group of kids and Silicon Valley have decided they're too addicted to their phones and Facebook. Now I am not recommending for your viewers and listeners that anybody do what these kids sounds good, are ready. Go. Hey, all right, so what they do is take an LSD breakfast. Now I don't take drugs. I think you can do this without the LSD, but they put a little bit of a hallucinogen under their skin in the morning and what they find is they lost interest in the boring interface in their phones because people on the bus suddenly looked so fascinating to them. The human face is an ratable interface. It can't be reproduced anywhere, Steve. You know, Johnny ive can't make it. They can't make it at Google. And that I think is something we will see young markets doing, which is this renewed appreciation for nature and analog for humans and for analog culture. >>That's right. The Navy is going to sextants and compasses. You may have seen training, they're training sailors on those devices because of the fear that GPS might be hacked. So you know, the young kids probably don't even know what a cup is is, well, I bought myself a compass recently because I suddenly was like, you know, we talk a lot about digital technology, but what the heck, this thing you can point toward the poles, right in my hands. You know, I was suddenly like, we are this floating ball with these poles with different magnetic charges. And I think it's time. I appreciate it. >>Okay, so I've got to ask the, um, the, the feedback that you've gotten from the book, um, again, we hear that every Geneva magic and loss, great, great book. Go by. It's fantastic and open your mind up. It's a, it's a thought provoking, but really specific good use cases. I got a think that, you know, when you talk at Google and when you talk to some of the groups that you're talking to, certainly book clubs and other online that there must be like, Oh my God, you hit the cultural nerve. What have you heard from some of these, um, folks from my age 50 down to the 20 something year olds? Have you had any aha moments where you said, Oh my God, I hit a nerve here. >>Did not want to, I mean, I didn't want to write one of those books. That's like the one thing you need to know to get your startup to succeed or whatever. You know, I was at the airport and every single one of them is like, pop the only thing you need to do to save this or whatever. And they, they do take a very short view. Now if you're thinking about, you know, whether if you're thinking about your quarterly return or your, you know, what you're going to do this quarter and when you're going to be profitable or user acquisition, those books are good manuals. But if you're going to buy a hardcover book and you're going to really invest in reading every page, not just the bolded part, not just the put, you know, the two points that you have to know. I really wanted readers and at what I had found on the internet, people like you, we have an interest in a long view. You know what, I need a really long view >>in a prose that's not for listicle or you know, shorts. It's like it's just a thought provoker but somebody can go, Hey, you know, at the beach on the weekend say, Hey wow, this is really cool. What F you know, we went analog for awhile or what if, what's best for my kids to let my kids play multiplayer games more Zika simulate life. That was my, so these are the kinds of questions that the digital parents are asked. >>Yeah. So you know, like let's take the parents question, which is, is, you know, a, surprisingly to me it's a surprisingly pressing question. I am a parent, but my kids' digital habits are not, you know, of obsessive interest to me. Sometimes I think the worry about our kids is a proxy for how we worry about ourselves. You know, it's funny because they're the, you know, the model of the parent saying my kid has attention deficit order, zero order. My kid has attention deficit disorder. The kids over here, the parents here, you know, who has the attention deficit disorder. But in any case I have realized that parents are talking about uh, computers on the internet as though something kids have to have a very ambivalent relationship with and a very wary relationship with. So limit the time, and it sounds a little bit like the abstinence movement around sexuality that like, you know, you only dip in, it's very, you know, they're only date, right, right, right. >>Instead of joining sides with their kids and helping to create a durable, powerful, interesting online avatar, which is what kids want to do. And it's also what we want to do. So like in your Facebook profile, there are all kinds of strategic groups you can make as a creator of that profile. We know it as adults. Like, do you, some people put up pictures of their kids, some people don't vacation pictures. Some people promote the heck out of themselves. Some people don't do so much of that. Um, do you put up a lot of photographs? Do whatever. Those are the decisions we started to make when went on Facebook at kitchen making the two small armor to have on their gaming profile. That's kind of how they want to play, you know, play for you, going to wear feathers. These are important things. Um, but the uh, you know, small questions like talking to your kids and I don't mean a touchy feely conversation, but literally during the write in all lower case commit, you know, Brighton, all lower case, you're cute and you're this and that means a certain thing and you should get it and you're going to write in all caps and you're going to talk about white nationalist ideology. >>Well that also has a set of consequences. What have you learned in terms of the virtual space? Actually augmented reality, virtual reality, these promise to be virtual spaces. What, what is one of them? They always hope to replicate the real world. The mean, yes. Will there be any parallels of the kind of commitment in the moment? Gives you one thing. I say kids that, you know, the subtitle of the book is the internet as art, magic and loss. The internet is art and the kind of art, the internet is, is what I think of as real estate art. It purports to be reality. You know, every technology pick a photography film says or think of even the introduction of a third dimension in painting, you know, in Renaissance painting perspective for ports to represent reality better than it's been represented before. And if you're right in sync with the technology, you're typically fooled by it. >>I mean, this is a seductive representation of reality. You know, people watching us now believe they're seeing us flush of let us talk. You know, they don't think they're seeing pixels that are designed in certain ways and certainly it's your ways. So trying to sort out the incredibly interesting immersive, artful experience of being online that has some dangers and has some emotions to do it from real life is a really important thing. And you know, for us to learn first and then a model for our kids. So I had a horrible day on Twitter one day, eight 2012 213 worst day ever on Twitter. It was a great day for me. I spent the day at the beach, my Twitter avatar took sniper fire for me all day. People called her an idiot separated amount. I separated them out. And anyone who like likes roleplay and games knows that like I'm not a high priestess in Dentons and dragons. >>You know, I'm a much smaller person than that. And in, in, you know, in the case of this Twitter battle, I'm a less embattled person than the one that takes your armor from me on Twitter. That's my art. Your armor. So let's talk about poetry. Twitter, you mentioned poetry, Twitter, 140 characters. I did 40 characters is a lot. If like a lot of internet users your to have pictographic language like Chinese. So 140 characters is a novel by, well not a novel, but it's a short story for, you know, a writer of short form, short form Chinese aphorisms like Confucius. So one of the things I wanted to say is there's nothing about it being short that makes it low culture. You know, there's, I mean it takes a second to take, to take an a sculpture or to take an a painting and yet like the amount of craft that went into that might be much more good tweeting and you're excellent at it, um, is not easy. You know, I know that times I've been like, I tagged the wrong person and then I have to delete it. Like, because the name didn't come up or you know, I get the hashtags wrong and then I'm like, Oh, it would have been better this other way or I don't have a smart enough interject >>it's like playing sports. Twitter's like, you know, firing under the tennis ball baseline rallies with people. I mean, it's like, it's like there's a cultural thing. And this is the thing that I love about your book is you really bring in the metaphors around art and the cultural aspect. Have you had any, have you found that there's one art period that we represent right now? That it could be a comparison? >>I mean, you know, it's always tempting to care everything to the Renaissance. But you know, obviously in the Italian Renaissance there was so much technological innovation and so much, um, and so much, uh, so much artistic innovation. But, um, you know, the other thing are the Dawn of it's might be bigger than that, which it sounds grounds grandiose, but we're talking about something that nearly 6 billion people use and have access to. So we're talking about something bigger than we've ever seen is the Donovan civilization. So like, we pay a lot of attention to the Aqua docks and Rome and, and you know, later pay to touch it to the frescoes. I attend in this book to the frescoes, to the sculpture, to the music, to the art. So instead of talking about frescoes as an art historian, might I talk about Instagram? Yeah. >>And you, and this thing's all weave together cause we can back to the global fabric. If you look at the civilization as you know you're not to use the world is flat kind of metaphor. But that book kind of brings out that notion of okay if you just say a one global fabric, yes you have poetry, you have photography of soiling with a Johnny Susana ad in London. He says, you know, cricket is a sport in England, a bug and a delicacy depending on where in the world you are. >>Love that is that, I wonder if that's the HSBC had time to actually a beautiful HSBC job has done a beautiful campaign. I should find out who did it about perspective. And that is also a wonderful way to think about the internet because you know, I know a lot of people who don't like Twitter, who don't like YouTube comments. I do like them because I am perpetually surprised at what people bring to their interpretation. Insights in the comments can be revealing. You know, you know, you don't wanna get your feelings hurt. Sometimes you don't want that much exposure to the micro flora and fauna of ideas that could be frightening. But you know, when you're up for it, it's a really nice test of your immune system, you know. All right. So what's next for you? Virginia Heffernan magic and last great book. I think I will continue to write the tech criticism, which is just this growing field. I at Sarah Watson had a wonderful piece today in the Columbia journalism review about how we really need to bring all our faculties to treat, treating to tech criticism meant and treating tech with, um, with Karen, with proper off. Um, and the next book is on anti digital culture. Um, I will continue writing journalism and you'll see little previews of that book in the next work. >>Super inspirational. And I think the culture needs this kind of rallying cry because you know, there is art and science and all this beautiful beauty in the internet and it's not about mutually exclusive analog world. You can look and take, can come offline. So it's interesting case study of this, this revolution I think, and I think the counter culture, if you'd go back and Southern John Markoff about this, when he wrote his first book, the Dormouse wander about the counter culture in Silicon Valley is what's your grade book? And counter cultures usually create a another wave of innovation. So the question that comes out of this one is there could, this could be a seminal moment in history. I mean, I think it absolutely is. You know, in some ways, every moment is a great moment if you know what to make of it. But I am just tired of people telling us that we're ruining our brands and that this is the end of innovation and that we're at some low period. >>I think we will look back and think of this as an incredibly fertile time for our imaginations. If we don't lose hope, if we keep our creativity fired and if we commit to this incredible period we're in Virginia. Thanks for spending the time here in the queue. Really appreciate where you're live at. Silicon Valley is the cube with author Virginia Heffernan magic. And loss. Great book. Get it. If you don't have it, hard copies still available, get it. We'll be right back with more live coverage here. This is the cube. I'm John furry right back with more if the short break.

>> Narrator: Live from Dublin, Ireland. It's theCUBE, covering Hadoop Summit Europe 2016. Brought to you by Hortonworks. Now your host, John Furrier and Dave Vellante. >> Okay, welcome back here, we are here live in Dublin, Ireland, it's theCUBE SiliconANGLEs flagship program where we go out to the events and extract the signal from the noise, I'm John Furrier, my cohost Dave Vellante, our next guest is Emer Coleman who's with Disruption Limited, Open Data Governance Board in Ireland and Transport API, a growing startup built self-sustainable, growing business, open data, love that keynote here at Hadoop Summit, very compelling discussion around digital goods, digital future. Emer, welcome to theCUBE. >> It's great to be here. >> So what was your keynote? Let's just quickly talk about what you talked about, and then we can get in some awesome conversation. >> Sure. So the topic yesterday was we need to talk about techno ethics. So basically, over the last couple of months, I've been doing quite a lot of research on ethics and technology, and many people have different interpretations of that, but yesterday I said it's basically about three things. It's about people, it's about privacy, and it's about profits. So it's asking questions about how do we look at holistic technology development that moves away from a pure technocratic play and looks at the deep societal impacts that technology has. >> One of the things that we're super excited about and passionate about is this new era of openness going to a whole another level. Obviously, open source tier one software development environment, cloud computing allows for instant access to resources, almost limitless at this point, as you can project it forward with Moore's Law and whatnot. But the notion that digital assets are not just content, it's data, it's people, it's the things you mentioned about, create a whole new operating environment or user experience, user expectations with mobile phones and Internet of Things and Transport API which you have, if it moves, you capture it, and you're providing value there. So a whole new economy is developing around digital capital. Share your thoughts around this, because this is an area that you're passionate about, you've just done work here, what's your thoughts on this new digital economy, digital capital, digital asset opportunity? >> I think there's huge excitement about the digital economy, isn't there? And I think one of the things I'm concerned about is that that excitement will lead us to the same place that we are now, where we're not really thinking through what are the equitable distribution in that economy, because it seems to me that the spoils are going to a very tiny elite at the tops. So if you look at Instagram, 13 employees when it was purchased by Facebook for a billion dollars, but that's all our stuff, so I'm not getting any shares in the billion, those 13 people are. That's fantastic that you can build a business, build it to that stage and sell, but you have to think about two things, really: what are we looking at in terms of sustainable businesses into the future that create ethical products, and also the demands from citizens to get some value for their data back, because we're becoming shadow employees, we're shadow employees of Google, so when we email, we're not just corresponding, we're creating value for that company. >> And Facebook is a great example. >> And Facebook, and the thing is, when we were at the beginning of that digital journey, it was quite naive. So we were very seduced by free, and we thought, "This is great," and so we're happy with the service. And then the next stage of that, we realize what if we're not paying for the service, we're the product? >> John: Yeah. >> But we were too embedded in the platform to extricate ourselves. But now, I think, when we look at the future of work and great uncertainty that people are facing, when their labor's not going to be required to the same degree, are we going to slavishly keep producing capital and value for companies like Google, and ask for nothing more than the service in return? I don't think so. >> And certainly, the future will be impacted, and one of the things we see now in our business of online media and online open data, is that the data's very valuable. We see that, I'll say data is the new capital, new oil, whatever phrases of the day is used, and the brand marketers are the first ones to react to it, 'cause they're very data driven. Who are you, how do I sell stuff to you? And so what we're seeing is, brand marketers are saying, "Hey, I'm going to money to try to reach out to people, "and I'm going to activate that base and connect with, "engage with them on Facebook or other platform. "I'm going to add value to your Facebook or Google platform, "but yet I'm parasitic to your platform for the data. "Why just don't I get it directly?" So again, you're starting to see that thinking where I don't want to be a parasite or parasitic to a network that the value's coming from. The users have not yet gotten there, and you're teasing that out. What's your thoughts there, progression, where we're at, have people realized this? Have you seen any movement in the industry around this topic? >> No, I think there's a silence around... Technology companies want to get all the data they can. They're not going to really declare as much as they should, because it bends their service model a bit. Also, the data is emergent. Zuckerberg didn't start Facebook as something that was going to be a utility for a billion people, he started it as a social network for a university. And what grew out of that, we learned as we went along. So I'm thinking, now that we have that experience, we know that happens, so let's start the thinking now. And also, this notion of just taking data because you can, almost speculatively getting data at the point of source, without even knowing what you want it for but thinking, "I'm going to monetize this in the end." Jaron Lanier in his book Who Owns The Future talks about micro licensing back content. And I think that's what we need to do. We start, at the very beginning, we need to start baking in two things: privacy by design and different business models where it's not a winner takes all. It's a dialog between the user and the service, and that's iterated together. >> This idea that it's not a zero sum game is very important, and I want to go back to your Instagram and Facebook example. At its peak, I think Eastman Kodak had hundreds of thousands of employees, maybe four or five hundred, 450,000 employees, huge. Facebook has many many more photos, but maybe a few thousand employees? Wow, so all the jobs are gone, but at the same time, we don't want to be protecting the past from the future, so how do you square that circle? >> Correct, but I think what we know is that the rise of robotics and software is going to eat jobs, and basically, there's going to be a hollowing out of the middle class. You know, for sure, whether it's medicine, journalism, retail, exactly. >> Dave: It's not future, it's now. (laughs) >> Exactly. So we maybe come into a point where large swaths of people don't have work. Now, what do you do in a world where your labor is no longer required? Think about the public policy implications of that. Do we say you either fit in this economy or you die? Are we going to look at ideas which they are looking at in Europe, which is like a universal wage? And all of these things are a challenge to government, because they're going to have a citizenry who are not included in this brave new world. So some public policy thinking has to go into what happens when our kids can't get jobs. When the jobs that used to be done by people like us are done by machines. I'm not against the movement of technology, what I'm saying is there are deep societal implications that need some thinking, because if we get to a point where we suddenly realize, if all of these people who are unemployed and can't get work, this isn't a future we envisioned where robots would take all the crap jobs and we would go off to do wonderful things, like how are we going to bring the bacon home? >> It seems like in a digital world that the gap is creativity to combine technologies and knowledge. I find that it's scary when you talk about maybe micromanaging wages and things like that, education is the answer, but that's... How do you just transfer that knowledge? That's sort of the discussion that we're having in the United States anyway. >> I think some of the issue is that the technology is so, we're kind of seduced by simplicity. So we don't see the complexity underneath, and that's the ultimate aim of a technology, is to make something so simple, that complexity is masked. That's what the iPhone did wonderfully. But that's actually how society is looking now. So we're seduced by this simplicity, we're not seeing the complexity underneath, and that complexity would be about what do we do in a world where our labor is no longer required? >> And one of the things that's interesting about the hollowing of the middle class is the assumption is there's no replacements, so one of the things that could be counter argued is that, okay, as the digital natives, my daughter, she's a freshman in high school, my youngest son's eighth grade, they're natives now, so they're going to commit. So what is the replacement capital and value for companies that can be sustained in the new economy versus the decay and the darwinism of the old? So the digital darwinism aspect's interesting, that's one dilemma. The other one is business models, and I want to get your thoughts on this 'cause this is something we were teasing out with this whole value extraction and company platform issue. A company like Twitter. Highly valuable company, it's a global network of people tweeting and sharing, but yet is under constant pressure from Wall Street and investors that they basically suck. And they don't, they're good, people love Twitter, so they're being forced to behave differently against their mission because their profit motive doesn't really match maybe something like Facebook, so therefore they're instantly devalued, yet the future of someone connecting on Twitter is significantly high. That being said, I want to get your thoughts on that and your advice to Twitter management, given the fact it is a global network. What should they do? >> It's the same old capitalism, just it's digital, it's a digital company, it's a digital asset. It's the same approach, right? Twitter has been a wonderful thing. I've been a Twitter user for years. How amazing, it's played a role in the Arab Spring, all sorts of things. So they're really good, but I think you need as a company, so for example, in our company, in Transport API, we're not really looking to build to this massive IPO, we're trying to build a sustainable company in a traditional way using digital. So I think if you let yourself be seduced by the idea of phenomenal IPO, you kind of take your eye off the ball. >> Or in case this, in case you got IPOed, now you're under pressure to produce-- >> Emer: Absolutely, yeah. >> Which changes your behavior. But in Twitter's management defense, they see the value of their product. Now, they got there by accident and everyone loves it, but now they're not taking the bait to try to craft a short term solution to essentially what is already a valuable product, but not on the books. >> Yes, and also I think where the danger is, we know that their generation shifts across channel. So teenagers probably look at Facebook, I think one of them said, like an awkward family dinner they can't quite leave. But for next gen, they're just not going to go there, 'cause that's where your grandmother is. So the same is true of Twitter and Snapchat, these platforms come and go. It's an interesting phenomenon then to see Wall Street putting that much money into something which is essentially quite ephemeral. I'm not saying that Twitter won't be around for years, it may be, but that's the thing about digital, isn't it? Something else comes in and it's well, that becomes the platform of choice. >> Well, it's interesting, right? Everybody, us included, we criticize the... Michael Dell calls it the 90 day shock clock. But it's actually worked out pretty well, I mean, economically, for the United States companies. Maybe it doesn't in the future. What are your thoughts on that, particularly from a European perspective? Where you're reporting maybe twice a year, there's not as much pressure, but yet from a technology industry standpoint, companies outside the Silicon Valley in particular seem to be less competitive, why? >> For example, in our company, in Transport API, we've got some pretty heavyweight clients, we have a wonderful angel investor who has given us two rounds of investment. And it isn't that kind of avaricious absolutely built this super price. And that's allowed us to build from starting off with 2, now to a team of 10, and we're just about coming into break even, so it's doable. But I think it's a philosophy. We didn't want necessarily to build something huge, although we want to go global, but it was let's do this in a sustainable way with reasonable wages, and we've all put our own soul and money into it, but it's a different cultural proposition, I think. >> Well, the valuations always drive the markets. It's interesting too, to your point about things come and go channels, kind of reminds me, Dave and I used to joke about social networks like nightclubs, they're hot and then it's just too crowded and nobody goes there, as Yogi Bear would say. And then they shift and they go out of business, some don't open with fanfare, no one goes 'cause it's got different context. You have a contextual challenge in the world now. Technology can change things, so I want to ask you about identity 'cause there was a great article posted by the founder of the company called Secret which is one of these anonymous apps like Yik Yak and whatnot, and he shut it down. And he wrote a post, kind of a postmortem, saying, "These things come and go, they don't work, "they're not sustainable because there's no identity." So the role of identity in a social global virtual world, virtual being not just virtual reality, is interesting. You live in a world, and your company, Transport API, provides data which enables stuff and the role of identity. So anonymous versus identity, thoughts there, and that impact to the future of work? If you know who you're dealing with, and if they're present, these are concepts that are now important, presence, identity, attention. >> And that's the interesting thing, isn't it? Who controls that identity? Mark Zuckerberg said, "You only have one identity," which is what he said when he set up Facebook. You think, really? No, that's what a young person thinks. When we're older, we know. >> He also said that young people are smarter than older people. >> Yeah, right, okay. (John laughs) He could be right there, he could be right there, but we all have different identities in different parts of our lives. Who we are here, the Hadoop summit is different from what we're at home to when we're with friends. So identity is a multifaceted thing. But also, who gets to determine your identity? So I have 16 years of my search life and Google. Now, who am I in that server, compared to who I am? I am the sum total of my searches. But I'm not just the sum total of my searches, am I? Or even that contextualized, so I'll give you an example. A number of years ago I was searching for a large, very large waterproof plastic bag. And I typed it in, and I thought, "Oh my god, that sounds like I'm going to murder my husband "and try to bury him." (John and Dave laugh) It was actually-- >> John: Into the compost. >> Right, right. And I thought, "Oh my god, what does this look like "on the other side?" Now, it was actually for my summer garden furniture. But the point is, if you looked at that in an analytic way, who would I be? And so I think identity is very, you know-- >> John: Mistaken. >> Yeah, and also this idea of what Frank Pasquale calls the black box society. These secret algorithms that are controlling flows of money and information. How do they decide what my identity is? What are the moral decisions that they make around that? What does it say if I search for one thing over another? If I search constantly for expensive shoes, does that make me shallow? What do these things say? If I search for certain things around health. >> And there's a value judgment now associated with that that you're talking about, that you do not control. >> Absolutely, and which is probably linked to other things which will determine things like whether I get credit or not, but these can almost be arbitrary decisions, 'cause I have no oversight of the logic that's creating that decision making algorithm. So I think it's not just about identity, it's about who's deciding what that identity is. >> And it's also the reality that you're in, context, situations. Dark side, bright side of technology in this future where this new digital asset economy, digital capital. There's going to be good and bad, education can be consumed non-linear, new forms of consumptions, metadata, as you're pointing out, with the algorithms. Where do you see some bright spots and where do you see the danger areas? >> I think the great thing is, when you were saying software is the future. It's our present, but it's going to be even more so in our future. Some of the brightest brains in the world are involved in the creation of new technology. I just think they need to be focusing a bit more of that intellectual rigor towards the impact they're having on society and how they could do it better. 'Cause I think it's too much of a technocratic solution. Technologists say, "We can do this." The questions is, should they? So I think what we need to do is to loop them back into the more social and philosophical side of the discussion. And of course it's a wonderful thing, hopefully technology is going to do amazing things around health. We can't even predict how amazing it's going to be. But all I'm saying is that, if we don't ask the hard questions now about the downsides, we're going to be in a difficult societal position. But I'm hoping that we will, and I'm hoping that raising issues like techno ethics will get more of that discussion going. >> Well, transparency and open data make a big difference. >> Emer: Absolutely. >> Well, and public policy, as you said earlier, can play a huge role here. I wonder if you could give us your perspective on... Public policy, we're in the US most of the time, but it's interesting when we talk to customers here. To hear about the emphasis, obviously, on privacy, data location and so forth, so in the digital world, do you see Europe's emphasis and, I think, leading on those types of topics as an advantage in a digital world, or does it create friction from an economic standpoint? >> Yeah, but it's not all about economics. Friction is a good thing. There are some times when friction is a good thing. Most technologists think all friction is bad. >> Sure, and I'm not implying that it's necessarily good or bad, I'm curious though, is it potentially an economic advantage to have thought through and have policy on some of those issues? >> Well, what we're seeing here-- >> Because I feel like the US is a ticking time bomb on a lot of these issues. >> I was talking to VCs, some VC friends of mine here in the UK, and what they said they're seeing more and more, VCs asking what we call SMEs, small to medium enterprises, about their data policies, and SMEs not being able to answer those questions, and VCs getting nervous. So I think over time it's going to be a competitive advantage that we've done that homework, that we're basically not just rushing to get more users, but that we're looking at it across the piece. Because, fundamentally, that's more sustainable in the longer term. People will not be dumb too forever. They will not, and so doing that thinking now, where we work with people as we create our technology products, I think it's more sustainable in the long term. When you look at economics, sustainability is really important. >> I want to ask you about the Transport API business, 'cause in the US, same thing, we've seen some great openness of data and amazing innovations that have come out of nowhere. In some cases, unheard of entrepreneurs and/or organizations that better society for the betterment of people, from delivering healthcare to poor areas and whatnot. What has been the coolest thing, or of things you've seen come out of your enablement of the transport data. Use cases, have you seen any things that surprised you? >> It's quite interesting, because when I worked for the mayor of London as his director of digital projects, my job was to set up the London data store, which was to open all of London's public sector data. So I was kind of there from the beginning as a lobbyist, and when I was asking agencies to open up their data, they'd go, "What's the ROI?" And I'd just say, "I don't know." Because government's one and oh, I'm saying that was a chicken and egg, you got to put it out there. And we had a funny incident where some of the IT staff in transport for London accidentally let out this link, which is to the tracker net feed, and that powers the tube notice boards that says, "Your next tube is in a minute," whatever. And so the developer community went, "Ooh, this is interesting." >> John: Candy! >> Yeah, and of course, we had no documentation with it because it kind of went out under the radar. And one developer called Mathew Somerville made this map which showed the tubes on a map in real time. And it was like surfacing the underground. And people just thought, "Oh my god, that is amazing." >> John: It's illuminating. >> Yeah. It didn't do anything, but it showed the possibility. The newspapers picked it up, it was absolutely brilliant example, and the guy made it in half a day. And that was the first time people saw their transport system kind of differently. So that was amazing, and then we've seen hundreds of different applications that are being built all the time. And what we're also seeing is integration of transport data with other things, so one of our clients in Transport API is called Toothpick, and they're an online dental booking agency. And so you can go online, you can book your dental appointment with your NHS dentist, and then they bake in transport information to tell you how to get there. So we have pubs using them, and screens so people can order their dinner, and then they say, "You've got 10 minutes till the next bus." So all sorts of cross-platform applications. >> That you never could've envisioned. >> Emer: Never. >> And it's just your point earlier about it's not a zero sum game, you're giving so many ways to create value. >> Emer: Right, right. >> Again, I come back to this notion of education and creativity in the United States education system, so unattainable for so many people, and that's a real concern, and you're seeing the middle class get hollowed out. I think the stat is, the average wage in the United States was 55,000 in 1999, it's 50,000 today. The political campaigns are obviously picking at that scab. What's the climate like in Europe from that standpoint? >> In terms of education? >> No, just in terms of, yes, the education, middle class getting hollowed out, the sentiment around that. >> I don't think people are up to speed with that yet, I really don't think that they're aware of the scale. I think when they think robots or automation, they don't really think software. They think robots like there were in the movies, that would come, as I say, and do those jobs nobody wanted. But not like software. So when I say to them, look, E-discovery software, when it's applied retrospectively, what it shows is that human lawyers are only 60% accurate compared to it. Now, that's a no-brainer, right? If software is 100% accurate, I'm going to use the software. And the ratio difference is 1 to 500. Where you needed 500 lawyers before you need 1. So I don't think people are across the scale of change. >> But it's interesting, you're flying to Heathrow, you fly in and out, you're dealing with a kiosk. You drive out, the billboards are all electronic. There aren't guys doing this anymore. So it's tangible. >> And I think, to your point about education, I'm not as familiar with the education system in the US, but I certainly think, in Europe and in the UK, the education system is not capable of dealing even with the latest digital natives. They're still structuring their classrooms in the same way. These kids, you know-- >> John: They have missed the line with the technology. >> Absolutely. >> So reading, writing and arithmetic, fine. And the cost of education is maybe acceptable. But they may be teaching the wrong thing. >> Asynchronous non-linear, is the thing. >> There's a wonderful example of an Indian academic called Sugata Mitra, who has a fabulous project called a Hole in the Wall. And he goes to non-English speaking little Indian villages, and he builds a computer, and he puts a roof over it so only the children can do it. They don't speak English. And he came back, and he leaves a little bit of stuff they have to get around before they can play a game. And he came back six months later, and he said to them, "What did you think?" And one of the children said, "We need a faster CPU and a better mouse." Now, his point is self-learning, once you have access to technology, is amazing, and I think we have to start-- >> Same thing with the non-linear consumption, asynchronous, all this, the API economy enabling new kinds of expectation and opportunities. >> And it was interesting because the example, some UK schools tried to follow his example. And six months later, they rang him up and they said, "It's not working," and he said, "What did you do?" And they said, "Well, we got every kid a laptop." He said, "That's not the point." The point was putting a scarce resource that the children had to collaborate over. So in order to get to the game, they had figure out certain things. >> I think you're right on some of these (mumbles) that no one's talking about. And Dave and I are very passionate on this, and we're actually investing in a whole new e-learning concept. But it's not about doing that laptop thing or putting courseware online. That's old workflow in a new model. Come on, old wine in a new bottle. So that's interesting. I want to get your thoughts, so a personal question to end this segment. What are you passionate about now, what are you working, outside of the venture, which is exciting. You have a lot of background going back to technology entrepreneurship, public policy, and you're in the front lines now, thought leading on this whole new wide open sea of opportunity, confusion, enabling it. What are you passionate about, what are you working on? Share with the folks that are watching. >> So one of the main things we're trying to do. I work as an associate with Ernst & Young in London. And we've been having discussions over the past couple of months around techno ethics, and I've basically said, "Look, let's see if we can get EY "to build to build an EY good governance index." Like, what does good governance look like in this space, a massively complex area, but what I would love is if people would collaborate with us on that. If we could help to draw up an ethical framework that would convene the technology industry around some ethical good governance issues. So that's what I'm going to be working on as hard as I can over the next while, to try and get as much collaboration from the community, because I think we'd be so much more powerful if the technology industry was to say, "Yeah, let's try and do this better "rather than waiting for regulation," which will come, but will be too clunky and not fit for purpose. >> And which new technology that's emerging do you get most excited about? >> Hmm. Drones. (laughter) >> How about anything with bitcoin, block chains? >> Absolutely, absolutely, block chain. Yeah, block chain, you have to say, yeah. I think, 'cause bitcoin, you know, it's worth 20 p today, it's worth 200,000 tomorrow. >> Dave: Yeah, but block chain. >> Right, right. I mean, that is incredible potentiality. >> New terms like federated, that's not a new term, but federation, universal, unification. These are the themes right now. >> Emer: Well, it's like the road's been coated, isn't it? And we don't know where it's going to go. What a time we live in, right? >> Emer Coleman, thank you so much for spending your time and joining us on theCUBE here, we really appreciate the conversation. Thanks for sharing that great insight here on theCUBE, thank you. It's theCUBE, we are live here in Dublin, Ireland. I'm John Furrier with Dave Vellante. We'll we right back with more SiliconANGLEs, theCUBE and extracting the signal from the noise after this short break. (bright music)

the cube and hadoop summit 2014 is brought to you by anchor sponsor Hortonworks we do have do and headline sponsor when disco we make hadoop invincible okay welcome back and when we're here at the dupe summit live is looking valance the cube our flagship program we go out to the events expect a signal from noise i'm john per year but Jeff Rick drilling down on the topics we're here with wind disco welcome welcome Brett room Stein about senior director tell us what's going on for you guys I'll see you at big presence here so all the guys last night you guys have a great great booth so causing and the crew what's happening yeah I mean the show is going is going very well what's really interesting is we have a lot of very very technical individuals approaching us they're asking us you know some of the tougher more technical in-depth questions about how our consensus algorithm is able to do all this distributor replication which is really great because there's a little bit of disbelief and then of course we get to do the demonstration for them and then suspend disbelief if you will and and I think the the attendance has been great for our brief and okay I always get that you always we always have the geek conversations you guys are a very technical company Jeff and I always comment certainly de volada and Jeff Kelly that you know when disco doesn't has has their share pair of geeks and that dudes who know they're talking about so I'm sure you get that but now them in the business side you talk to customers I want to get into more the outcome that seems to be the show focused this year is a dupe of serious what are some of the outcomes then your customers are talking about when they get you guys in there what are their business issues what are they tore what are they working on to solve yeah I mean I think the first thing is to look at you know why they're looking at us and then and then with the particular business issues that we solve and the first thing and sort of the trend that we're starting to see is the prospects and the customers that we have are looking at us because of the data that they have and its data that matters so it's important data and that's when people start to come to is that's when they look to us as they have data that's very important to them in some cases if you saw some of the UCI stuff you see that the data is you know doing live monitoring of various you know patient activity where it's not just about about about a life and monitoring a life but potentially about saving the life and systems that go down not only can't save lives but they can potentially lose them so you have a demos you want to jump into this demo here what is this all about you know the demo that the demonstration that I'm going to do for you today is I want to show you our non-stop a new product i'm going to show you how we can basically stand up a single HDFS or a single Hadoop cluster across multiple data centers and I think that's one of the tough things that people are really having trouble getting their heads wrapped around because most people when they do multi data center Hadoop they tend to do two different clusters and then synchronize the data between the two of them the way they do that is they'll use you know flume or they'll use some form of parallel ingest they'll use technologies like dis CP to copy data between the data centers and each one of those has sort of an administrative burden on them and then some various flaws in their and their underlying architecture that don't allow them to do a really really detailed job as ensuring that all blocks are replicated properly that no mistakes are ever made and again there's the administrative burden you know somebody who always has to have eyes in the system we alleviate all those things so I think the first thing I want to start off with we had somebody come to our booth and we were talking about this consensus algorithm that we that we perform and the way we synchronize multiple name nodes across multiple geographies and and again and that sort of spirit of disbelief I said you know one of the key tenants of our application is it doesn't underlie it doesn't change the behavior of the application when you go from land scope to win scope and so I said for example if you create a file in one data center and 3,000 miles apart or 7,000 miles apart from that you were to hit the same create file operation you would expect that the right thing happens what somebody gets the file created and somebody gets file already exists even if at 7,000 miles distance they both hit this button at the exact same time I'm going to do a very quick demonstration of that for you here I'm going to put a file into HDFS the my top right-hand window is in Northern Virginia and then 3,000 miles distance from that my bottom right-hand window is in Oregon I'm going to put the etsy hosts file into a temp directory in Hadoop at the exact same time 3,000 miles distance apart and you'll see that exact behavior so I've just launched them both and again if you look at the top window the file is created if you look at the bottom window it says file already exists it's exactly what you'd expect a land scope up a landscape application and the way you'd expect it to behave so that is how we are ensure consistency and that was the question that the prospect has at that distance even the speed of light takes a little time right so what are some of the tips and tricks you can share this that enable you guys to do this well one of the things that we're doing is where our consensus algorithm is a majority quorum based algorithm it's based off of a well-known consensus algorithm called paxos we have a number of significant enhancements innovations beyond that dynamic memberships you know automatic scale and things of that nature but in this particular case every transaction that goes into our system gets a global sequence number and what we're able to do is ensure that those sequence numbers are executed in the correct order so you can't create you know you can't put a delete before a create you know everything has to happen in the order that it actually happened occurred in regardless of the UN distance between data centers so what is the biggest aha moment you get from customer you show them the demo is it is that the replication is availability what is the big big feature focus that they jump on yeah I think I think the biggest ones are basically when we start crashing nodes well we're running jobs we separate the the link between the win and maybe maybe I'll just do that for you now so let's maybe kick into the demonstration here what I have here is a single HDFS cluster it is spanning two geographic territory so it's one cluster in Northern Virginia part of it and the other part is in Oregon I'm going to drill down into the graphing application here and inside you see all of the name notes so you see I have three name nodes running in Virginia three name nodes running in Oregon and the demonstration is as follows I'm going to I'm going to run Terrigen and Terra sort so in other words i'm going to create some data in the cluster I'm then going to go to sort it into a total order and then I'm going to run Tara validate in the alternate data center and prove that all the blocks replicated from one side to the other however along the way I'm going to create some failures I am going to kill some of that active name nodes during this replication process i am going to shut down the when link between the two data centers during the replication paris's and then show you how we heal from from those kinds of conditions because our algorithm treats failure is a first class citizen so there's really no way to deal in the system if you will so let's start unplug John I'm active the local fails so let's go ahead and run the Terrigen in the terrorists or I'm going to put it in the directory called cube one so we're creating about 400 megabytes of data so a fairly small set that we're going to replicate between the two data centers now the first thing that you see over here on the right-hand side is that all of these name nodes kind of sprung to life that is because in an active active configuration with multiple name nodes clients actually load balance their requests across all of them also it's a synchronous namespace so any change that I make to one immediately Curzon immediately occurs on all of them the next thing you might notice in the graphing application is these blue lines over and only in the Oregon data center the blue lines essentially represent what we call a foreign block a block that is not yet made its way across the wide area network from the site of ingest now we move these blocks asynchronously from the site of in jeff's oh that I have land speed performance in fact you can see I just finished the Terrigen part of the application all at the same time pushing data across the wide area network as fast as possible now as we start to get into the next phase of the application here which is going to run terrace sort i'm going to start creating some failures in the environment so the first thing I'm going to do is want to pick two named nodes I'm going to fail a local named node and then we're also going to fail a remote name node so let's pick one of these i'm going to pick HD p 2 is the name of the machine so want to do ssh hd2 and i'm just going to reboot that machine so as I hit the reboot button the next time the graphing application updates what you'll notice here in the monitor is that a flat line so it's no longer taking any data in but if you're watching the application on the right hand side there's no interruption of the service the application is going to continue to run and you'd expect that to happen maybe in land scope cluster but remember this is a single cluster a twin scope with 3,000 miles between the two of them so I've killed one of the six active named nodes the next thing I'm going to do is kill one of the name nodes over in the Oregon data center so I'm going to go ahead and ssh into i don't know let's pick the let's pick the bottom one HTTP nine in this case and then again another reboot operation so I've just rebooted two of the six name nose while running the job but if again if you look in the upper right-hand corner the job running in Oregon kajabi running in North Virginia continues without any interruption and see we just went from 84 to eighty eight percent MapReduce and so forth so again uninterruptedly like to call continuous availability at when distances you are playing that what does continuous availability and wins because that's really important drill down on yeah I mean I think if you look at the difference between what people traditionally call high availability that means that generally speaking the system is there there is a very short time that the system will be unavailable and then it will then we come available again a continuously available system ensures that regardless of the failures that happen around it the system is always up and running something is able to take the request and in a leaderless system like ours where no one single node actually it actually creates a leadership role we're able to continue replication we're and we're also able to continue the coordinator that's two distinct is high availability which everyone kind of know was in loves expensive and then continues availability which is a little bit kind of a the Sun or cousin I guess you know saying can you put in context and cost implementation you know from a from a from a from a perspective of a when disco deployment it's kind of a continuously available system even though people look at us as somewhat traditional disaster recovery because we are replicating data to another data center but remember it's active active that means both data centers are able to write at the same time you have you get to maximize your cluster resources and again if we go back to one of the first questions you asked what are what a customer's doing this with this what a prospects want to do they want to maximize their resource investment if they have half a million dollars sitting in another data center that only is able to perform an emergency recovery situation that means they either have to a scale the primary data center or be what they want to do is utilize existing resource in an active active configuration which is why i say continuous availability they're able to do that in both data centers maximizing all their resource so you versus the consequences of not having that would be the consequences of not being able to do that is you have a one-way synchronization a disaster occurs you then have to bring that data center online you have to make sure that all the appropriate resources are there you have to you have an administrative burden that means a lot of people have to go into action very quickly with the win disco systems right what that would look like I mean with time effort cost and you have any kind of order of magnitude spec like a gay week called some guy upside dude get in the office login you have to look at individual customer service level agreements a number that i hear thrown out very very often is about 16 hours we can be back online within 16 hours really RTO 44 when disco deployment is essentially zero because both sites are active you're able to essentially continue without without any doubt some would say some would say that's contingent availability is high available because essentially zero 16 that's 16 hours I mean any any time down bad but 16 hours is huge yeah that's the service of level agreement then everyone says but we know we can do it in five hours the other of course the other part of that is of course ensuring that once a year somebody runs through the emergency configure / it you know procedure to know that they truly can be back up in line in the service level agreement timeframe so again there's a tremendous amount of effort that goes into the ongoing administrating some great comments here on our crowd chatter out chat dot net / hadoop summit joined the conversation i'll see ya we have one says nice he's talking about how the system has latency a demo is pretty cool the map was excellent excellent visual dave vellante just weighed in and said he did a survey with Jeff Kelly said large portion twenty-seven percent of respondents said lack of enterprises great availability was the biggest barriers to adoption is this what you're referring to yeah this is this is exactly what we're seeing you know people are not able to meet the uptime requirements and therefore applications stay in proof-of-concept mode or those that make it out of proof of concept are heavily burdened by administrators and a large team to ensure that same level of uptime that can be handled without error through software configuration like Linda scope so another comment from Burt thanks Burt for watching there's availability how about security yeah so security is a good one of course we are you know we run on standard dupe distributions and as such you know if you want to run your cluster with on wire encryption that's okay if you want to run your cluster with kerberos authentication that's fine we we fully support those environments got a new use case for crowd chapel in the questions got more more coming in so send them in we're watching the crowd chat slep net / hadoop summit great questions and a lot of people aren't i think people have a hard time partial eh eh versus continues availability because you can get confused between the two is it semantics or is it infrastructure concerns what is what is the how do you differentiate between those two definitions me not I think you know part of it is semantics but but but also from a win disco perspective we like to differentiate because there really isn't that that moment of downtime there is there really isn't that switch over moment where something has to fail over and then go somewhere else that's why I use that word continuous availability the system is able to simply continue operating by clients load balancing their requests to available nodes in a similar fashion when you have multiple data centers as I do here I'm able to continue operations simply by running the jobs in the alternate data center remember that it's active active so any data ingest on one side immediately transfers to the other so maybe let me do the the next part I showed you one failure scenario you've seen all the nodes have actually come back online and self healed the next part of this I want to do an separation I want to run it again so let me kick up kick that off when I would create another directory structure here only this time I'm going to actually chop the the network link between the two data centers and then after I do that I'm going to show you some some of our new products in the works give you a demonstration of that as well well that's far enough Britain what are some of the applications that that this enables people to use the do for that they were afraid to before well I think it allows you know when we look at our you know our customer base and our prospects who are evaluating our technologies it opens up all the all the regulated industries you know things like pharmaceutical companies financial services companies healthcare companies all these people who have strict regulations auditing requirements and now have a very clear concise way to not only prove that they're replicating data that data has actually made its way it can prove that it's in both locations that it's not just in both locations that it's the correct data sometimes we see in the cases of like dis CP copying files between data centers where the file isn't actually copied because it thinks it's the same but there is a slight difference between the two when the cluster diverges like that it's days of administration hour depending on the size of the cluster to actually to put the cluster you know to figure out what went wrong what went different and then of course you have to involve multiple users to figure out which one of the two files that you have is the correct one to keep so let me go ahead and stop the van link here of course with LuAnn disco technology there's nothing to keep track of you simply allow the system to do HDFS replication because it is essentially native HDFS so I've stopped the tunnel between the two datacenters while running this job one of the things that you're going to see on the left-hand size it looks like all the notes no longer respond of course that's just I have no visibility to those nodes there's no longer replicating any data because the the tunnel between the two has been shut down but if you look on the right hand side of the application the upper right-hand window of course you see that the MapReduce job is still running it's unaffected and what's interesting is once I start replicating the data again or once i should say once i start the tunnel up again between the two data centers i'll immediately start replicating data this is at the block level so again when we look at other copy technologies they are doing things of the file level so if you had a large file and it was 10 gigabytes in size and for some reason you know your your file crash but in that in that time you and you were seventy percent through your starting that whole transfer again because we're doing block replication if you had seventy percent of your box that had already gone through like perhaps what I've done here when i start the tunnel backup which i'm going to do now what's going to happen of course is we just continue from those blocks that simply haven't made their way across the net so i've started the tunnel back up the monitor you'll see springs back to life all the name nodes will have to resync that they've been out of sync for some period of time they'll learn any transactions that they missed they'll be they'll heal themselves into the cluster and we immediately start replicating blocks and then to kind of show you the bi-directional nature of this I'm going to run Tara validate in the opposite data center over in Oregon and I'll just do it on that first directory that we created and in what you'll see is that we now wind up with foreign blocks in both sides I'm running applications at the same time across datacenters fully active active configuration in a single Hadoop cluster okay so the question is on that one what is the net net summarized that demo reel quick bottom line in two sentences is that important bottom line is if name notes fail if the wind fails you are still continuously operational okay so we have questions from the commentary here from the crowd chat does this eliminate the need for backup and what is actually transferring certainly not petabytes of data ? I mean you somewhat have to transfer what what's important so if it's important for you to I suppose if it was important for you to transfer a petabyte of data then you would need the bandwidth that support I transfer of a petabyte of data but we are to a lot of Hollywood studios we were at OpenStack summit that was a big concern a lot of people are moving to the cloud for you know for workflow and for optimization Star Wars guys were telling us off the record that no the new film is in remote locations they set up data centers basically in the desert and they got actually provisioned infrastructure so huge issues yeah absolutely so what we're replicating of course is HDFS in this particular case I'm replicating all the data in this fairly small cluster between the two sites or in this case this demo is only between two sites I could add a third site and then a failure between any two would actually still allow complete you know complete availability of all the other sites that still participate in the algorithm Brent great to have you on I want to get the perspective from you in the trenches out in customers what's going on and win disco tell us what the culture there what's going on the company what's it like to work there what's the guys like I mean we we know some of the dudes there cause we always drink some vodka with him because you know likes to tip back a little bit once in a while but like great guy great geeks but like what's what's it like it when disco I think the first you know you touched on a little piece of it at first is there are a lot of smart people at windows go in fact I know when I first came on board I was like wow I'm probably the most unsmoked person at this company but culturally this is a great group of guys they like to work very hard but equally they like to play very hard and as you said you know I've been out with cause several times myself these are all great guys to be out with the culture is great it's a it's a great place to work and you know so you know people who are who are interested should certainly yeah great culture and it fits in we were talking last night very social crowd here you know something with a Hortonworks guide so javi medicate fortress ada just saw him walk up ibm's here people are really sociable this event is really has a camaraderie feel to it but yet it's serious business and you didn't the days they're all a bunch of geeks building in industry and now it's got everyone's attention Cisco's here in Intel's here IBM's here I mean what's your take on the big guys coming in I mean I think the big guys realize that that Hadoop is is is the elephant is as large as it appears elephant is in the room and exciting and it's and everybody wants a little piece of it as well they should want a piece of it Brett thanks for coming on the cube really appreciate when discs are you guys a great great company we love to have them your support thanks for supporting the cube we appreciate it we right back after this short break with our next guest thank you

>>The queue at Hadoop summit, 2014 is brought to you by anchor sponsor Hortonworks. We do, I do. And headline sponsor when disco we make Hadoop invincible >>Okay. Welcome back. Everyone live here in Silicon valley in San Jose. This is a dupe summit. This is Silicon angle and Wiki bonds. The cube is our flagship program. We go out to the events and extract the signal to noise. I'm John barrier, the founder SiliconANGLE joins my cohost, Jeff Kelly, top big data analyst in the, in the community. Our next guest, Jack Norris, COO of map R security enterprise. That's the buzz of the show and it was the buzz of OpenStack summit. Another open source show. And here this year, you're just seeing move after, move at the moon, talking about a couple of critical issues. Enterprise grade Hadoop, Hortonworks announced a big acquisition when all in, as they said, and now cloud era follows suit with their news. Today, I, you sitting back saying, they're catching up to you guys. I mean, how do you look at that? I mean, cause you guys have that's the security stuff nailed down. So what Dan, >>You feel about that now? I think I'm, if you look at the kind of Hadoop market, it's definitely moving from a test experimental phase into a production phase. We've got tremendous customers across verticals that are doing some really interesting production use cases. And we recognized very early on that to really meet the needs of customers required some architectural innovation. So combining the open source ecosystem packages with some innovations underneath to really deliver high availability, data protection, disaster recovery features, security is part of that. But if you can't predict the PR protect the data, if you can't have multitenancy and separate workflows across the cluster, then it doesn't matter how secure it is. You know, you need those. >>I got to ask you a direct question since we're here at Hadoop summit, because we get this question all the time. Silicon lucky bond is so successful, but I just don't understand your business model without plates were free content and they have some underwriters. So you guys have been very successful yet. People aren't looking at map are as good at the quiet leader, like you doing your business, you're making money. Jeff. He had some numbers with us that in the Hindu community, about 20% are paying subscriptions. That's unlike your business model. So explain to the folks out there, the business model and specifically the traction because you have >>Customers. Yeah. Oh no, we've got, we've got over 500 paying customers. We've got at least $1 million customer in seven different verticals. So we've got breadth and depth and our business model is simple. We're an enterprise software company. That's looking at how to provide the best of open source as well as innovations underneath >>The most open distribution of Hadoop. But you add that value separately to that, right? So you're, it's not so much that you're proprietary at all. Right. Okay. >>You clarify that. Right. So if you look at, at this exciting ecosystem, Hadoop is fairly early in its life cycle. If it's a commoditization phase like Linux or, or relational database with my SQL open source, kind of equates the whole technology here at the beginning of this life cycle, early stages of the life cycle. There's some architectural innovations that are really required. If you look at Hadoop, it's an append only file system relying on Linux. And that really limits the types of operations. That types of use cases that you can do. What map ours done is provide some deep architectural innovations, provide complete read-write file systems to integrate data protection with snapshots and mirroring, et cetera. So there's a whole host of capabilities that make it easy to integrate enterprise secure and, and scale much better. Do you think, >>I feel like you were maybe a little early to the market in the sense that we heard Merv Adrian and his keynote this morning. Talk about, you know, it's about 10 years when you start to get these questions about security and governance and we're about nine years into Hadoop. Do you feel like maybe you guys were a little early and now you're at a tipping point, whereas these more, as more and more deployments get ready to go to production, this is going to be an area that's going to become increasingly important. >>I think, I think our timing has been spectacular because we, we kind of came out at a time when there was some customers that were really serious about Hadoop. We were able to work closely with them and prove our technology. And now as the market is just ramping, we're here with all of those features that they need. And what's a, what's an issue. Is that an incremental improvement to provide those kind of key features is not really possible if the underlying architecture isn't there and it's hard to provide, you know, online real-time capabilities in a underlying platform that's append only. So the, the HDFS layer written in Java, relying on the Linux file system is kind of the, the weak underbelly, if you will, of, of the ecosystem. There's a lot of, a lot of important developments happening yarn on top of it, a lot of really kind of exciting things. So we're actively participating in including Apache drill and on top of a complete read-write file system and integrated Hindu database. It just makes it all come to life. >>Yeah. I mean, those things on top are critical, but you know, it's, it's the underlying infrastructure that, you know, we asked, we keep on community about that. And what's the, what are the things that are really holding you back from Paducah and production and the, and the biggest challenge is they cited worth high availability, backup, and recovery and maintaining performance at scale. Those are the top three and that's kind of where Matt BARR has been focused, you know, since day one. >>So if you look at a major retailer, 2000 nodes and map bar 50 unique applications running on a single cluster on 10,000 jobs a day running on top of that, if you look at the Rubicon project, they recently went public a hundred million add actions, a hundred billion ad auctions a day. And on top of that platform, beats music that just got acquired for $3 billion. Basically it's the underlying map, our engine that allowed them to scale and personalize that music service. So there's a, there's a lot of proof points in terms of how quickly we scale the enterprise grade features that we provide and kind of the blending of deep predictive analytics in a batch environment with online capabilities. >>So I got to ask you about your go to market. I'll see Cloudera and Hortonworks have different business models. Just talk about that, but Cloudera got the massive funding. So you get this question all the time. What do you, how do you counter that army and the arms race? I think >>I just wrote an article in Forbes and he says cash is not a strategy. And I think that was, that was an excellent, excellent article. And he goes in and, you know, in this fast growing market, you know, an amount of money isn't necessarily translate to architectural innovations or speeding the development of that. This is a fairly fragmented ecosystem in terms of the stack that runs on top of it. There's no single application or single vendor that kind of drives value. So an acquisition strategy is >>So your field Salesforce has direct or indirect, both mixable. How do you handle the, because Cloudera has got feet on the street and every squirrel will find it, not if they're parked there, parking sales reps and SCS and all the enterprise accounts, you know, they're going to get the, squirrel's going to find a nut once in awhile. Yeah. And they're going to actually try to engage the clients. So, you know, I guess it is a strategy if they're deploying sales and marketing, right? So >>The beauty about that, and in fact, we're all in this together in terms of sharing an API and driving an ecosystem, it's not a fragmented market. You can start with one distribution and move to another, without recompiling or without doing any sort of changes. So it's a fairly open community. If this were a vendor lock-in or, you know, then spending money on brand, et cetera, would, would be important. Our focus is on the, so the sales execution of direct sales, yes, we have direct sales. We also have partners and it depends on the geographies as to what that percentage is. >>And John Schroeder on with the HP at fifth big data NYC has updated the HP relationship. >>Oh, excellent. In fact, we just launched our application gallery app gallery, make it very easy for administrators and developers and analysts to get access and understand what's available in the ecosystem. That's available directly on our website. And one of the featured applications there today is an integration with the map, our sandbox and HP Vertica. So you can get early access, try it and get the best of kind of enterprise grade SQL first, >>First Hadoop app store, basically. Yeah. If you want to call it that way. Right. So like >>Sure. Available, we launched with close to 30, 30 with, you know, a whole wave kind of following that. >>So talk a little bit about, you know, speaking of verdict and kind of the sequel on Hadoop. So, you know, there's a lot of talk about that. Some confusion about the different methods for applying SQL on predicts or map art takes an open approach. I know you'll support things like Impala from, from a competitor Cloudera, talk about that approach from a map arts perspective. >>So I guess our, our, our perspective is kind of unbiased open source. We don't try to pick and choose and dictate what's the right open source based on either our participation or some community involvement. And the reality is with multiple applications being run on the platform, there are different use cases that make difference, you know, make different sense. So whether it's a hive solution or, you know, drill drills available, or HP Vertica people have the choice. And it's part of, of a broad range of capabilities that you want to be able to run on the platform for your workflows, whether it's SQL access or a MapReduce or a spark framework shark, et cetera. >>So, yeah, I mean there is because there's so many different there's spark there's, you know, you can run HP Vertica, you've got Impala, you've got hive. And the stinger initiative is, is that whole kind of SQL on Hadoop ecosystem, still working itself out. Are we going to have this many options in a year or two years from now? Or are they complimentary and potentially, you know, each has its has its role. >>I think the major differences is kind of how it deals with the new data formats. Can it deal with self-describing data? Sources can leverage, Jason file does require a centralized metadata, and those are some of the perspectives and advantages say the Apache drill has to expand the data sets that are possible enabled data exploration without dependency on a, on an it administrator to define that, that metadata. >>So another, maybe not always as exciting, but taking workloads from existing systems, moving them to Hadoop is one of the ways that a lot of people get started with, to do whether associated transformation workloads or there's something in that vein. So I know you've announced a partnership with Syncsort and that's one of the things that they focus on is really making it as easy as possible to meet those. We'll talk a little bit about that partnership, why that makes sense for you and, and >>When your customer, I think it's a great proof point because we announced that partnership around mainframe offload, we have flipped comScore and experience in that, in that press release. And if you look at a workload on a mainframe going to duke, that that seems like that's a, that's really an oxymoron, but by having the capabilities that map R has and making that a system of record with that full high availability and that data protection, we're actually an option to offload from mainframe offload, from sand processing and provide a really cost effective, scalable alternative. And we've got customers that had, had tried to offload from the mainframe multiple times in the past, on successfully and have done it successfully with Mapbox. >>So talk a little bit more about kind of the broader partnership strategy. I mean, we're, we're here at Hadoop summit. Of course, Hortonworks talks a lot about their partnerships and kind of their reseller arrangements. Fedor. I seem to take a little bit more of a direct approach what's map R's approach to kind of partnering and, and as that relates to kind of resell arrangements and things like, >>I think the app gallery is probably a great proof point there. The strategy is, is an ecosystem approach. It's having a collection of tools and applications and management facilities as well as applications on top. So it's a very open strategy. We focus on making sure that we have open API APIs at that application layer, that it's very easy to get data in and out. And part of that architecture by presenting standard file system format, by allowing non Java applications to run directly on our platform to support standard database connections, ODBC, and JDBC, to provide database functionality. In addition to kind of this deep predictive analytics really it's about supporting the broadest set of applications on top of a single platform. What we're seeing in this kind of this, this modern architecture is data gravity matters. And the more processing you can do on a single platform, the better off you are, the more agile, the more competitive, right? >>So in terms of, so you're partnering with people like SAS, for example, to kind of bring some of the, some of the analytic capabilities into the platform. Can you kind of tell us a little bit about any >>Companies like SAS and revolution analytics and Skytree, and I mean, just a whole host of, of companies on the analytics side, as well as on the tools and visualization, et cetera. Yeah. >>Well, I mean, I, I bring up SAS because I think they, they get the fact that the, the whole data gravity situation is they've got it. They've got to go to where the data is and not have the data come to them. So, you know, I give them credit for kind of acknowledging that, that kind of big data truth ism, that it's >>All going to the data, not bringing the data >>To the computer. Jack talk about the success you had with the customers had some pretty impressive numbers talking about 500 customers, Merv agent. The garden was on with us earlier, essentially reiterating not mentioning that bar. He was just saying what you guys are doing is right where the puck is going. And some think the puck is not even there at the same rink, some other vendors. So I gotta give you props on that. So what I want you to talk about the success you have in specifically around where you're winning and where you're successful, you guys have struggled with, >>I need to improve on, yeah, there's a, there's a whole class of applications that I think Hadoop is enabling, which is about operations in analytics. It's taking this, this higher arrival rate machine generated data and doing analytics as it happens and then impacting the business. So whether it's fraud detection or recommendation engines, or, you know, supply chain applications using sensor data, it's happening very, very quickly. So a system that can tolerate and accept streaming data sources, it has real-time operations. That is 24 by seven and highly available is, is what really moves the needle. And that's the examples I used with, you know, add a Rubicon project and, you know, cable TV, >>The very outcome. What's the primary outcomes your clients want with your product? Is it stability? And the platform has enabled development. Is there a specific, is there an outcome that's consistent across all your wins? >>Well, the big picture, some of them are focused on revenues. Like how do we optimize revenue either? It's a new data source or it's a new application or it's existing application. We're exploding the dataset. Some of it's reducing costs. So they want to do things like a mainframe offload or data warehouse offload. And then there's some that are focused on risk mitigation. And if there's anything that they have in common it's, as they moved from kind of test and looked at production, it's the key capabilities that they have in enterprise systems today that they want to make sure they're in Hindu. So it's not, it's not anything new. It's just like, Hey, we've got SLS and I've got data protection policies, and I've got a disaster recovery procedure. And why can't I expect the same level of capabilities in Hindu that I have today in those other systems. >>It's a final question. Where are you guys heading this year? What's your key objectives. Obviously, you're getting these announcements as flurry of announcements, good success state of the company. How many employees were you guys at? Give us a quick update on the numbers. >>So, you know, we just reported this incredible momentum where we've tripled core growth year over year, we've added a tremendous amount of customers. We're over 500 now. So we're basically sticking to our knitting, focusing on the customers, elevating the proof points here. Some of the most significant customers we have in the telco and financial services and healthcare and, and retail area are, you know, view this as a strategic weapon view, this is a huge competitive advantage, and it's helping them impact their business. That's really spring our success. We've, you know, we're, we're growing at an incredible clip here and it's just, it's a great time to have made those calls and those investments early on and kind of reaping the benefits. >>It's. Now I've always said, when we, since the first Hadoop summit, when Hortonworks came out of Yahoo and this whole community kind of burst open, you had to duke world. Now Riley runs at it's a whole different vibe of itself. This was look at the developer vibe. So I got to ask you, and we would have been a big fan. I mean, everyone has enough beachhead to be successful, not about map arbors Hortonworks or cloud air. And this is why I always kind of smile when everyone goes, oh, Cloudera or Hortonworks. I mean, they're two different animals at this point. It would do different things. If you guys were over here, everyone has their quote, swim lanes or beachhead is not a lot of super competition. Do you think, or is it going to be this way for awhile? What's your fork at some? At what point do you see more competition? 10 years out? I mean, Merv was talking a 10 year horizon for innovation. >>I think that the more people learn and understand about Hadoop, the more they'll appreciate these kind of set of capabilities that matter in production and post-production, and it'll migrate earlier. And as we, you know, focus on more developer tools like our sandbox, so people can easily get experienced and understand kind of what map are, is. I think we'll start to see a lot more understanding and momentum. >>Awesome. Jack Norris here, inside the cube CMO, Matt BARR, a very successful enterprise grade, a duke player, a leader in the space. Thanks for coming on. We really appreciate it. Right back after the short break you're live in Silicon valley, I had dupe December, 2014, the right back.

>>Winston Edmundson here at Hadoop summit. We've got Steve woolens from Teradata. He's going to talk to me a little bit about a exciting new announcement that you had with Hortonworks today. Tell me a little bit about that. >>Yeah. So Teradata has been in the data management analytics space for over 30 years. And with the announcement today, we announced data portfolio for Hadoop, which is a collection of products, services, and customer support for an entire portfolio for the products. So we've got turnkey appliances, we've got commodity offerings and with Hortonworks, we've got a shared customer support model, so we can give our customers everything they need around >>Ultimate support. Pretty exciting. Now this seems like it must've been a long process to put all this together. >>Well, we've had a partnership with Hortonworks for about a year. We've had Hadoop product offerings in the market for about six months. We've seen a lot of uptake from our customers, and it's really about broadening that to make sure that customers can buy a dupe standalone integrated in with the rest of their data architecture and make it a trusted component within that next generation data architecture. >>Tell me what excites you right now with the customers that you're helping, you're meeting their needs. Where do you see things going? What trends are you following right now? >>The big thing we're seeing is customers. Our customers want to better serve their customers. And there's so many new interaction points that they have with those customers through social networks, email, and being able to take things like the call center voice records, but that's been data that hasn't really been explored in the past to figure out how to better serve those customers. So now with Hadoop and other MapReduce technologies, we can incorporate that analysis into how we better serve our customers, customers at the end of the day. If that makes sense, that's ultimately, it's about getting deeper insights into how to better service the customers. And I think with all the new data that's out there and the hype around big data, that's really what it's about. >>Do you find the customers are coming to you with their own ideas or are they looking to you for suggestions on just how they can bring these different data sets together and how they can maximize and leverage some of this data? >>Well, the problem is there's so much hysteria in this market. I mean, it's an exciting place to be, but there's a lot of technologies, right? So I think the thing with Teradata is we do provide that trusted advisor status. I mean, we've been implementing data analytics solutions for a long, long time and a lot of the problems aren't new, they're just incorporating new analytics techniques. So they have ideas in terms of things they've heard about. They're not really sure how to implement it sometimes. So part of our offering is we have services, so we can look across their entire data architecture and figure out where does the dupe really fit? What are the best use cases for it? How do we integrate that across the enterprise? So the end users and the applications that can benefit from that data can really get the value from it. >>How important do you think it is or how much is an advantage that you are tried and true. You've been here. I mean, some of these solution providers, you can call them fly by night. I mean, they just, they're just here on, you know, they've just formed. They don't have a track record. It's your track record of success? One of the main things that customers are attracted to? >>I think so. I mean, the reality is we have, we're like in the trenches with our customers, it's not just the technology, but when we have business consulting, people that come in with domain expertise from a given industry, so you can call it a track record or whatever it is, but it's really understanding, not just technology, but the business and how these things come together to really get the most value from all the cool technology that's out there. So yeah, a lot of the fly by nighters, I mean, there's a lot of innovative things that are happening. And at duke five years ago, it was one of those very new things. And so we've been looking at it for a while and now we figured out the best way to incorporate it into our solution portfolio and to roll it out to customers >>When you're helping a customer. And you're, you're looking at the here and now, this is what they, they need to be addressing. I would imagine a lot of customers want to know what's around the corner, what's around the bend that we should be aware of, that we should try to be, be prepared for. What do you, what do you tell them? >>Well, I think, you know, everybody will say there's just more and more data coming at you. I think other analytic techniques like graph analysis is something that people particularly with social networks are trying to figure out how are people interrelated to each other. So it's a lot of different use cases and there's different analytic techniques that can be combined in unique ways. So a lot of our R and D investment is going into how do we bring more of those analytic techniques and unify them for people in one system. So that regardless of your data scientists or business analysts, you can ask really interesting, tough questions that you couldn't answer ask before. So it's about giving answers to sometimes the unknown questions and helping them explore that data through unique ways. >>What would you say are some of the industries that are maybe there's probably more urgency for them to adopt some of these strategies or perhaps just, they're more likely to have a big return on investment? What industries would you point to? >>I mean, for us, it's a lot of the traditional industries where you have a lot of consumers, right? Telecommunications, retail, retail, financial services, anybody who's working with. A lot of customers that have a lot of products, just have a lot of complexity, a lot of customer interaction touchpoints. So I think those are the people that typically we see adopting new technology and really thinking about how to better serve their customers >>For folks that are watching tuning in. And they're pretty excited about what you might be able to help them with. What's the best way for them to get in touch with you or, or >>You just go to teradata.com and check us out there. That's probably the best way to reach us. >>Right. Fantastic. Thanks for your time. Winston Edmondson here with studio B signing out.

live at hadoop summit this is SiliconANGLE and wiki bonds exclusive coverage of hadoop summit this is the cube our flagship program would go out the advanced extract the signal from the noise i'm to enjoy my co-host Jeff Kelly Jeff welcome to the cube Scott welcome to the cube great to have you here so you kicked off help kick off the show this morning with your keynote talking about a number of things among them the new teradata plans for Hadoop brought it on stage which I thought was great i love i love some i was joined by a dancing appliance okay great it was fantastic a good-looking appliance it was but why don't you tell us a little bit about yourself kind of your role and then we'll kind of get into what tara date is doing here at the show and some of the some of the strategies you're taking towards the big data market okay great well I'm Scott now I'm from tarde de labs and turny two labs is actually organization within teradata that is responsible for research development engineering product management product marketing all the products all of the technology that we roll out kind of the innovation engine of teradata is what we're responsible for and we've been obviously affiliated with hadoop summit we were here last year it's really great to be back having been in the in the data warehouse big data kind of data analytics business for a long time the one thing I have to say about this whole movement in the Hadoop space is that it's unlike anything else I've seen in that it's every geography it's every industry and there's so much energy and emotion around it's unlike any other transition that I've seen and even the difference between our visit here last year and this year where we've seen the the promise turned into reality where we've got customers who are implementing where we've got businesses who are driving value from the solutions that they're really that they're integrating with the solutions that they've already got and and being able to demonstrate that value really emphasizes the importance and I think will help to continue the momentum that we feel in this market Scott one of the things I want to ask you was obviously the theme at had dude was off loading data warehouses what they do is a benefit there but you have a relationship with Hortonworks and we've had we were talking early with Murph was an analyst at Gartner was talking about the the early adopters and the mainstream getting it now and but there's always a question of value right where's the value because his legacy involved right so the most of the web based companies are going to be cloud they'll be SAS they might have a Greenfield clean sheet of paper to work with on big data but an existing enterprise large financial institutions insurance company or what have you they have legacy technology and they have to but they want Hadoop they want to bring it in when you talk to folks out there what are some of the challenges and opportunities they have with that environment and the technology specifically sure that was like a long question there's a lot of a lot of threads in there I want to really try to hit on a couple of important themes because you know you hear it here I get asked a lot about it you know one of the things that people often say is you know this why are you here this whole Hadoop thing is offloading data warehouses isn't that bad doesn't that bother you and the answer is absolutely not certainly there's some hype around that and you know those some marketing around that but when you really look at the technology and the value of what it brings to the table it's a new technology that really allows us to harness new kinds of data and store those new kinds of data in the native format and you know storing detailed data in the native format really enables the best world-class analytics we've seen this happen for you know as long as my career is in the traditional data space so that's a really good thing the way I view it though is sure will some work load move around the infrastructure from the data warehouse to a Hadoop cluster potentially right and by the way if Hadoop is a great solution for it it should go there all right but at the same time there is more demand than there is supply of technology and what I mean by that is the demand for analytics is so extreme that actually adding this tool to the toolkit gives customers more choice and gives them the opportunity to really catch up with the backlog of things that they've wanted to invest in overtime and then the final point really I view what's happening here as perhaps one of the single largest opportunities for expansion of the role and size and scope of the data warehouse in an enterprise because one of the big things that Hadoop brings to the table is a whole lot of raw material a whole lot more data data that used to be thrown away data that never existed a year ago is now going to be able to capture be captured be stored be refined be analyzed and as companies start to find relationships as companies start to find actionable tidbits from the analytics in this huge source of raw material I think it's actually an opportunity for upside for them to integrate more data into their data warehouse where they can actually do the real-time interaction and streaming that's going to get them to the demonstrable business benefit so it's the modernization of the enterprise it's its modernization the way I look at it is also it's sometimes the word incremental can be it can sound like it we're trying to downplay it but I see it as incremental in that it's different data and it's incremental data it's incremental subject areas its new stuff that's going to come into the environment and based on what we've seen in the history of analytics right that there's no end to the value that companies find and there's no end to competition in their businesses so this is a huge opportunity for the entire community to deliver more analytics and i think that there's actually more upside for traditional legacy data warehouse vendors and there is anything I think that's a really important point because as you said a lot of people think about that offloading workloads but it's also about offloading we're close but bringing in new data doing more analytics and then moving some of that into back into the data warehouse you can actually create more value from it yeah I mean one of the things that I've seen is you know over time and Moore's law is something that's been going on for some time right and and cost erosion in Hardware has been going on for a long time and you think about the thing that you buy today for your bi implementation the hardware costs what twenty percent of what it costs three four years ago and you know what revenues continue to increase because they're such pent-up demand that as it gets less expensive it becomes more consumable and I think the same thing it's really going to continue to happen as we add in these new technologies and these new data types so one of the things I want to commend teradata for doing is focusing on kind of that reference protector and helping customers understand how this new technology of Hadoop and big data fits in with everything else that they're doing talk a little bit a bit about how from a reference architecture and then maybe even from a product perspective how teradata goes about turning this into a reality for enterprise customers who you know really you know they're not looking to just kick the tires of the Duke they want they want to use this for its really support you know applications and workflows they're really you know critical to their business yeah I think you know one of the biggest things that we can do to help the industry and to help our customers really is to define a realistic roadmap that's consumable for them in their enterprise and so while it's certainly easy to have marketing release or press release it says uh this new technology does everything in slices bread it washes your car does all these things in reality there are very few things like that in the world right but the new technologies and the new innovations really do fit into some very interesting new use cases and so by providing this integrated roadmap of how customers can deploy and fit these technologies together is a really great education process and it's been extremely well received by our customers and prospects I have to tell you that even in advance of the announcement of the things that we had here today we've already got customers who have gone down this path with us because it's such a compelling value proposition the other thing is that we don't actually put specific technology in those boxes it's a reference architecture we hope that there's some teradata product in there but at the same time we you know our customers understand that there is choice in the marketplace and the best solution is going to win and by providing this reference architecture I think we helped elevate ourselves to more of a trusted advisor status with with the the industry and in how we see these things fitting together and providing very effective very low-risk kinds of solutions well I think you hit on something that trusted advisor I think companies and enterprises are just crying out for some leadership and to help to help them really understand how they're going to make this a reality in their organizations and you know you mentioned kind of the openness and being you know allowing enterprises shoots a technology that fits that fits the the work case of course you know you hope that stared at in a lot of cases but it could be something else so talk a little bit about your relationship with hortonworks so I know you announced today kind of a reseller agreement you're going to be actually reselling the the subscription service to Hortonworks service offering talk about that a little bit and also I want to dive into the tech as well the Hadoop appliance I mentioned earlier like you announced and maybe just kind of walk us through some of the news to them sure so I mean obviously we have a strategic relationship with Hortonworks and it's our second year here at Summit and it really started with I think a very common view of what's happening in the marketplace and how these technologies should really play well together at the same time we also really believe that it's important that the community embrace the open source Apache version of the software so that it doesn't become fragmented and become obsolete right so Horton is spot-on in terms of business model and putting everything back into the Apache open source version so that means that I think this is the version that will win and this will be the version that companies can count on to be sustainable so i think that there's an advantage there implied so that's said i think it fits into the right place we've got a great engineering relationship and a great common vision on how the enterprise architecture and how the pieces can fit together and be optimized for different workloads for different service levels and for different applications so having that common vision and kind of I think bringing to Best of Breed providers together with Wharton works on the on the Hadoop side and teradata for what we're very well known for I think it's really the best of all worlds and we work together to lay out this reference architecture and so it's not just you know tur data came down from the mountain said this should be your reference architecture we've got some validation we got some validation of use cases and then we went to work from an engineering perspective on how we go build these things out and make them work and optimize them and support them end to end because obviously not only in you know with the all of the new solutions is their kind of a scarcity of talent and some confusion support becomes really really important so one of the things we added to our portfolio we announced today is an expanded relationship on the support side where customers can come to teradata for integrated support of all of their data analytics environments whether it be teradata whether it be asked her whether it be Hadoop with hdb and you know that's a really nice thing where there's one phone number to call we've got fully integrated processes we can help with a global footprint in the 80 countries where we do business and obviously Hortonworks with the with the extreme depth and ability to manage the content of the kernel can get it done unlike anyone else Scott we've been talking enterprise-grade all morning as you did those the theme of the keynote mer from our garden about security compliance I mean these are meat and potatoes enterprise issues right so I got to ask you what's what are you guys looking at what's what's coming next obviously the platform to do has a stabilized developers going to want to program on it in different environments but the reality in the enterprise is a certain requirement so what are you looking at in the labs that's coming around the corner that's it going to be really really important for customers to realize the value of scaling and harnessing the big data of Hadoop with the existing infrastructure yeah I mean I think there are two things that will continue to do one is will look to build out kind of that framework of ecosystem and in all of the keynotes this morning you know everyone talked about the value of the ecosystem and it's amazing the ecosystem how they're just more and more logos this year than there were last year and I think that that will continue but really building out that ecosystem so that those things that are important can be realized and they can be realized in a very repeatable fashion I think in addition to that kind of ease of use right because despite the fact that we have burgeoning numbers of newly minted data scientists and people getting into the marketplace that's really good there still aren't enough and so de-risking things by making them easier to deploy and easier to support i think is a key focus area and then you know finally I said two things but now third you know finally it will say to me I'd all right we'll continue to look at performance and just making sure that we have the best density the best performance the cost performance value proposition that our customers will want because I also continue to believe that the supply of data will outstrip any customers ability to invest in infrastructure I'd love to get your take on want to go back to mention to what you mentioned about the you know the Hadoop distribution focusing on a patchy and moving a patchy compatible so I take that number one to me and Tara day is not going to be coming out with their own Hadoop distribution absolutely not but how do you think about that yeah I think we can say that pretty definitively so but what about how do you see this whole Hadoop market playing out them you've got a Hortonworks Cloudera map are some others how do you see this playing out in the next year or so I mean is this you mentioned you think again that's kind of the open source of patchy versions going to kind of win when do you think that's going to happen you've got some competitors in the market and different business models hot yeah you know there are different business models and different innovators and you know my crystal ball is probably only about as clear as anyone elses but you know kind of for the long term I think it's best for the industry if if it mimics a model similar to the way Linux is deployed where this kind of a duopoly maybe three vendors it's very largely open source there's a lot of portability between I think that really strengthens the position of Hadoop as a tech as a core technology and foundation for some of the things that we're doing and so I would hope that in you know the most successful outcome would be that we'd end up with a duopoly or or you know maybe three kind of providers around a similar colonel because that would that would remove fragmentation from the market by the way I think it you know where we are software company so I think it's fair for companies to have value add proprietary software that's not a bad thing but at the file system level at a core two level I think the open source community cannot be out innovated right and and so I think that that's a really important thing so I think you know hopefully we'll get to that duopoly or maybe three companies that kind of have that I don't know if we will but I sure hope we do and I think the if I were to bet on it I would say it's odds on that that will be the case now will that be 18 months three years five years I don't know Scott thanks for coming inside the cube obviously you guys have a great position in the market place and the enterprise message is straw here that's what the demand is we're seeing a lot of trends out there that want the enterprise grade big data which is not just once there's but Hadoop's a big part of it Thanks coming inside the cube and sharing your perspective and what you got working on certainly having the new products come out to be great so thanks for coming onto the cube this is SiliconANGLE and wiki bonds coverage of hadoop summit we'll be right back with our next guest after this short break you

>>Ash it's, you know, what will that mean to my investment? And the announcement fusion IO is that, you know, we're 25 times faster on read intensive HBase applications. The combination. So as organizations are deploying Hadoop, and they're looking at technology changes coming down the pike, they can rest assured that they'll be able to take advantage of those in a much more aggressive fashion with map R than, than other distribution. >>Jack, how I got to ask you, we were talking last night at the Hadoop summit, kind of the kickoff party and, you know, everyone was there. All the top execs were there and all the developers, you know, we were in the queue. I think, I think that either Dave or myself coined the term, the big three of big data, you guys ROMs cloud Cloudera map R and Hortonworks, really at the, at the beginning of the key players early on and Charles from Cloudera was just recently on. And, and he's like, oh no, this, this enterprise grade stuff has been kicked around. It's been there from the beginning. You guys have been there from the beginning and Matt BARR has never, ever waffled on your, on your messaging. You've always been very clear. Hey, we're going to take a dupe open source a dupe and turn it into an enterprise grade product. Right. So that's clear, right? That's, that's, that's a great, that's a great, so what's your take on this because now enterprise grade is kind of there, I guess, the buzz around getting the, like the folks that have crossed the chasm implemented. So what can you comment on that about one enterprise grade, the reality of it, certainly from your perspective, you haven't been any but others. And then those folks that are now rolling it out for the first time, what can you share with them around? What does it mean to be enterprise grade? >>So enterprise grade is more about the customer experience than, than a marketing claim. And, you know, by enterprise grade, what we're talking about are some of the capabilities and features that they've grown to expect in their, their other enterprise applications. So, you know, the ability to meet full S SLA is full ha recovery from multiple failures, rolling upgrades, data protection was consistent snapshots business continuity with mirroring the ability to share a cluster across multiple groups and have, you know, volumes. I mean, there's a, there's a host of features that fall under the umbrella enterprise grade. And when you move from no support for any of those features to support to a few of them, I don't think that's going to, to ha it's more like moving to low availability. And, and there's just a lot of differences in terms of when we say enterprise grade with those features mean versus w what we view as kind of an incomplete story. So >>What do you, what do you mean by low availability? Well, I mean, it's tongue in cheek. It's nice. It's a good term. It's really saying, you know, just available when you sometimes is that what you mean? Is this not true availability? I mean, availability is 99.9%. Right? >>Right. So if you've got a, an ha solution that can't recover from multiple failures, that's downtime. If you've got an HBase application that's running online and you have data that goes down and it takes 10 to 30 minutes to have the region servers recover it from another place in the distribution, that's downtime. If you have snapshots that aren't consistent across the cluster, that doesn't provide data protection, there's no point in time recovery for, for a cluster. So, you know, there's a lot of details underneath that, but what it, what it amounts to is, do you have interruptions? Do you have downtime? Do you have the potential for losing data? And our answer is you need a series of features that are hardened and proven to deliver that. >>What about recoverability? You mentioned that you guys have done a lot of work in that area with snapshotting, that's kind of being kicked around, are our folks addressing, what are the comp what's your competition doing in those areas of recoverability just mentioned availability. Okay, got that. Recoverability security, compliance, and usability. Those are the areas that seem to be the hot focus areas what's going on in the energy. How would you give them the grade, the letter grade, if you will, candidly, compared to what you guys offer? Well, the, >>The first of all, it's take recoverability. You know, one of the tenants is you have a point in time recovery, the ability to restore to a previous point that's consistent across the cluster. And right now there's, there's no point in time recovery for, for HDFS, for the files. And there's no point in time recovery for HBase tables. So there's snapshot support. It's being talked about in the open source community with respect to snapshots, but it's being referred to in the JIRAs as fuzzy snapshots and really compared to copy table. >>So, Jack, I want to turn the conversation to the, kind of the topic we've talked about before kind of the open versus a proprietary that, that whole debate we've, we've, we've heard about that. We talked about that before here on the cube. So just kind of reiterate for us your take. I mean, we, we hear perhaps because of the show we're at, there's a lot of talk about the open source nature of Hadoop and some of the purists, as you might call them are saying, it's gotta be open a hundred percent Patrick compatible, et cetera. And then there's others that are taking a different approach, explain your approach and why you think that's the key way to make, to really spur adoption of a dupe and make it >>W w we're we're a part of the community we're, we've got, you know, commitment going on. We've, you know, pioneered and pushed a patchy drill, but we have done innovations as well. And I think that those innovations are really required to support and extend the, the whole ecosystem. So canonical distributes RN, three D distribution. We've got, you know, all our, our packages are, are available on get hub and, and open source. So it's not, it's not a binary debate. And I think the, the point being that there's companies that have jumped ahead and now that Peloton is, is, you know, pedaling faster and, and we'll, we'll catch up. We'll streamline. I think the difference is we rearchitected. So we're basically in a race car and, you know, are, are racing ahead with, with enterprise grade features that are required. And there's a lot of work that still needs to be done, needs to be accomplished before that full rearchitecture is, is in place. >>Well, I mean, I think for me, the proof is really in the pudding when you, when it comes to talk about customers that are doing real things and real production, grade mission, critical applications that they're running. And to me that shows the successor or relative success of a given approach. So I know you guys are working with companies like ancestry.com, live nation and Quicken loans. Maybe you could, could you walk us through a couple of those scenarios? Let's take ancestry.com. Obviously they've got a huge amount of data based on the kind of geological information, where do you guys do >>With them? Yeah, so they've got, I mean, they've got the world's largest family genealogy services available on the web. So there's a massive amount of data that they make accessible and, and, you know, ability for, for analysis. And then they've rolled out new features and new applications. One of which is to ship a kit out, have people spit in a tube, returned back and they do DNA matching and reveal additional details. So really some really fabulous leading edge things that are being done with, with the use of, of Hadoop. >>Interesting. So talk about when you went to, to work with them, what were some of their key requirements? Was it around, it was more around the enterprise enterprise, grade security and uptime kind of equation, or was it more around some of the analytics? What, what, what's the kind of the killer use case for them? >>It's kind of, you know, it's, it's hard with a specific company or even, you know, to generalize across companies. Cause they're really three main areas in terms of ease of use and administration dependability, which includes the full ha and then, and then performance. And in some cases, it's, it's just one of those that kind of drives it. And it's used to justify, in other cases, it's kind of a collection. The ease of use is being able to use a cluster, not only as Hadoop, but to access it and treat it like enterprise storage. So it's a complete POSIX compliance file system underneath that allows the, the mounting and access and updates and using it in dynamic read-write. So what that means from an application level, it's, it's faster, it's much easier to administer and it's much easier and reliable for developers to, to utilize. >>I got to ask you about the marketing question cause I see, you know, map our, you guys have done a good job of marketing. Certainly we want to be thankful to you guys is supporting the cube in the past and you guys have been great supporters of our mission, but now the ecosystem's evolving a lot more competition. Claudia mentioned those eight companies they're tracking in quote Hadoop, and certainly Jeff and I, and, and SiliconANGLE by look at there's a lot more because Hadoop washing has been going on now for the term Hadoop watching me and jumping in and doing Hadoop, slapping that onto an existing solution. It's not been happening full, full, full bore for a year. At least what's the next for you guys to break above the noise? Obviously the communities are very active projects are coming online. You guys have your mission in the enterprise. What's the strategy for you guys going forward is more of the same and anything new even share. >>Yeah, I, I, I think as far as breaking above the noise, it will be our customers, their success and their use cases that really put the spotlight on what the differences are in terms of, of, you know, using a big data platform. And I think what, what companies will start to realize is I'd rather analogy between supply chain and the big, the big revolution in supply chain was focusing on inventory at each stage in the supply chain. And how do you reduce that inventory level and how do you speed the, the flow of goods and the agility of a company for competitive advantage. And I think we're going to view data the same way. So companies instead of raw data that they're copying and moving across different silos, if they're able to process data in place and send small results sets, they're going to be faster, more agile and more competitive. >>And that puts the spotlight on what data platform is out there that can support a broad set of applications and it can have the broadest set of functionality. So, you know, what we're delivering is a mission grade, you know, enterprise grade mission, critical support platform that supports MapReduce and does that high performance provides NFS POSIX access. So you can use it like a file system integrates, you know, enterprise grade, no SQL applications. So now you can do, you know, high-speed consistent performance, real time operations in addition to batch streaming, integrated search, et cetera. So it's, it's really exciting to provide that platform and have organizations transform what they're doing. >>How's the feedback on with Ted Dunning? I haven't seen a lot of buzz on the Twittersphere is getting positive feedback here. He's a, a tech athlete. He's a guru, he's an expert. He's got his hands in all the pies. He's a scientist type. What's he up to? What's his, what's his role within Mapa and he's obviously playing in the open-source community. What's he up to these days, >>Chief application architect, he's on the leading edge of my house. So machine learning, so, you know, sharing insights there, he was speaking at the storm meetup two nights ago and sharing how you can integrate long running batch, predictive analytics with real-time streaming and how the use of snapshots really that, that easy and possible. He travels the world and is helping organizations understand how they can take some very complex, long running processes and really simplify and shorten those >>Chance to meet him in New York city had last had duke world at a, at a, a party and great guy, fantastic geek, and certainly is doing a great work and shout out to Ted. Congratulations, continue up that support. How's everyone else doing? How's John and Treevis doing how's the team at map are we're pedaling as best as you can growing >>Really quickly. No, we're just shifting gears. Would it be on pedaling >>Engine? >>Yeah. Give us an update on the company in terms of how the growth and kind of where you guys are moving that. >>Yeah. We're, we're expanding worldwide, you know, just this, you know, last few months we've opened up offices and in London and Munich and Paris, we're expanding in Asia, Japan and Korea. So w our, our sales and services and engineering, and basically across the whole company continues to expand rapidly. Some really great, interesting partnerships and, and a lot of growth Natalie's we add customers, but it's, it's nice to see customers that continue to really grow their use of map are within their organization, both in terms of amount of data that they're analyzing and the number of applications that they're bringing to bear on the platform. >>Well, that a little bit, because I think, you know, one of the, one of the trends we do see is when a company brings in big data, big data platform, and they might start experiment experimenting with it, build an application. And then maybe in the, maybe in the marketing department, then the sales guys see it and they say, well, maybe we can do something with that. How is that typically the kind of the experience you're seeing and how do you support companies that want to start expanding beyond those initial use cases to support other departments, potentially even other physical locations around the world? How do you, how do you kind of, >>That's been the beauty of that is if you have a platform that can support those new applications. So if you know, mission critical workloads are not an issue, if you support volumes so that you can logically separate makes it much easier, which we have. So one of our customers Zions bank, they brought in Matt BARR to do fraud detection. And pretty soon the fact that they were able to collect all of that data, they had other departments coming to them and saying, Hey, we'd like to use that to do analysis on because we're not getting that data from our existing system. >>Yeah. They come in and you're sitting on a goldmine, there are use cases. And you also mentioned kind of, as you're expanding internationally, what's your take on the international market for big data to do specifically is, is the U S kind of a leaps and bounds ahead of the rest of the world in terms of adoption of the technology. What are you seeing out there in terms of where, where the rest of the, >>I wouldn't say leaps and bounds, and I think internationally, they're able to maybe skip some of the experimental steps. So we're seeing, we're seeing deployment of class financial services and telecom, and it's, it's fairly broad recruit technologies there. The largest provider of recruiting services, indeed.com is one of their subsidiaries they're doing a lot with, with Hadoop and map are specifically, so it's, it's, it's been, it's been expanding rapidly. Fantastic. >>I also, you know, when you think about Europe, what's going on with Google and some of the, the privacy concerns even here, or I should say, is there, are there different regulatory environments you've got to navigate when you're talking about data and how you use data when you're starting to expand to other, other locales? >>Yeah. There's typically by vertical, there's different, different requirements, HIPAA and healthcare, and basal to, and financial services. And so all of those, and it, it, it basically, it's the same theme of when you're bringing Hadoop into an organization and into a data center, the same sorts of concerns and requirements and privacy that you're applying in other areas will be applied on Hindu. >>I'm now kind of turning back to the technology. You mentioned Apache drill. I'd love to get an update on kind of where, where that stands. You know, it's put, then put that into context for people. We hear a lot about the SQL and Hadoop question here, where does drill fit into that, into that equation? >>Well, the, the, you know, there's a lot of different approaches to provide SQL access. A lot of that is driven by how do you, how do you leverage some of the talent and organization that, you know, speak SQL? So there's developments with respect to hive, you know, there's other projects out there. Apache drill is an open source project, getting a lot of community involvement. And the design center there is pretty interesting. It started from the beginning as an open source project. And two main differences. One was in looking at supporting SQL it's, let's do full ANSI SQL. So it's full 2003 ANSI, sequel, not a SQL like, and that'll support the greatest number of applications and, you know, avoid a lot of support and, and issues. And the second design center is let's support a broad set of data sources. So nested sources like Jason scheme on discovery, and basically fitting it into an enterprise environment, which sometimes is kinda messy and can get messy as acquisitions happen, et cetera. So it's complimentary, it's about, you know, enabling interactive, low latency queries. >>Jack, I want to give you the final word. We are out of time. Thanks for coming on the cube. Really preached. Great to see you again, keep alumni, but final word. And we'll end the segment here on the cube is your quick thoughts on what's happening here at Hadoop world. What is this show about? Share with the audience? What's the vibe, the summary quick soundbite on Hadoop. >>I think I'll go back to how we started. It's not, if you used to do putz, how you use to do and, you know, look at not only the first application, but what it's going to look like in multiple applications and pay attention to what enterprise grade means. >>Okay. They were secure. We got a more coverage coming, Jack Norris with map R I'll say one of the big three original, big three, still on the, on the list in our mind, and the market's mind with a unique approach to Hadoop and the mid-June great. This is the cube I'm Jennifer with Jeff Kelly. We'll be right back after this short break, >>Let's settle the PR program out there and fighting gap tech news right there. Plenty of the attack was that providing a new gadget. Let's talk about the latest game name, but just the.

>>Come back here. This is Silicon Valley coverage of ADU Summit. I'm John Fur, the founder. We're, we're pleased to have a friend inside the cube. It's rare to have such luminaries, Ama Aala, good friend and also co-founder of Cloudera. Really the pioneer in the space that helped build this industry that we're living here at at Hadoop Summit. I'm with Dave Ante from wiba.org. Amour, welcome back to the Cube Cub alumni. Thank you for having me here. Wow, what a journey. Are you co-founded Cloudera? I remember when you in Stealth Mo, I really can't talk about it. And, and then of course the history of Silicon Angle being, you know, founded and kind of built in in your office when you only had like 20 something employees. Yep. We owe a great deal of gratitude to you and, and congratulations to you Michael Olson, the team for building an industry. So I just wanted Thank you. Thank you. And welcome to the Cube. >>Thank you. It was great to be here. >>So what do you think, what's your take on the current Hadoop ecosystem right now? I mean, obviously a lot's happened. I mean it's big now. It's growing up fast. Yeah. The word enterprise grade is out there. You're seeing it move from, you know, trying to change the world. Our first interview, you said, I've seen the future, I want to bring it to the mainstream. It's here. Yeah. It's hitting mainstream right now. Yeah. What's your take of the current situation of the ecosystem and it's, and its value? >>Yeah, so I, I have a quick question first. Should I look to you or look to the camera? Look to >>The camera or both? Whatever you, whatever you'd like. >>So I think it's, the ecosystem is definitely growing, which is very, very healthy. However, there is a side question there, which is what do you think of all the competition coming into the space? So five years ago when Cloudera was started was just Cloudera. There was no other commercial vendor trying to support or enable Hadoop in the, in the industry for enterprises. And today there is at least 10 of them trying to compete with us, right? And that includes big companies, established companies that decided, hey, we gonna start addressing the space, but includes many, many newcomers who like Hortonworks, who were founded over the last couple of years. That's a healthy thing. I mean, that's absolutely a sign of a growing market. If the market wasn't growing, if there wasn't money in the market, if there wasn't, if it was just hype, there wouldn't have been all of these new companies and new ventures showing up. That said, I never look at competition as something that worries me, that I'm afraid now or what's gonna happen to me, or that's normal. That's exactly what happens to successful companies. If you look at Red Hat, when Red Hat was launching with the Linux, they had 25 competitors or even more 30 competitors. That's when Red Hat was forming out. And today, even of these 25, 30 competitors, they still have six or seven still left. So I think it's a very, very healthy sign of the graph of this market and the maturity that's reaching. >>What do you think about some of the, the white spaces that are evolving? You guys have obviously been involved in a lot of deployments at Cloudera. Again, you're doing a lot of, lot of work with the top, top names and the clients that you have aren't usually disclosed cuz you really can't disclose them. What, what are you seeing right now as the white spaces for things to do in the Hado platform? >>It's a very, very good question. So first I can't talk about future, future roadmap. Right now we're becoming a big company at that level where we can't comment on future roadmaps. >>Ah, that's sinus sign of the >>Time. You're well media train, good to see they're doing a good job keeping you >>A, You want more information on that? I can connect you with a pt, >>Please. No, no, no, we're good. We're good. We'll get it outta you. But, >>But our vision, our vision for Cloudera from day one, like you were saying earlier, we saw the future, right? So our vision from from day one was really to build this data system where we can have detail of any type, whether that data is structured or unstructured or images, it doesn't matter. And then on top of that data run any type of workloads. That workload could be the initial genesis of Hado, which is map use, which is batch processing. But now as as we made many announcements through the last few years, we also now have Impala for interactive analytics as a workload. We have a very, very strong partner partnership with SaaS for doing machine learning and statistics as a workload. And a few weeks ago we announced search as another workload. So you have multiple types of workloads that can handle different types of problems that you have within your organization and bring all of these workloads to all of your data regardless of type. And that's the vision that we'll continue to deliver on. That's exactly what we're building going into the >>Future. So how's that fit in with yarn, right? We're hearing a lot at this conference about yarn, the ability to, you know, do more with less in a lot of the things that you typically hear with the enter within the enterprise. And, and so talk about that a little bit. >>Yarn is a very core part to our platform. In fact, yarn has been part of CDH four for more than a year now out in the, in the markets. So we did bring, we were one of the, I think we were the first vendor who brought yarn into a distribution of Hado out there. It's very, very fundamental to us because that is how we're gonna coordinate. We are gonna be using yarn to coordinate launching all of these different type of workloads. You're gonna have the map produce workload, which is very batch oriented. The Impala workload, which is very latency sensitive. The, the search workload, which is also very latency sensitive. The machine learning workload, which is more batch oriented, et cetera, et cetera. And yarn is a very, very central piece to helping us coordinate all of these different types of workloads onto the >>Platform. Cloudera has been a great citizen in the community also. You, you mentioned and, and we witnessed that your team create the industry. You guys were there, you took the chance, you were the first ones commercially funded by the venture capitalists, you know, then others will follow and I'll see huge ecosystem here. Yes. A lot of noise. A lot of people trying to get attention. So I got to ask you, because I want you to address this because I know it's been talked about in some of the other blogs is there's a lot of fud going on around who's doing what? Who's doing what, and in some cases maybe flat out, you know, misinformation and that happens in a growing market, you know, the elbows get sharp. Yes. So I want you share with the audience anything that you want say about the fud around what people say about Cloudera or about others or what you're doing. Just to clarify, cuz there has been, I mean I've gotten back channel information around, you know, not sure the committers this, and it's been, it's been well documented. There's a lot of fu out there. What, what would you say to the folks out there to clarify >>That? Yes, I, I would say that our focus should be to continue to work as a community, to push the platform forwards. I would say that at Cloudera we do a lot of contributions. Horton works definitely is one of the top contributors out there as well. I'll acknowledge that. So as many, many, many other companies and we wanna continue to see the platform evolve. I will stress though that at Cloudera we do have a number of the original project founders working at the company. So it's not just the, the contribution that we bring, but the fact that we have the founders of these projects working at Cloudera. And some of these projects actually were created at Cloudera from day one as opposed to created in some other company. And then you hire the employee and they work for you. So I gave you what examples from Cloudera dot cutting. >>He is the creator of Hudu dot Cutting is also the creator of Luine, which became solar, which is part of the search project that we launched recently. Dot Cutting wasn't with Cloudera from day one, right? So, so when he created these technologies, he actually was at Tia for example, when he created had he was at ta, wasn't at Cloudera. However, he now works for Cloudera. So we get that because now that cutting works for Cloudera. So that's one example. On the flip side, there is projects like Flume and Scoop that are now part of every single distribution out there. And flu and Scoop were both created at Calera. They were actually created inside of Cloudera. Yeah. So the key point is, and and that's what I would like all of the vendors out there that are trying to leverage had and get benefit about out Hadoop is please don't be just takers. >>There are some vendors out there who are just takers. Just wanna take from the open source, take from the open source and don't give back. Right? I'm not gonna name them, but there is a few of them out there. Please, please, please. I mean that that, that is very, very a selfish behavior. It's not gonna help the ecosystem in the long term. We would like to see you both take and give at the same time. So that would be my core message. And that's for example, like I thank Hortonworks because that's exactly what Hortonworks is doing. They're both giving and taking at the same >>Time. You guys have always been clear on that. Nobody, I mean here contribution to open source has been well documented and there's, there's no question about that. John and I have talked about it a lot that you guys help get it all started. And even Haak when we had 'em on a couple years ago, when Horton Works came to the market said, Hey, the more people work on an open source, the better. >>Yeah, >>Exactly. So yeah, it's always been, been your posture. You're not playing games there. Anyways, having said that, you you, you have a strategy to layer on top of that open source some of your own proprietary code. And so you have choices to make Yes. In terms of how you allocate those resources. So as an engineering manager, how do you allocate those resources in terms of, okay, what do we do for the community and what do we do for our own, you know, future because of the business model that we chose? How do you make those trade offs? >>Yes, that's a very, very good question. So first it's important to stress that our core platform, CDH, is open source. Everything we put in the core platform is open source. So for example, in Palo, which we launched very recently as a ga, now we launched beta last year, but now's ga is a hundred percent Apache license, a hundred percent open source search, which we announced very recently is also open source. So the platform itself, we're committing to everything in there to be open source. Now we believe fundamentally just from having lots of history in studying the open source markets from our ceo Mike Olson himself being one of the very first open source people in the world with, with sleepy cats, the company that he sold to Oracle before founding Cloudera from our investors, helping many other open source companies. To have a successful open co open source company, you need to have a very good engine between the business model that generates revenue and between the product that you are creating. If you don't have a good feedback loop there between these two, you won't be able to sustain the innovation to continue to push the, the boundaries of how good the product is. So we strongly believe in that if you are, if your product is literally a hundred percent open source, meaning both the management and every, there is nothing proprietary whatsoever inside of your products. I can't tell what that is. It's >>Taking a picture. >>Oh, sorry, I thought somebody was waiting >>For me. >>Sorry about that. >>It's a cheap signal. >>It >>Was like a's really good. >>I thought it's like a card of paper with some writing. You, >>You, you have a fan fans out there. They're storming the, the concert here. >>Okay, that's, that's good to hear. That's good to hear. Sorry about that interruption. So if, if, if you have everything a hundred percent open source, that creates two problems. First you have no differentiation whatsoever, meaning another big corporation without naming who the big corporations could be, we just can take everything you do, literally every single bit of source code you have and say, Hey, we can do it too. Come to us, don't work with those guys. Right? We have the latest, greatest things that they have. Why do you wanna continue to work with them? So no, no differentiation is number one, which is very dangerous. And number two, when it becomes, if, if it's a hundred percent open source and there is lots of other vendors able to take the art, the open source artifact and work with it, then it becomes now purely about maintenance and insurance on the products, which is a commodity product, which obviously the prices for that will go down to the ground and you won't be able to have this sustain this positive feedback effect between your business model and between your product code map and won't be able to build a long-lasting company. >>So that's why we do have a combination of open source artifacts and proprietary artifacts. Now our pro proprietary AR artifacts is always around the management of the system, right? So how do we manage the security of the system? How do we manage the, the data flow within the system? How do we manage the services inside the, of the system across all layers, right? Not just the Hado player but the edge based layer, the zookeeper layer, et cetera, et cetera. So that's where we focus our efforts going forward and that's how we differentiate ourself from our, from other vendors out there. Cloudera manager, Cloudera navigator are very unique to us. Nobody else has anything close to those capabilities out there. >>So it sounds like the contributions you make to open source are cultural of, of, in nature, I mean DNA of sorts of Right. And so you're, that's something that you guys do cuz you've always done it. Absolutely. And then the, the artifacts that are proprietary are essentially around rationalizing the revenue opportunity with the expense that you're gonna apply there and making a business case decided >>How to balance. That's that's one. And then two, the differentiation from other competitors. So these two things, Yes. >>Okay. >>I believe that's fundamental to business to open source business models. >>Yeah, I mean there are many open source business models, right? You can go pure service, you can go, like you said, you can totally bogart the code. >>There is no, there is no pure service open source model company that was able to build the longlasting surviving public company, never happened in history. They always get acquired because it becomes a commodity. I >>Mean, right. I mean, I mean and even ibm, right? >>Tom or I want to ask you about the storage thing. We were talking before camera, the, the hor and worst announcement storage you, what's your take on that? >>Which one? The Gluster, the one with Red Hats? Yes. Yes. So Red Hats and yeah, there has been recent news about Red Hat with, with Hor Works having a version of the Haddo platform that uses map use for the computation but uses Red Hat for the storage, right? So Red Hat has a new storage offering that was built based off of a company they acquired was called Guster. And that, that news was very, very surprising to me. And it, the reason why it was surprising, it's correlated also with a shift in messaging from, from Horton works. If you look at Horton Works last year at had Summit last year, one of the key messages that they deliver to us is that within the next five years or by 2015, the tagline back then by 2015, and you're doing research right now to see if I'm saying the right thing. By 2015, half the world data data will be on, will be stored in had would be stored in had. Yes. If you look today at the slides, it >>Doesn't say that it says within five years, >>Right? No, no, no. It says, well >>That was the second iteration was within five years. And now they say something >>Different. Now say they say within 2015 by, sorry, by 2015, half the world's data will be processed by Hado and instead of stored by Hado. And that's a very, very fundamental So >>It's a nuance. >>It's a, it's a very important >>Nuance. Well it's a big deal because yes, when I first saw that I said, Hmm, what does this all mean? And then it sounds 2015 sounds a little early. Yes. And now you're saying processed by, Okay that's different. >>Yes, exactly. And and the reason why now is we believe s GFS is very, very core to the had platform. S GFS is very core to had platform, the storage system of had we want. It's really the layer that Mid had with is more than anything else is how scalable, how reliable and how economical the sdfs storage layer is. So we, we really, I mean ask qu works and ask all the companies working in the, in the had community not to fragment at the storage layer. We need the storage for had to stay inside of had and not to fragment that out. That's very, very critical. >>Okay. So but so >>You're saying that they're in indicating through the gesture that, that they're not come out saying we're going to fragment Hgfs, but the way that this is position might signal >>No, no, no. The announcement, the announcement with Red Hat is >>That is the direct signal. It's >>Literally, we, you'll be able to run map produce directly on top of Red Hat storage instead of sdfs. >>Okay. So >>I >>Interpreted it, I interpret it as they were just hortonwork was hedging on its prediction, which I said Okay, I'll give 'em a break on that. You're saying it's something different, >>It's a shift in strategy potentially. Yeah. Which can be dangerous. It's shift in strategy. >>Is that a compliance issue? Cuz you know, the, the Dishon Hads poss Yeah. Red Hat does have a lot of enterprise customers. Yeah. So is that just maybe if >>Then invest in making had poss compliance, which actually by the way, we are as a community investing in that. Yeah. Yes. You must have. Yeah. So we are investing in adding compulsive poss compliance to had, we're investing in adding snapshots into had, which will be coming very, very soon overnight. >>Well, do you think that that pick a year, I don't care if it's 2015 2000, 22,000 whenever that the majority of the world's data will be running into do >>The majority of worse data that has to do with analytics. Yes. Okay. So so there is, >>So that is that >>Is it's very important, the caveat. Yes, exactly. Because there is lots of types of data that are not very suitable for, had at all. For example, that data storage for Oracle systems, for Oracle database systems. No, you wanna store that in an NetApp emc you don't wanna store that in Hao the, the, the, the, the data storage for streaming video files, right? For just streaming lots and lots of video files. No, you don't wanna store that indu. It's >>A huge >>Proportion of the data. Yeah. Which is a huge, huge >>Proportion of data files, in fact that could overwhelm the data. >>Yeah. So the new nuance, like I would say like I agree that the half thing but the half thing within the world of data for the purpose of analysis. >>Yeah. Okay. So that's, that's >>Narrow down the >>Yeah, okay. But it's a more reasonable, But I've, I >>Never, It's still a huge market by the way. It is. Yeah, >>It is. Yes. Okay. So, so what's next for you? A are you, you, you've gone on this, this journey, you start this company. You've, you've been traveling around like crazy working with customers. What's the next phase of aara do's, you know, career? >>What >>Do you want to have happen next? I mean, what, what do you, what excites you? What do you, what are you working on? >>Yeah, it's just to continue to grow cloud there to be the biggest company it can be. I mean, we want to be literally, we want be one of the very few companies that we're able to take an open source model and turn that into a large publicly traded corporation. >>So you've talked about that you guys brought a new CEO on Right. Look at the background of the ceo and it's, you know, clearly it's got some IPO chops. Yes. So that's, that's an aspiration that you guys have put forth. Okay. >>And you're outward facing now. So you're doing a lot of travel. Yes. So what, what, where have, what have your travels taken now? You've been in China, you obviously you've got a European office Yeah. Open. So what's going on internationally? Give us some sound bites of, of what's happening in the field. Yeah, >>So in, in internationally, I mean, Europe definitely is our next big focus right now. And we now have a big operation in Europe and we have an office presence in, in Europe and a big team down there. And it's growing very quickly. I would say Europe is about two years behind the US kind of like that's how the, how the growth usually matters. What's happening here. And yeah, so we, our, our next big market is Europe. We are looking at China. We don't have a big process in China right now. Japan, we have a big presence in Japan. Japan is growing very quickly. So yeah, I mean we're obviously Canada with the US growing very quickly as well. >>Great to have you on the cube again, for me personally and, and for, for Dave. And I wanna say thanks to Cloudera for some great support over the years. You guys have been fantastic. You know, I say it's built a great company. It's so hard to build a company. You guys have done a great job. I gotta ask you the final question because you did bring that first sound bite, which was, I saw the future, this is back when you guys were just in your B round in, in Palo Alto office, just ramping up, just starting to ramp what's next? What do you see as around the corner? Obviously we're on a trajectory right now. A lot of things gonna get done. Positive compliance, a lot of stuff's gonna fill in. The platform's gonna get stronger. Yeah. We think that open source will win. Yeah. Through all the democratization of open source. What's next? What's the, what's around the corner that you're watching personally that you're, that's interesting to you? A or around where this will take us? >>Yeah. So what, what's next is having this, having this vision become true. Having this future vision that, that you refer to become true. Meaning having a single platform that can store all of your data and that can, regardless of the type of that data, and allow you to extract value for different types of workloads, whether that be batch, interactive machine learning or search or more, right? There will be more things that will come to the platform, but how to bring your applications, all of your data applications, how to bring them to your data and all of your data as opposed to have the data go to them. >>And what are the landmines out there that you need to avoid Yes. In the industry and community needs to avoid to make that a reality. >>The, the key landmine, it's, it's a bit technical. The landmine is a bit technical, which is making sure that they, they are vision continues to evolve and that we have the capability to properly have a multi workload resource management system that allows me to run all of these type of workloads without having them step on each other's steps. That's the key key step going forward. And >>Of course, playing well together in the sandbox. And as always, competitive competition is good. And again, Hadup is doing great. Amma Aala, co-founder of Cloudera inside the Cube. This is Silicon Angle and Wiki Bond's exclusive coverage of ADU Summit here in Silicon Valley. Right back with our next guest after the short break.

>>Okay. We're back live in Silicon valley and San Jose, California for the continuous coverage of siliconangle.tv and have duke world 2012. This is ground zero for the alpha geeks in big data. Uh, just the tech elite. We call them tech athletes and, uh, we're excited to cover it on the ground. Extract the signal from the noise here. This is the cube, our flagship telecast. I'm joining my co-host Jeff Kelly from Wiki bond.org, the best analyst in the business. Jeff, welcome back for another segment. End of the day, day one loving every minute. Okay. We're here with our guest. Jack Norris is a cm of map bar Jack. Welcome back to the cube. You've been on a few times. Um, so you guys have some news. Yes. So let's get right to the news. So you guys are a player in the business, so share with your news, the folks. Excellent jump right in. >>So, uh, two big announcements today, we announced that Amazon is integrating map bar as part of their Lastic MapReduce service and both edition or, or free edition. M three is available as well as M five directly with Amazon, Amazon in the cloud. >>So what's the value proposition. Why would a customer say, all right, I want to do this in the cloud manpower, an Amazon cloud rather than doing it on premise. >>Okay. So let's start with, I mean, there's a lot of value propositions, all balled up into one here. Uh, first of all, in the cloud, it allows them to spin up very quickly. Within a couple minutes, you can get, uh, you know, hundreds of nodes available. Um, and, uh, and depending on where you're processing the data, if you've got a lot of data in the cloud already makes a lot of sense to do the Hadoop processing directly there. So that's, that's one area. A second is you might have an on-premise cloud deployment and need to have a disaster recovery. So map R provides point in time, snapshots, uh, as well as, as a white area replication. So you can use mirroring having Amazon available as a target is a huge advantage. And then there's also a third application area where you can do processing of the data in the cloud and then synchronize those results to an on-premise. So basically process where the data is combined the results into a cluster on premise. So you >>Don't have to move the raw data. Uh, >>On-premise actually, it's all about let's do the processing on the data. Well, you know, the whole, >>The value proposition and big data in general is let's not move, move data as little as possible. Yep. Uh, you know, so you bring the computation to the data, if you can. Uh, so what are your take on this event? I mean, we've got, uh, this is a, you know, the 4th of June summit, uh, you know, Hortonworks is now fully taken over the show and talk about what you see out here in terms of, uh, the other vendors that play. And, uh, just to kind of the attendees, the vibe you're seeing, >>Uh, it's a lot of excitement. I think a big difference between last year, which seemed to be very developer focused. We're seeing a lot of, a lot of presentations by customers. A lot of information was shared by our customers today. It was fun to see that, uh, comScore's shared, uh, shared their success. Boeing gap map is, uh, it was great for us. >>Fantastic. We look at Amazon, Amazon, first of all, is the gold standard for public cloud. Right? They've knocked it out of the park. Everyone knows Amazon. Um, but they've been criticized on the big data front because of the cycle times involve on. Um, and some developers and mean for web service spending up and down. No problem. Um, and we're seeing businesses like Netflix run on Amazon. So Amazon is not a stranger to running scale for cloud, but Hadoop has kind of been a klugey thing for Amazon. So I think, you know, talk about why Amazon and you guys is a good fit out to the market. The market reach is great. So you guys know and have a huge addressable market. Are you guys helping solve some of that complexity with the, uh, with the MapReduce side? What's, >>What's the core, I guess the first comment first response would be, I think every customer should have that type of Kluge. Uh, uh, they could have the success that Amazon has in Hadoop. They have a huge number of, of, uh, of Hadoop deployments have been very, very successful. I think, >>I mean, you know what I mean by it's natural, it's, cloogy everywhere right now. That's the problem. But Amazon has huge scale, um, and had not a natural fit. There >>Is not a natural fit >>For the data for the data component. And, uh, uh, the HBase for example, >>Component. So where were Amazons, you know, made it very frictionless is the ability to spin up Hadoop to do the analysis. The gap that was missing is some of the, the ha capabilities. The data protection features the disaster recovery, and, you know, we're map are now it gives options to those customers. You know, if they want those kinds of enterprise enterprise grade features, now they have an option within EMR. It can select a M five and, and get moving if they want a performance. And in NFS, they've got the M three options. >>Well, congratulations. I think it's a great deal for you guys and for Amazon customers. My question for you is, as you guys explore the enterprise ready equation, which has been a big topic this week, um, what does that mean to you guys? Cause it means different things to different people depends on where, how high up to OLTB do you go? Right? I mean, we're how far from batch to real time transactional, um, levels you go, I mean, low bash, no problem. But as you start to get more near real time, it's going to be a little bit different gray in this house used security HDFS. Yeah. >>Yeah. So, so duke represents the strategic platform, right? Deploying that in an organization, um, you know, moving from kind of an experimental kind of lab based to production environment creates a different set of feature requirements. How available is it? How easy is it to integrate, right? How do I kind of protect that information and how do I share it? So when we say enterprise grade, we mean you can have SLA, she can put the data there and, and be confident that the data will remain there, that you can have a point in time recovery for an application error or user mistake. Uh, you can have a disaster recovery features in place. And then the integration is about not recreating the wheel to get access to the information. So Hadoop is very powerful, but it requires interacting through an HDFS API. If you can leverage it like through map bar with NFS standard file based access standard ODBC access, open it up. >>So I can use a standard file browser applications to see and manipulate the data really opens up the use cases. And then finally, what we announced in two dot oh, was multitenancy features. So as you share that information, all of a sudden the SLA is of different groups and well, these guys need it immediately. And if you've got some low grade batch jobs are going to impact that. So you want the ability to protect, to isolate, to secure information, and basically have virtual clusters within a cluster. And those features are important to cloud, but they're also important to on-premise >>So great for the hybrid cloud environments out there. I mean, the multitenancy cracking the code on that. Exactly huge. I mean, that is basically, I mean, right now most enterprises are like private cloud because it's like, they're basically extension of their data center and you're seeing a lot more activity in the hybrid cloud as a gateway to the public cloud. So, >>And, and, you know, frankly, people are kind of struggling with in an experimental with Apache Hadoop and the other distributions, the policies are either at the individual file level or the whole cluster. And it all almost forced the creation of separate physical clusters, which kind of goes against the whole Hadoop concept. So the ability to manage it, a logical layer have separate volumes where you can apply policies to apply that applies to all the content underneath really kind of makes it much, much easier for administrators to kind of deal with these multiple use cases. >>Amazon, Amazon has always been one of those cases for the enterprise where it's been one of those and they've, this has been talked about for years, put the credit card down, go play on Amazon, but then bring it back into the it group for certification. And so I think this is a nice product for you guys to bring that comfort. You know, we're very >>Excited the enterprise saying, Hey, >>Come play in Amazon. It's Bulletproof enterprise. Ready? So congratulations. >>I wonder, can we talk, uh, talk use cases. So what are you seeing in terms of, uh, evolving use cases as, as, uh, duke continues to become more enterprise grade, uh, depending on your definition, uh, but how is that impacting what you're seeing in terms of, even if it's just, uh, you know, the, the, um, the mindset even people think now, okay, now it's enterprise grade, well, maybe, you know, in, in, depending on who you talk to, it's been that way for a bit, but what kind of, uh, use cases are you seeing develop now that it's kind of starting to gain acceptance? It's like, okay, we can trust our data is going to be there, et cetera. >>So th there's a huge range of use cases that, uh, different by industry, different by kind of dataset that's being used against everything from really a deep store where you can do analytics on it. So you're selecting the content to something that's very, very analytic machine learning intensive, where you're doing sophisticated clustering algorithms, uh, et cetera, um, where we've seen kind of an expansion of use cases are around real-time streaming and you get streaming data sets that are kind of entering into the cloud. And, um, some of the more mission, critical data moving beyond just maybe click stream data or things that if you happen to drop a few, you know, not a big deal, right. Versus the kind of trust the business type of content. >>Talk a little bit about the streaming, uh, aspects, uh, because of course, you know, we think of duke, we think of a batch system in terms of streaming data into Hadoop. You know, that's, that's a different, uh, that's something we don't, we haven't heard a lot about. So how do you guys approach that? >>So, uh, one of the artifacts of, of HDFS, which is a, is a distributed file system that scores in the underlying Linux file system, it's append only. So as an administrator, you decide, how frequently do I close the file item? I going to do that an hourly basis on it every eight hours, because you have to close the file for other applications to see the data that's been written. Right? So one of the innovations that, uh, that we pursued was to rewrite that create this dynamic read-write layer. So you can continue to write data in any application is seeing the latest data that's written. So you can Mount the cluster as if it's storage and just continue to write data. There really opens up what's, uh, what's possible companies like Informatica, they're all from a messaging product integrates directly in with, with Matt BARR and provides. >>So what kind of advantage does that provide to the end user? What w w translate that into real business value? Why, why is that important? >>Well, so one example is comScore, comScore handles 30 billion, uh, objects a day, uh, as they go out and try to measure the use of, of the web and being able to continually write and stream that information and scale and handle that in a real time and do analytics and turn around data faster, has tremendous business value to them. If they're stuck in a batch environment where the load times lengthen to the point where all of a sudden they can't keep up and they're actually reporting on, you know, old news. And I think the analogy is forecasting rain a day after it's wet. Isn't exactly valuable. >>Yeah. So you guys, obviously a great deal of the enterprise ready for Amazon, big story, big coup for the company. What's next for you. I want to ask that and make sure you get that out there on your agenda for the next year, but then I want you to take a step back a year, maybe a year and a half ago. Look back at how much has changed in this landscape. Um, share your perspective because the market has gone through an evolution where there's been a market opportunity, and then everyone goes, oh my God, it's bigger than we actually thought. I mean, Jeff, Kelly's a groundbreaking report about the $50 billion market is now being talked about as too low. So big data has absolutely opened up to a huge, and it's changed some of the tactics around strategies. So your strategy, Hortonworks strategy, even cloud era. So, and it's still evolving. So what's changed for the folks out there from a year and a half ago, a year ago to today, and then look out for the next 12 months. What's on your agenda. >>Well, if, if you look back, I think we've been fairly consistent. Um, uh, I'm, I'm not going to take credit for the vision of our CEO and CTO. Uh, but they recognized early on that Hadoop was, uh, was a strategic platform and to be a strategic platform that applied to the broadest number of use cases and organizations required some, some areas, uh, of innovation and particularly the how it, how it scaled, how it was managed, how you stored and protected the information needed a rearchitecture. And I think that, you know, architecture matters when you're going through a paradigm shift, having the right one in place creates this, this ability, you know, to speed innovation. And I think that's, if there's anything that's changed, I think it's the speed of innovation has even increased in the Hadoop community. I think it's, it's created a focus on these enterprise grade features on how do we store this valuable information and, and continue to explore. >>And I think one of the observations I'll make is that on that note is that it really focuses everyone to be just mind your own business and get the products out. You know what I'm saying? We've seen everyone, the product focus be the number one conversation. >>What we've seen is customers, you know, start and they expand rapidly. Some of that student data growth, but a lot of it is student more and more applications are being delivered and, and, uh, and, and the values kind of extracted from the hoop platform and success breeds success. Well, >>Congratulations for all your success, great win with Amazon web services and make that a little bit more easier, more robust, and more, more features for them and you, uh, more revenue for part of our, um, and I want to personally thank you for your support to the cube. Uh, we've expanded with a new studio B software for extra extra interviews, um, and wanna expand the conversation, thanks to your generous support. You can bring the independent coverage out to the market and, um, great community, thanks for helping us out. And we appreciate it. So thank you. Okay. Jack Dorsey with Matt bar, we'll be right back to wrap up day one with that. Jeff and I will give our analysis right at the short break.

>> Live from New York, it's theCUBE covering theCUBE New York City 2018. Brought to you by SiliconANGLE Media and its ecosystem partners. (techy music) >> Hello, everyone, welcome to this CUBE special presentation here in New York City for CUBENYC. I'm John Furrier with Dave Vellante. This is our ninth year covering the big data industry, starting with Hadoop World and evolved over the years. This is our ninth year, Dave. We've been covering Hadoop World, Hadoop Summit, Strata Conference, Strata Hadoop. Now it's called Strata Data, I don't know what Strata O'Reilly's going to call it next. As you all know, theCUBE has been present for the creation at the Hadoop big data ecosystem. We're here for our ninth year, certainly a lot's changed. AI's the center of the conversation, and certainly we've seen some horses come in, some haven't come in, and trends have emerged, some gone away, your thoughts. Nine years covering big data. >> Well, John, I remember fondly, vividly, the call that I got. I was in Dallas at a storage networking world show and you called and said, "Hey, we're doing "Hadoop World, get over there," and of course, Hadoop, big data, was the new, hot thing. I told everybody, "I'm leaving." Most of the people said, "What's Hadoop?" Right, so we came, we started covering, it was people like Jeff Hammerbacher, Amr Awadallah, Doug Cutting, who invented Hadoop, Mike Olson, you know, head of Cloudera at the time, and people like Abi Mehda, who at the time was at B of A, and some of the things we learned then that were profound-- >> Yeah. >> As much as Hadoop is sort of on the back burner now and people really aren't talking about it, some of the things that are profound about Hadoop, really, were the idea, the notion of bringing five megabytes of code to a petabyte of data, for example, or the notion of no schema on write. You know, put it into the database and then figure it out. >> Unstructured data. >> Right. >> Object storage. >> And so, that created a state of innovation, of funding. We were talking last night about, you know, many, many years ago at this event this time of the year, concurrent with Strata you would have VCs all over the place. There really aren't a lot of VCs here this year, not a lot of VC parties-- >> Mm-hm. >> As there used to be, so that somewhat waned, but some of the things that we talked about back then, we said that big money and big data is going to be made by the practitioners, not by the vendors, and that's proved true. I mean... >> Yeah. >> The big three Hadoop distro vendors, Cloudera, Hortonworks, and MapR, you know, Cloudera's $2.5 billion valuation, you know, not bad, but it's not a $30, $40 billion value company. The other thing we said is there will be no Red Hat of big data. You said, "Well, the only Red Hat of big data might be "Red Hat," and so, (chuckles) that's basically proved true. >> Yeah. >> And so, I think if we look back we always talked about Hadoop and big data being a reduction, the ROI was a reduction on investment. >> Yeah. >> It was a way to have a cheaper data warehouse, and that's essentially-- Well, what did we get right and wrong? I mean, let's look at some of the trends. I mean, first of all, I think we got pretty much everything right, as you know. We tend to make the calls pretty accurately with theCUBE. Got a lot of data, we look, we have the analytics in our own system, plus we have the research team digging in, so you know, we pretty much get, do a good job. I think one thing that we predicted was that Hadoop certainly would change the game, and that did. We also predicted that there wouldn't be a Red Hat for Hadoop, that was a production. The other prediction was is that we said Hadoop won't kill data warehouses, it didn't, and then data lakes came along. You know my position on data lakes. >> Yeah. >> I've always hated the term. I always liked data ocean because I think it was much more fluidity of the data, so I think we got that one right and data lakes still doesn't look like it's going to be panning out well. I mean, most people that deploy data lakes, it's really either not a core thing or as part of something else and it's turning into a data swamp, so I think the data lake piece is not panning out the way it, people thought it would be. I think one thing we did get right, also, is that data would be the center of the value proposition, and it continues and remains to be, and I think we're seeing that now, and we said data's the development kit back in 2010 when we said data's going to be part of programming. >> Some of the other things, our early data, and we went out and we talked to a lot of practitioners who are the, it was hard to find in the early days. They were just a select few, I mean, other than inside of Google and Yahoo! But what they told us is that things like SQL and the enterprise data warehouse were key components on their big data strategy, so to your point, you know, it wasn't going to kill the EDW, but it was going to surround it. The other thing we called was cloud. Four years ago our data showed clearly that much of this work, the modeling, the big data wrangling, et cetera, was being done in the cloud, and Cloudera, Hortonworks, and MapR, none of them at the time really had a cloud strategy. Today that's all they're talking about is cloud and hybrid cloud. >> Well, it's interesting, I think it was like four years ago, I think, Dave, when we actually were riffing on the notion of, you know, Cloudera's name. It's called Cloudera, you know. If you spell it out, in Cloudera we're in a cloud era, and I think we were very aggressive at that point. I think Amr Awadallah even made a comment on Twitter. He was like, "I don't understand "where you guys are coming from." We were actually saying at the time that Cloudera should actually leverage more cloud at that time, and they didn't. They stayed on their IPO track and they had to because they had everything betted on Impala and this data model that they had and being the business model, and then they went public, but I think clearly cloud is now part of Cloudera's story, and I think that's a good call, and it's not too late for them. It never was too late, but you know, Cloudera has executed. I mean, if you look at what's happened with Cloudera, they were the only game in town. When we started theCUBE we were in their office, as most people know in this industry, that we were there with Cloudera when they had like 17 employees. I thought Cloudera was going to run the table, but then what happened was Hortonworks came out of the Yahoo! That, I think, changed the game and I think in that competitive battle between Hortonworks and Cloudera, in my opinion, changed the industry, because if Hortonworks did not come out of Yahoo! Cloudera would've had an uncontested run. I think the landscape of the ecosystem would look completely different had Hortonworks not competed, because you think about, Dave, they had that competitive battle for years. The Hortonworks-Cloudera battle, and I think it changed the industry. I think it couldn't been a different outcome. If Hortonworks wasn't there, I think Cloudera probably would've taken Hadoop and making it so much more, and I think they wouldn't gotten more done. >> Yeah, and I think the other point we have to make here is complexity really hurt the Hadoop ecosystem, and it was just bespoke, new projects coming out all the time, and you had Cloudera, Hortonworks, and maybe to a lesser extent MapR, doing a lot of the heavy lifting, particularly, you know, Hortonworks and Cloudera. They had to invest a lot of their R&D in making these systems work and integrating them, and you know, complexity just really broke the back of the Hadoop ecosystem, and so then Spark came in, everybody said, "Oh, Spark's going to basically replace Hadoop." You know, yes and no, the people who got Hadoop right, you know, embraced it and they still use it. Spark definitely simplified things, but now the conversation has turned to AI, John. So, I got to ask you, I'm going to use your line on you in kind of the ask-me-anything segment here. AI, is it same wine, new bottle, or is it really substantively different in your opinion? >> I think it's substantively different. I don't think it's the same wine in a new bottle. I'll tell you... Well, it's kind of, it's like the bad wine... (laughs) Is going to be kind of blended in with the good wine, which is now AI. If you look at this industry, the big data industry, if you look at what O'Reilly did with this conference. I think O'Reilly really has not done a good job with the conference of big data. I think they blew it, I think that they made it a, you know, monetization, closed system when the big data business could've been all about AI in a much deeper way. I think AI is subordinate to cloud, and you mentioned cloud earlier. If you look at all the action within the AI segment, Diane Greene talking about it at Google Next, Amazon, AI is a software layer substrate that will be underpinned by the cloud. Cloud will drive more action, you need more compute, that drives more data, more data drives the machine learning, machine learning drives the AI, so I think AI is always going to be dependent upon cloud ends or some sort of high compute resource base, and all the cloud analytics are feeding into these AI models, so I think cloud takes over AI, no doubt, and I think this whole ecosystem of big data gets subsumed under either an AWS, VMworld, Google, and Microsoft Cloud show, and then also I think specialization around data science is going to go off on its own. So, I think you're going to see the breakup of the big data industry as we know it today. Strata Hadoop, Strata Data Conference, that thing's going to crumble into multiple, fractured ecosystems. >> It's already starting to be forked. I think the other thing I want to say about Hadoop is that it actually brought such great awareness to the notion of data, putting data at the core of your company, data and data value, the ability to understand how data at least contributes to the monetization of your company. AI would not be possible without the data. Right, and we've talked about this before. You call it the innovation sandwich. The innovation sandwich, last decade, last three decades, has been Moore's law. The innovation sandwich going forward is data, machine intelligence applied to that data, and cloud for scale, and that's the sandwich of innovation over the next 10 to 20 years. >> Yeah, and I think data is everywhere, so this idea of being a categorical industry segment is a little bit off, I mean, although I know data warehouse is kind of its own category and you're seeing that, but I don't think it's like a Magic Quadrant anymore. Every quadrant has data. >> Mm-hm. >> So, I think data's fundamental, and I think that's why it's going to become a layer within a control plane of either cloud or some other system, I think. I think that's pretty clear, there's no, like, one. You can't buy big data, you can't buy AI. I think you can have AI, you know, things like TensorFlow, but it's going to be a completely... Every layer of the stack is going to be impacted by AI and data. >> And I think the big players are going to infuse their applications and their databases with machine intelligence. You're going to see this, you're certainly, you know, seeing it with IBM, the sort of Watson heavy lift. Clearly Google, Amazon, you know, Facebook, Alibaba, and Microsoft, they're infusing AI throughout their entire set of cloud services and applications and infrastructure, and I think that's good news for the practitioners. People aren't... Most companies aren't going to build their own AI, they're going to buy AI, and that's how they close the gap between the sort of data haves and the data have-nots, and again, I want to emphasize that the fundamental difference, to me anyway, is having data at the core. If you look at the top five companies in terms of market value, US companies, Facebook maybe not so much anymore because of the fake news, though Facebook will be back with it's two billion users, but Apple, Google, Facebook, Amazon, who am I... And Microsoft, those five have put data at the core and they're the most valuable companies in the stock market from a market cap standpoint, why? Because it's a recognition that that intangible value of the data is actually quite valuable, and even though banks and financial institutions are data companies, their data lives in silos. So, these five have put data at the center, surrounded it with human expertise, as opposed to having humans at the center and having data all over the place. So, how do they, how do these companies close the gap? How do the companies in the flyover states close the gap? The way they close the gap, in my view, is they buy technologies that have AI infused in it, and I think the last thing I'll say is I see cloud as the substrate, and AI, and blockchain and other services, as the automation layer on top of it. I think that's going to be the big tailwind for innovation over the next decade. >> Yeah, and obviously the theme of machine learning drives a lot of the conversations here, and that's essentially never going to go away. Machine learning is the core of AI, and I would argue that AI truly doesn't even exist yet. It's machine learning really driving the value, but to put a validation on the fact that cloud is going to be driving AI business is some of the terms in popular conversations we're hearing here in New York around this event and topic, CUBENYC and Strata Conference, is you're hearing Kubernetes and blockchain, and you know, these automation, AI operation kind of conversations. That's an IT conversation, (chuckles) so you know, that's interesting. You've got IT, really, with storage. You've got to store the data, so you can't not talk about workloads and how the data moves with workloads, so you're starting to see data and workloads kind of be tossed in the same conversation, that's a cloud conversation. That is all about multi-cloud. That's why you're seeing Kubernetes, a term I never thought I would be saying at a big data show, but Kubernetes is going to be key for moving workloads around, of which there's data involved. (chuckles) Instrumenting the workloads, data inside the workloads, data driving data. This is where AI and machine learning's going to play, so again, cloud subsumes AI, that's the story, and I think that's going to be the big trend. >> Well, and I think you're right, now. I mean, that's why you're hearing the messaging of hybrid cloud and from the big distro vendors, and the other thing is you're hearing from a lot of the no-SQL database guys, they're bringing ACID compliance, they're bringing enterprise-grade capability, so you're seeing the world is hybrid. You're seeing those two worlds come together, so... >> Their worlds, it's getting leveled in the playing field out there. It's all about enterprise, B2B, AI, cloud, and data. That's theCUBE bringing you the data here. New York City, CUBENYC, that's the hashtag. Stay with us for more coverage live in New York after this short break. (techy music)

(techno music) >> Hey welcome back everybody, Jeff Frick here with theCUBE. We're in Austin, Texas at the Dell EMC HPC and AI Innovation Lab. As you can see behind me, there's racks and racks and racks of gear, where they build all types of vessel configurations around specific applications, whether its Oracle or S.A.P. And more recently a lot more around artificial intelligence, whether it's machine learning, deep learning, so it's a really cool place to be. We're excited to be here. And our next guest is Bala Chandrasekaran. He is in the technical staff as a systems engineer. Bala, welcome! >> Thank you. >> So how do you like playing with all these toys all day long? >> Oh I love it! >> I mean you guys have literally everything in there. A lot more than just Dell EMC gear, but you've got switches and networking gear-- >> Right. >> Everything. >> And not just the gear, it's also all the software components, it's the deep learning libraries, deep learning models, so a whole bunch of things that we can get to play around with. >> Now that's interesting 'cause it's harder to see the software, right? >> Exactly right. >> The software's pumping through all these machines but you guys do all types of really, optimization and configuration, correct? >> Yes, we try to make it easy for the end customer. And the project that I'm working on, machine learning for Hadoop, we try to make things easy for the data scientists. >> Right, so we got all the Hadoop shows, Hadoop World, Hadoop Summit, Strata, Big Data NYC, Silicone Valley, and the knock on Hadoop is always it's too hard, there aren't enough engineers, I can't get enough people to do it myself. It's a cool open source project, but it's not that easy to do. You guys are really helping people solve that problem. >> Yes and what you're saying is true for the infrastructure guys. Now imagine a data scientist, right? So Hadoop cluster accessing it, securing it, is going to be really tough for them. And they shouldn't be worried about it. Right? They should be focused on data science. So those are some of the things that we try to do for them. >> So what are some of the tips and tricks as you build these systems that throw people off all the time that are relatively simple things to fix? And then what are some of the hard stuff where you guys have really applied your expertise to get over those challenges? >> Let me give you a small example. So this is a new project A.I. we hired data scientists. So I walk the data scientist through the lab. He looked at all he cluster and he pulled me aside and said hey you're not going to ask me to work on these things, right? I have no idea how to do these things. So that kind of gives you a sense of what a data scientist should focus on and what what they shouldn't focus on. So some of the things that we do, and some of the things that are probably difficult for them is all the libraries that are needed to run their project, the conflicts between libraries, the dependencies between them. So one of the things that we do deliver this pre-configured engine that you can readily download into our product and run. So data scientist don't have to worry about what library I should use. >> Right. >> They have to worry about the models and accuracy and whatever data science needs to be done, rather than focusing on the infrastructure. >> So you not only package the hardware and the systems, but you've packaged the software distribution and all the kind of surrounding components of that as well. >> Exactly right. Right. >> So when you have the data scientists here talking about the Hadoop cluster, if they didn't want to talk about the hardware and the software, what were you helping them with? How did you engage with the customers here at the lab? >> So the example that I gave is for the data scientist that we newly hired for our team so we had to set up environments for them. so that was the example, but the same thing applies for a customer as well. So again to help them in solving the problem we tried to package some of the things as part of our product and deliver it to them so it's easy for them to deploy and get started on things. >> Now the other piece that's included and again is not in this room is the services -- >> Right. >> And the support so you guys have a full team of professional services. Once you configure and figure out what the optimum solution is for them then you got a team that can actually go deploy it at their actual site. >> So we have packaged things even for our services. So the services would go to the customer side. They would apply the solution and download and deploy our packages and be able to demonstrate how easy it is to think of them as tutorials if you like. So here are the tutorials. Here's how you run various models. So here's how easy it is for you to get started. So that's what they would train the customer on. So there's not just the deployment piece of it but just packaging things for them so they can show customers how to get started quickly, how everything works and kind of of give a green check mark if you will. >> So what are some of your favorite applications that people are using these things for? Do you get involved in the applications stack on the customer side? What are some of the fun use cases that people use in your technology to solve? >> So for the application my project is about mission learning on Hadoop via packaging Cloudera's CDSW that's Cloudera Data Science Workbench as part of the product. So that allows data science access to the Hadoop cluster and abstracting the complexities of the cluster. So they can access the cluster. They can access the data. They can have security without worrying about all the intricacies of the cluster. In addition to that they can create different projects, have different libraries in different projects. So they don't have to conflict with each other and also they can add users to it. They can work collaboratively. So basically choose to help data scientists, software developers, do their job and not worry about the infrastructure. >> Right. >> They should not be. >> Right great. Well Bala it's pretty exciting place to work. I'm sure you're having a ball. >> Yes I am thank you. >> All right. Well thanks for taking a few minutes with us and really enjoyed the conversation. >> I appreciate it thank you. All right he's Bala. I'm Jeff. You're watching theCUBE from Austin, Texas at the Dell EMC High Performance Computing and Artificial Intelligence Labs. Thanks for watching. (techno music)

(energetic electronic music) >> Hey, welcome back everybody. Jeff Frick here with The Cube. We're in a very special place. We're in Austin, Texas at the Dell EMC HPC and AI Innovation Lab. High performance computing, artificial intelligence. This is really where it all happens. Where the engineers at Dell EMC are putting together these ready-made solutions for the customers. They got every type of application stack in here, and we're really excited to have our next guest. He's right in the middle of it, he's Michael Bennett, Senior Principal Engineer for Dell EMC. Mike, great to see you. >> Great to see you too. >> So you're working on one particular flavor of the AI solutions, and that's really machine learning with Hadoop. So tell us a little bit about that. >> Sure yeah, the product that I work on is called the Ready Solution for AI Machine Learning with Hadoop, and that product is a Cloudera Hadoop distribution on top of our Dell powered servers. And we've partnered with Intel, who has released a deep learning library, called Big DL, to bring both the traditional machine learning capabilities as well as deep learning capabilities to the product. Product also adds a data science workbench that's released by Cloudera. And this tool allows the customer's data scientists to collaborate together, provides them secure access to the Hadoop cluster, and we think all-around makes a great product to allow customers to gain the power of machine learning and deep learning in their environment, while also kind of reducing some of those overhead complexities that IT often faces with managing multiple environments, providing secure access, things like that. >> Right, cause the big knock always on Hadoop is that it's just hard. It's hard to put in, there aren't enough people, there aren't enough experts. So you guys are really offering a pre-bundled solution that's ready to go? >> Correct, yeah. We've built seven or eight different environments going in the lab at any time to validate different hardware permutations that we may offer of the product as well as, we've been doing this since 2009, so there's a lot of institutional knowledge here at Dell to draw on when building and validating these Hadoop products. Our Dell services team has also been going out installing and setting these up, and our consulting services has been helping customers fit the Hadoop infrastructure into their IT model. >> Right, so is there one basic configuration that you guys have? Or have you found there's two or three different standard-use cases that call for two or three different kinds of standardized solutions? >> We find that most customers are preferring the R7-40XC series. This platform can hold 12 3 1/2" form-factor drives in the front, along with four in the mid-plane, while still providing four SSDs in the back. So customers get a lot of versatility with this. It's also won several Hadoop benchmarking awards. >> And do you find, when you're talking to customers or you're putting this together, that they've tried themselves and they've tried to kind of stitch together and cobble together the open-source proprietary stuff all the way down to network cards and all this other stuff to actually make the solution come together? And it's just really hard, right? >> Yeah, right exactly. What we hear over and over from our product management team is that their interactions with customers, come back with customers saying it's just too hard. They get something that's stable and they come back and they don't know why it's no longer working. They have customized environments that each developer wants for their big data analytics jobs. Things like that. So yeah, overall we're hearing that customers are finding it very complex. >> Right, so we hear time and time again that same thing. And even though we've been going to Hadoop Summit and Hadoop World and Stratus, since 2010. The momentum seems to be a little slower in terms of the hype, but now we're really moving into heavy-duty real time production and that's what you guys are enabling with this ready-made solution. >> So with this product, yeah, we focused on enabling Apache Spark on the Hadoop environment. And that Apache Spark distributed computing has really changed the game as far as what it allows customers to do with their analytics jobs. No longer are we writing things to disc, but multiple transformations are being performed in memory, and that's also a big part of what enables the big DL library that Intel released for the platform to train these deep-learning models. >> Right, cause the Sparks enables the real-time analytics, right? Now you've got streaming data coming into this thing, versus the batch which was kind of the classic play of Hadoop. >> Right and not only do you have streaming data coming in, but Spark also enables you to load your data in memory and perform multiple operations on it. And draw insights that maybe you couldn't before with traditional map-reduce jobs. >> Right, right. So what gets you excited to come to work every day? You've been playing with these big machines. You're in the middle of nerd nirvana I think-- >> Yeah exactly. >> With all of the servers and spin-discs. What gets you up in the morning? What are you excited about, as you see AI get more pervasive within the customers and the solutions that you guys are enabling? >> You know, for me, what's always exciting is trying new things. We've got this huge lab environment with all kinds of lab equipment. So if you want to test a new iteration, let's say tiered HGFS storage with SSDs and traditional hard drives, throw it together in a couple of hours and see what the results are. If we wanted to add new PCIE devices like FPGAs for the inference portion the deep-learning development we can put those in our servers and try them out. So I enjoy that, on top of the validated, thoroughly-worked-through solutions that we offer customers, we can also experiment, play around, and work towards that next generation of technology. >> Right, 'cause any combination of hardware that you basically have at your disposal to try together and test and see what happens? >> Right, exactly. And this is my first time actually working at a OEM, and so I was surprised, not only do we have access to anything that you can see out in the market, but we often receive test and development equipment from partners and vendors, that we can work with and collaborate with to ensure that once the product reaches market it has the features that customers need. >> Right, what's the one thing that trips people up the most? Just some simple little switch configuration that you think is like a minor piece of something, that always seems to get in the way? >> Right, or switches in general. I think that people focus on the application because the switch is so abstracted from what the developer or even somebody troubleshooting the system sees, that oftentimes some misconfiguration or some typo that was entered during the switch configuration process that throws customers off or has somebody scratching their head, wondering why they're not getting the kind of performance that they thought. >> Right, well that's why we need more automation, right? That's what you guys are working on. >> Right yeah exactly. >> Keep the fat-finger typos out of the config settings. >> Right, consistent reproducible. None of that, I did it yesterday and it worked I don't know what changed. >> Right, alright Mike. Well thanks for taking a few minutes out of your day, and don't have too much fun playing with all this gear. >> Awesome, thanks for having me. >> Alright, he's Mike Bennett and I'm Jeff Frick. You're watching The Cube, from Austin Texas at the Dell EMC High Performance Computing and AI Labs. Thanks for watching. (energetic electronic music)

>> Live from San Jose, in the heart of Silicon Valley, it's theCube. Covering DataWorks Summit 2018. Brought to you by Hortonworks. >> Welcome back to day two of theCube's live coverage of DataWorks here in San Jose, California. I'm your host, Rebecca Knight along with my co-host James Kobielus. James, it's great to be here with you in the hosting seat again. >> Day two, yes. >> Exactly. So here we are, this conference, 2,100 attendees from 32 countries, 23 industries. It's a relatively big show. They do three of them during the year. One of the things that I really-- >> It's a well-established show too. I think this is like the 11th year since Yahoo started up the first Hadoop summit in 2008. >> Right, right. >> So it's an established event, yeah go. >> Exactly, exactly. But I really want to talk about Hortonworks the company. This is something that you had brought up in an analyst report before the show started and that was talking about Hortonworks' cash flow positivity for the first time. >> Which is good. >> Which is good, which is a positive sign and yet what are the prospects for this company's financial health? We're still not seeing really clear signs of robust financial growth. >> I think the signs are good for the simple reason they're making significant investments now to prepare for the future that's almost inevitable. And the future that's almost inevitable, and when I say the future, the 2020s, the decade that's coming. Most of their customers will shift more of their workloads, maybe not entirely yet, to public cloud environments for everything they're doing, AI, machine learning, deep learning. And clearly the beneficiaries of that trend will be the public cloud providers, all of whom are Hortonworks' partners and established partners, AWS, Microsoft with Azure, Google with, you know, Google Cloud Platform, IBM with IBM Cloud. Hortonworks, and this is... You know, their partnerships with these cloud providers go back several years so it's not a new initiative for them. They've seen the writing on the wall practically from the start of Hortonworks' founding in 2011 and they now need to go deeper towards making their solution portfolio capable of being deployable on-prem, in cloud, public clouds, and in various and sundry funky combinations called hybrid multi-clouds. Okay, so, they've been making those investments in those partnerships and in public cloud enabling the Hortonworks Data Platform. Here at this show, DataWorks 2018 here in San Jose, they've released the latest major version, HDP 3.0 of their core platform with a lot of significant enhancements related to things that their customers are increasingly doing-- >> Well I want to ask you about those enhancements. >> But also they have partnership announcements, the deep ones of integration and, you know, lift and shift of the Hortonworks portfolio of HDP with Hortonworks DataFlow and DataPlane Services, so that those solutions can operate transparently on those public cloud environments as the customers, as and when the customers choose to shift their workloads. 'Cause Hortonworks really... You know, like Scott Gnau yesterday, I mean just laid it on the line, they know that the more of the public cloud workloads will predominate now in this space. They're just making these speculative investments that they absolutely have to now to prepare the way. So I think this cost that they're incurring now to prepare their entire portfolio for that inevitable future is the right thing to do and that's probably why they still have not attained massive rock and rollin' positive cash flow yet but I think that they're preparing the way for them to do so in the coming decade. >> So their financial future is looking brighter and they're doing the right things. >> Yeah, yes. >> So now let's talk tech. And this is really where you want to be, Jim, I know you. >> Oh I get sleep now and I don't think about tech constantly. >> So as you've said, they're really doing a lot of emphasis now on their public cloud partnerships. >> Yes. >> But they've also launched several new products and upgrades to existing products, what are you seeing that excites you and that you think really will be potential game changers? >> You know, this is geeky but this is important 'cause it's at the very heart of Hortonworks Data Platform 3.0, containerization of more... When you're a data scientist, and you're building a machine learning model using data that's maintained, and is persisted, and processed within Hortonworks Data Platform or any other big data platform, you want the ability increasingly for developing machine learning, deep learning, AI in general, to take that application you might build while you're using TensorFlow models, that you build on HDP, they will containerize it in Docker and, you know, orchestrate it all through Kubernetes and all that wonderful stuff, and deploy it out, those AI, out to increasingly edge computing, mobile computing, embedded computing environments where, you know, the real venture capital mania's happening, things like autonomous vehicles, and you know, drones, and you name it. So the fact is that Hortonworks has made that in many ways the premier new feature of HDP 3.0 announced here this week at the show. That very much harmonizes with what their partners, where their partners are going with containerization of AI. IBM, one of their premier partners, very recently, like last month, I think it was, announced the latest version of IBM, what do they call it, IBM Cloud Private, which has embedded as a core feature containerization within that environment which is a prem-based environment of AI and so forth. The fact that Hortonworks continues to maintain close alignment with the capabilities that its public cloud partners are building to their respective portfolios is important. But also Hortonworks with its, they call it, you know, a single pane of glass, the DataPlane Services for metadata and monitoring and governance and compliance across this sprawling hybrid multi-cloud, these scenarios. The fact that they're continuing to make, in fact, really focusing on deep investments in that portfolio, so that when an IBM introduces or, AWS, whoever, introduces some new feature in their respective platforms, Hortonworks has the ability to, as it were, abstract above and beyond all of that so that the customer, the developer, and the data administrator, all they need to do, if they're a Hortonworks customer, is stay within the DataPlane Services and environment to be able to deploy with harmonized metadata and harmonized policies, and harmonized schemas and so forth and so on, and query optimization across these sprawling environments. So Hortonworks, I think, knows where their bread is buttered and it needs to stay on the DPS, DataPlane Services, side which is why a couple months ago in Berlin, Hortonworks made a, I think, the most significant announcement of the year for them and really for the industry, was that they announced the Data Steward Studio in Berlin. Tech really clearly was who addressed the GDPR mandate that was coming up but really did a stewardship as an end-to-end workflow for lots of, you know, core enterprise applications, absolutely essential. Data Steward Studio is a DataPlane Service that can operate across multi-cloud environments. Hortonworks is going to keep on, you know... They didn't have a DPS, DataPlane Services, announcements here in San Jose this week but you can best believe that next year at this time at this show, and in the interim they'll probably have a number of significant announcements to deepen that portfolio. Once again it's to grease the wheels towards a more purely public cloud future in which there will be Hortonworks DNA inside most of their customers' environments going forward. >> I want to ask you about themes of this year's conference. The thing is is that you were in Berlin at the last big Hortonworks DataWorks Summit. >> (speaks in foreign language) >> And really GDPR dominated the conversations because the new rules and regulations hadn't yet taken effect and companies were sort of bracing for what life was going to be like under GDPR. Now the rules are here, they're here to stay, and companies are really grappling with it, trying to understand the changes and how they can exist in this new regime. What would you say are the biggest themes... We're still talking about GDPR, of course, but what would you say are the bigger themes that are this week's conference? Is it scalability, is it... I mean, what would you say we're going, what do you think has dominated the conversations here? >> Well scalability is not the big theme this week though there are significant scalability announcements this week in the context of HDP 3.0, the ability to persist in a scale-out fashion across multi-cloud, billions of files. Storage efficiency is an important piece of the overall announcement with support for erasure coding, blah blah blah. That's not, you know, that's... Already, Hortonworks, like all of their cloud providers and other big data providers, provide very scalable environments for storage, workload management. That was not the hugest, buzzy theme in terms of the announcements this week. The buzz of course was HDP 3.0. Containerization, that's important, but you know, we just came out of the day two keynote. AI is not a huge focus yet for a lot of the Hortonworks customers who are here, the developers. They're, you know, most of their customers are not yet that far along in their deep learning journeys and whatever but they're definitely going there. There's plenty of really cool keynote discussions including the guy with the autonomous vehicles or whatever that, the thing we just came out of. That was not the predominant theme this week here in terms of the HDP 3.0. I think what it comes down to is that with HDP 3.0... Hive, though you tend to take it for granted, it's been in Hadoop from the very start, practically, Hive is now a full enterprise database and that's the core, one of the cores, of HDP 3.0. Hive itself, Hive 3.0 now is its version, is ACID compliant and that may be totally geeky to the most of the world but that enables it to support transactional applications. So more big data in every environment is supporting more traditional enterprise application, transactional applications that require like two-phase commit and all that goodness. The fact is, you know, Hortonworks have, from what I can see, is the first of the big data vendors to incorporate those enhancements to Hive 3.0 because they're so completely tuned in to the Hive environment in terms of a committer. I think in many ways that is the predominant theme in terms of the new stuff that will actually resonate with the developers, their customers here at the show. And with the, you know, enterprises in general, they can put more of their traditional enterprise application workloads on big data environments and specifically, Hortonworks hopes, its HDP 3.0. >> Well I'm excited to learn more here at the on theCube with you today. We've got a lot of great interviews lined up and a lot of interesting content. We got a great crew too so this is a fun show to do. >> Sure is. >> We will have more from day two of the.

>> Announcer: From San Mateo, California, it's theCUBE! Covering SnapLogic, Innovation Day, 2018. Brought to you by SnapLogic. >> Hey welcome back everybody, Jeff Frick here with theCUBE. We are in San Mateo, at what they call the crossroads, it's 92 and 101. If you're coming by and probably sitting in a traffic, look up and you'll see SnapLogic. It's their new offices. We're really excited to be here for Innovation Day. We're excited to have this CTO, James Markarian. James, great to see you and I guess, we we last talked was a couple years ago in New York City. >> Yeah that's right, and why was I there? It was like a big data show. >> That's right. >> And we we are two years later talking about big data. >> Big data, big data is fading a little bit, because now big data is really an engine, that's powering this new thing that's so exciting, which is all about analytics, and machine learning, and we're going to eventually stop saying artificial intelligence and say augmented intelligence, 'cause there's really nothing artificial about it. >> Yeah and we might stop saying big data and just talk about data because it's becoming so ubiquitous. >> Jeff: Right. >> I know that big data, it's not necessarily going away but it's sort of how we're thinking about handling it is, like kind of evolved over time, especially in the last couple of years. >> Right. >> That's what we're kind of seeing from our customers. >> 'Cause there's kind of an ingredient now, right? It's no longer this new shiny object now. It's just part of the infrastructure that helps you get everything else done. >> Yeah, and I think when you think about it, from like, an enterprise point of view, that that shift is going from experimentation to operationalizing. I think that the things you look for in experimentation, there's like, one set of things here looking for proving out the overall value, regardless maybe of cost and uptime and other things and as you operationalize you start thinking about other considerations that obviously Enterprise IT has to think about. >> Right, so if you think back to like, Hadoop Summit and Hadoop World who were first cracking their teeth, like in 2010 or around that time frame, one of the big discussions that always comes up and that was before kind of the rise of public cloud, you know which has really taken off over the last several years, there's this kind of ongoing debate between, do you move the data to the compute or do you move the compute to the data? There was always like, this monster data gravity issue which was almost insurmountable and many would say, oh, you're never going to get all your data into the cloud. It's just way too hard and way too expensive. But, now Amazon has Snowball and Snowball isn't big enough. They actually had a diesel truck that'll come and help you come move your data. Amazon rolled that thing across the stage a couple of years ago. The data gravity thing seems to be less and if you think of a world with infinite compute, infinite stored, infinite networking asyndetically approaching zero, not necessarily good news for some vendors out there but that's a world that we're eventually getting to that changes the way that you organize all this stuff. >> Yeah, I think so and so much has changed. I was fortunate to be one of the early speakers, like I used to do Worlds and everything, and I was adamantly proclaiming you know, the destiny of Hadoop as bright and shiny and there's this question about what really happened. I think that there's a kind of a few different variables that kind of shifted at the same time. One, is of course, this like glut of computing in the cloud happened and there are so many variables moving at once. It's like, How much time do you have Jeff? >> Ask them to get a couple more drinks for us. >> Seeing our lovely new headquarters here and one of the things is that there is no big data center. We have a little closet with some of the servers we keep around but mostly, everything we do is on Amazon. You're even looking at things like, commercial real estate is changing because I don't need all the cooling and the power and the space for my data center that I once had. >> Jeff: Right, right. >> I become a lot more space efficient than I used to be and so the cloud is really kind of changing everything. On the data side, you mention this like, interesting philosophical shift, going from I couldn't possibly do it in the cloud to why in the world would we not do things in the cloud. Maybe the one stall word in there being some fears about security. Obviously there's been a lot of breaches. I think that there's still a lot of introspection everyone needs to do about, are my on premise systems actually more secure than some of these cloud providers? It's really not clear that we know the answer to that. In fact, we suspect that some of the cloud providers are actually more secure because they are professionals about it and they have the best practice. >> And a whole lot of money. >> The other thing that happened that you didn't mention, that's approaching infinity and we're not quite there yet, is interconnect speeds. So it used to be the case that I have a bunch of mainframes and I have a tier rating system and I have a high speed interconnect that puts the two together. Now with fiber networks and just in general, you can run super high speed, like WAN. Especially if you don't care quite as much about latency. So if 500 millisecond latency is still okay with you. >> Great. >> You can do a heck of a lot and move a lot to the cloud. In fact, it's so good, that we went from worrying, could I do this in the cloud at all to well, why wouldn't I do somethings in Amazon and some things in Microsoft and some things in Google? Even if it meant replicating my data across all these environments. The backdrop for some of that is, we had a lot of customers and I was thinking that people would approach it this way, they would install on premise Hadoop, whether it's like Apache or Cloud Air or the other vendors and I would hire a bunch of folks that are the administrators and retire terra data and I'm going to put all my ETL jobs on there, etc. It turned out to be a great theory and the practice is real for some folks but it turned out to be moving a lot of things to kind of shifting sands because Hadoop was evolving at the time. A lot of customers were putting a lot of pressure on it, operational pressure. Again, moving from experimentation phase over to like, operational phase. >> Jeff: Right, right. >> When you don't have the uptime guarantee and I can't just hire somebody off the street to administer this, it has to be a very sharp, knowledgeable person that's very expensive, people start saying, what am I really getting from this and can I just dump it all in S3 and apply a bunch of technology there and let Amazon worry about keeping this thing up and running? People start to say, I used to reject that idea and now it's sounding like a very smart idea. >> It's so funny we talk about people processing tech all the time, right? But they call them tech shows, they don't call them people in process shows. >> Right. >> At least not the ones we go to but time and time again I remember talking to some people about the Hadoop situation and there's just like, no Hadoop people. Sometimes technology all day long. There just aren't enough people with the skills to actually implement it. It's probably changed now but I remember that was such a big problem. It's funny you talk about security and cloud security. You know, at AWS, on Tuesday night of Reinvent, they have a special, kind of a technical keynote speak and like, James Hamilton would go. In the amount of resources, and I just remember one talk he gave just on their cabling across the ocean, and the amount of resources that he can bring to bear, relative to any individual company, is so different; much less a mid-tier company or a small company. I mean, you can bring so much more resources, expertise and knowledge. >> Yeah, the economy is a scale, their just there. >> They're just crazy. >> That's right and that why you know, you sort of assume that the cloud sort of, eventually eats everything. >> Right, right. >> So there's no reason to believe this won't be one of those cases. >> So you guys are getting Extreme. So what is Snaplogic Extreme? >> Well, Snaplogic Extreme is kind of like a response to this trend of data moving from on premise to the cloud and there are some interesting dynamics of that movement. First of all, you need to get data into the cloud, first of all and we've been doing that for years. Connect to everything, dump it in S3, ADLS, etc. No problem. The thing we're seeing with cloud computing is like, there's another interesting shift. Not only is it kind of like mess for less, and let Amazon manage all this, and I probably refer to Amazon more than other vendors would appreciate. >> Right, right. They're the leaders so let's call a spade a spade. >> Yeah. >> Certainly Google and Microsoft are out there as well so those are the top three and we've acknowledged that. >> One of the interesting things about it is that you couldn't really adequately achieve on premises is the burstiness of your compute. I run at a steady state where I need, you know, 10 servers or a 100 servers, but every once in a while, I need like, 1,000 or 10,000 servers to apply to something. So what's the on premise model? Rack and stack, 10,000 machines, and it's like waiting for the great pumpkin, waiting for that workload to come that I've been waiting months and months for and maybe it never comes but I've been paying for it. I paid for a software license for the thing that I need to run there. I'm paying for the cabling and the racking and everything and the person administering. Make sure the disks are all operating in the case where it gets used. Now, all of a sudden, we are taking Amazon and they're saying, hey, pay us for what you're using. You can use reserved pricing and pay a lower rate for the things you might actually care about on a consistent basis but then I'm going to allow you to spike, and I'll just run the meter. So this has caused software vendors like us, to look at the way we charge and the way that we deploy our resources and say, hey, that's a very good model. We want to follow that and so we introduced Snaplogic Extreme, which has a few different components. Basically, it enables us to operate in these elastic environments, shift our thinking in pricing so that we don't think about like, node based or god forbid, core based pricing and say like, hey, basically pay us for what you do with your data and don't worry about how many servers it's running on. Let Snaplogic worry about spinning up and spinning down these machines because a lot of these workloads are data integration or application workloads that we know lots about. >> Right. >> So first of all, we manage these ephemeral, what we call ephemeral or elastic clusters. Second of all, the way that we distribute our workload is by generating Spark code currently. We use the same graphic environment that you use for everything but instead of running on our engines, we kind of spit out Spark code on the end that takes advantage of the massive scale out potential for these ephemeral environments. >> Right. >> We've also kind of built this in such a way that it's Spark today but it could be like, Native or some other engine like Flank or other things that come up. We really don't care like what back end engine actually is as long as it can run certain types of data oriented jobs. It's actually like lots of things in one. We combine out data acquisition and distribution capability with this like, massive elastic scale out capability. >> Yeah, it's unbelievable how you can spin that up and then of course, most people forget you need to spin it down after the event. >> James: Yeah, that's right. >> We talked to a great vendor who talked about, you know, my customer spends no money with me on the weekend, zero. >> James: Right. >> And I'm thrilled because they're not using me. When they do use me, then they're buying stuff. I think what's really interesting is how that changes. Also, your relationship with your customer. If you have a recurring revenue model, you have to continue to deliver a value. You have to stay close to your customer. You have to stay engaged because it's not a one time pop and then you send them the 15% or 20% maintenance bill. It's really this ongoing relationship and they're actually gaining value from your products each and every time you use that. It's a very different way. >> Yeah, that's right. I think it creates better relationships because you feel like, what we do is unproportionate to what they do and vise versa, so it has this fundamental fairness about it, if you will. >> Right, it's a good relationship but I want to go down another path before you turn the cameras on. Talk a little bit about the race always between the need for compute and the compute. It used to be personified best with Microsoft and Intel until we come out with a new chip and then Microsoft OS would eat up all the extra capacity and then they'd come up with a new chip and it was an ongoing thing. You made an interesting comment that, especially in the cloud world where the scale of these things is much, much bigger, that ran a world now where the compute and the storage have kind of, outpaced the applications, if you will, and there's an opportunity for the application to catch up. Oh by the way, we have this cool new thing called machine learning and augmented intelligence. I wonder if you could, is that what's going to fill or kind of rebalance the consumption pattern? >> Yeah, it seems that way and I always think about kind of like, compute and software spiraling around each other like a helix. >> Like at one point, one is leading the other and they sort of just, one eventually surpasses the other and then you need innovation on the other side. I think for a while, like if you turn the clock way back to like, when the Pentium was introduced and everyone was like, how are we ever going to use all of the compute power. >> Windows 95, whoo! >> You know, power of like the Pentium. Do I really need to run my spreadsheets 100% faster? There's no business value whatsoever in transacting faster, or like general user interface or like graphical user interfaces or rendering web pages. Then you start seeing this new glut, often led by like researchers first. Like, software applications coming up that use all of this power because in academia you can start saying, what if I did have infinite compute? What would I do differently? You see things, you know like VR and advanced gaming, come up on the consumer side. Then I think the real answer on the business side is AI and ML. The general trend I start thinking of is something I used to talk about, back in the old days, which is conversion of like, having machines work for us instead of us working for machines. The only way we're ever going to get there is by having higher and higher intelligence on the application side so that it kind of intuits more based on what it's seen before and what it knows about you, etc., in terms of the task that needs to get done. Then there's this whole new breed of person that you need in order to wield all that power because like Hadoop, it's not just natural. You don't just have people floating around like, hey, you know, I'm going to be an Uzi expert or a yarn expert. You don't run into people everyday that's like, oh, yeah, I know neural nets well. I'm a gradient descent expert or whatever you're model is. It's really going to drive like, lots of changes I think. >> Right, well hopefully it does and especially like we were talking about earlier, you know, within core curriculums at schools and stuff. We were with Grace Hopper and Brenda Wilkerson, the new head of the Anita Borg organization, was at this Chicago public school district and they're actually starting to make CS a requirement, along with biology and and physics and chemistry and some of these other things. >> Right. >> So we do have a huge, a huge dearth of that but I want to just close out on one last concept before I let you go and you guys are way on top of this. Greg talked about what you just talked about, which is making the computers work for us versus the other way around. That's where the democratization of the power that we heard a lot about the democratization of big data and the tools and now you guys you guys are talking about the democratization of the integration, especially when you have a bunch of cloud based applications that everybody has access to and maybe, needs to stitch together a different way. But when you look at this whole concept of democratization of that power, how do you see that kind of playing out over the next several years? >> Yeah, that's a very big- >> Sorry I didn't bring you a couple of beer before I brought that up. >> Oh no, I got you covered. So it's a very big, interesting question because I think that you know, first of all, it's one of these, god knows, we can't predict with a lot of accuracy how exactly that's going to look because we're sort of juxtaposing two things. One is, part of the initial move to the cloud was the failure to properly democratize data inside the enterprise, for whatever reason, and we didn't do it. Now we have the computer resources and the central, kind of web based access to everything. Great. Now we have Cambridge Analytica and like, Facebook and people really thinking about data privacy and the fact that we want ubiquitous safe access. I think we know how to make things ubiquitous. The question is, do we know how to make it safe and fair so that the right people are using the right data and the right way? It's a little bit like, you know, there's all these cautionary tales out there like, beware of AI and robotics and everything and nobody really thinks about the danger of the data that's there. It's a much more immediate problem and yet it's sort of like the silent killer until some scandal comes up. We start thinking about these different ways we can tackle it. Obviously there's great solutions for tokenization and encryption and everything at the data level but even if you have the access to it, the question is, how do you control that wildfire that could happen as soon as the horse leaves the barn. Maybe not in it's current form, but when you look at things like Blockchain, there's been a lot of predictions about how Blockchain can be used around like, data. I think that this privacy and this curation and tracking of who has the data, who has access to it and can we control it, I think you are looking at even more like, centralized and guarded access to this private data. >> Great, interesting times. >> Yeah, yeah Jeff, for sure. >> Alright James, well thanks for taking a couple of minutes with us. I really enjoyed the conversation. >> Yeah, it's always great. Thanks for having me Jeff. >> It's James on Jeff and you're watching theCUBE We're at the Snaplogic headquarters in San Mateo, California and thanks for watching. (electronic music)