from galvanized San Francisco extraction signal from the noise it's the kue cover the apache spark community event brought you IBM now your host John free George okay welcome back everyone we are live in San Francisco for this special q presentation with the IBM sparkman the event here live at galvanized in San Francisco workspace incubator great place for developer education IBM's big announcement today their commitment to spark they didn't see any numbers but I'm counting in the hundreds of millions of years to quote Papa Chiana on my call with him on Friday with rod $17 fuck yeah holler last for hundreds of millions yeah hundred millions of dollars getting late in the day going to be your coming rod Smith's our next guest rod welcome to the cube thank you very much with a catalyst behind spark at IBM worked hard on it yeah you guys tell a story what's the story well we worked on big data and I have a group of folks that go out and work with customers all the time and what we were doing Hadoop we would do these cool applications that sometimes you know small clusters 20 minutes you get a result and a customer would say can you do that in a couple seconds kind of look around and go what changed it means it did the business problem and they couldn't tell us but it's one of those data points in your head that go something's not quite right you know what's what's changing or what are they trying to tell me that they can't and that's when we started learning you know customers were looking for technology that they could iterate on quickly you know open-ended questions it wasn't the give me a problem do the game pew pew output I'm done this was oh gee there's the journey I now see some interesting insights I have other questions was it was something not right the data that they got didn't match their hypothesis or was it the expectation that if I can do it fast on google and find a Thai restaurant down the block well so I can it went that way something doesn't right what was with me that said why can't you tell me what you're really trying to accomplish what I learned is that as we go through these kind of digital transfer mation real real time they were thinking about how their business is going to change so fast and so the problems always been for technologists and vendors like IBM tell us the problem we pick out the technology and you're pretty well stuck with it it stays that way and they wanted more flexibility open-ended questions lots of different data sources on demand when they had to have it on this they wanted to see results along the way and they would rather have analytics be approximation that they could use quickly rather than after the fact and more accurate okay so you know when you went through that it wasn't they couldn't find a bi person to talk bad about and I couldn't find a data person so you know it was fun to try to put piece puzzles together and that's where spark came into this so I see a lot of other trends are kind of vectoring into that convergence which is in-memory databases you know the community flash for persistence store on the storage side so this you as a close to all that action what was the aha moment for for within IBM is han hey you know what this spark thing is the next Linux me we got to get out in front of this and help the community go faster and then kind of rising tide floats elbows what was that flash point flow we we had two of them one was that in our commerce group there's ways that they work on online pricing and there's a vendor stander which takes about a week when you get data off of a site or retail site they analyze that they correct the analytics they put it back up again takes about a week but we showed them a spark we could do it in about four hours a week down to four hours and now they started to think oh you know what do we offer customers now we have ways to have not just one product many products let's bring in other data location data traffic data weather data social data so that kind of exploded internally on this is a big change this is something that we can relate to cus of multiple data source of the need for unification and speed and and speed speed first because be first that's a heck all the speed i want to bring other data sets and it's time to value i mean if you're going to be a digital business and look at real time where it's going Netflix others have really set the standard on ok so then i'm a so let's take a next level so rod you're crazy we can't do that it would disrupt all these other businesses we have so how does that conversation happen within IBM the way that happens in IBM is rod you are crazy and you're going to cause me odds it up so please go away and I don't go away easily but you keep pushing on this and part of my job is to work with customers can I show value so I can take the product team saying you need to take this more seriously I've got currency now and then as you just said the marketplace starts to light up spark is on the front page as people are talking about how they're using it well Hadoop is growing too at the same time so it loop does it seeds the market seats the Mars you see you're playing ahead do but if you see the customer challenges and you're like you guys just connect the dots and and then it's back to the customer is talking about what their problems they want to use or the solutions are looking for so yeah it takes time because it's it's risky meaning that all of us have quarterly is what we're doing but how do we now make it safer for people in IBM jump in the water so that eventually they don't hate me so what's your what's your comment when a friend says hey rod you know linux was great but it's a different era oh you know here with cloud and mobile open source with the patch he's evolved to the point where it's very manageable for vendors to be contributed as well with with non company contributors how do you guys see the difference between those two worlds because really this is a Linux moment but there's no big bad main many many computer companies name frames out there but their specialized for like the Z systems are great but like this is scale out commodity hardware a dupe now that's growing how do you how do you describe that because there is a Linux correlation what linux was for open source then operating systems now this is kind of distributed analytics I think you're you're you know the the part of this is kind of real-time digital business transformations and while there is not a you know bad company out there you know amazon and others have shown how they can be online businesses and use analytics and be very effective but i'm a brick and mortar company and an online business how do i do the same thing and spark starts to really show that no they don't have a corner on the market we can compete so that's the big factor on this is well it's not one company doing this it's I need to be able to compete at the speed the businesses that didn't have to see that Amazon started kind of post recession or you know Dom bubble bursting you know web services was just kind of kicking through if we remember our history lessons and what happened was they really had no traction they built some building blocks right they made a good decision to integrate to core building blocks compute and storage and they built from there so in a way you guys can enable companies to have their own amazon like extensive experience because it's a fresh clean cute paper right it is and I think we're spark it's interesting is like you said in two verticals what do i do to retail what do I do in health care what are we doing finance right very specialized I we've shown in Watson you can do Watson for cancer research you can do Watson for cooking right but they're very vertical now so specialized domain expertise becomes really interesting right that's the big part and that's the part I really liked about spark they were the community really thought about solution developers you know they stayed away kind of middle ground I you don't have to be a deep dated person or a deep analytics API person what's the problem you want to solve how can I help you do that I think that's a you know that's interesting is that that's because most people go Jay this is speeds and feeds software we look at the solutions more holistic but then you're really talk about customer problems right the so-called outcomes that go on well that's what and I think that's the part that I've enjoyed is I want to talk to you you know about what your problem is I don't want to talk technology I you know I don't want to have to make a technology choice from stay one spark helps me with that I don't notify programming while all those things come together so I can concentrate we can concentrate on talking to the customer but you know learn from them what are you trying to accomplish so you watch the next things on your list good I just gonna say you know looking at your LinkedIn page i love this at BP emerging technologies for 20 some odd years so you see here you've seen a lot of technology's come a lot of emerging technologies and the acceleration of these technologies is only going more right you have a whole lot more in your portfolio you have to look at today then then you did yesterday or five years ago yeah why is sparks a special in the cornucopia of technologies that you've seen coming over the years it's a good question and and as I've done merging technologies I've learned that I have to you know listen to customers very carefully on it and when I hear those kind of repeatable business patterns do I see an economic change a transformation that really sticks with me and sometimes the old things have start really big you know they start out good and then they fade away but I always look for technologies that seem to have lots of dimensions to them from a business value standpoint that's what attracted me to spark and my team working with some customers on pocs we could do them quickly you know I really like to get to the point where you know we an industry we with notebooks and others we can do solutions in less than four hours for a customer what better thing to take your you know employee to lunch and spat them on the back for you know something that you didn't expect for weeks well one of the exciting things that you guys have done is you shine the spotlight on spark and you opened up the conversation globally around IBM is making a big move spark was a little bit of an outlier and the mainstream press I mean the press we're picking up spark oh yeah berkeley some credibility of great people behind it but now it's like wow it's going to get the attention of CX cxos out there and they're going to be like hmm if ibm's looking at it must be relevant because of the history you guys have with innovation but they're going to ask you the question I'm going to ask you which is it's not baked out yet where are we with this what are you guys going to do how does IBM work with the community to continue to bake out spark because a lot of people are using it bringing it in but it's evolving super fast and that's going to be the question is it baked and how does it get baked faster so I think there is lots of areas that if we just talked about if I'm doing retail or health care or fine it's going to be lots of specialized analytics because that's what spark for me is is enabling custom analytics on this second part is as you think about how you want to look at bigger problems I think that many times are learning is to try to you know once we got a technology lets make everything fit it rather than starting to separate it by business problems and I think we can do that now or we can bring to the table technology learning best practices around this and solutions I think you know at the end of the day it's house part can be integrated into a business solution and our customers very quickly and hopefully those customers see it broadly from interoperability standpoint of what they're going to do so the final question I have for you is what was the biggest learning that you've taken away from this process that was magnified through this whole journey of a taking IBM from being a participant in the as a citizen in the community early on as a founding member of spark this is back in two thousand nine so it wasn't like no one knew he was going on and you know we bird cover on Hadoop from the beginning so we'd love to watch these ecosystems grow but from from the early days to now today mmm what was the biggest thing that you learned that was magnified out of all the reactions all the feedback all the customers what can you share I I think for me when we did a spark hack you know our hackathon piece when 28,000 IBM ER showed up with ideas that told us twenty eight thousand 28,000 so now you stopped and 28,000 people who were focused on the customer so they had a thought of how this could be relevant this is great I mean this isn't like back talking for this isn't one little vein with a little stream it's big and it big was what we can do for our customer when was that um about two months ago how did you pull that off just out an email blast all the IBM's put on the message board to a crowd chat what did you do well when you put out an email blast the second one is you put on a webcam to explain to people what you're going to do with it what you'd like them to do and I'll we're setting it up and and then you step back and you know kind of cross your fingers hope people show up and then when you know you invite ten thousand and twenty eight thousand show up you kind of know that we're turning a corner as a company on understanding how we can use that for this this also highlights this whole connectedness apps internet of things and people are things to so their mobile device when you have that kind of people close to the action the creativity is there right there on the front lines and they don't feel like that the work they do is going to be taken by the machinery in the old days I got to go back all these hurdles I gotta jump now they could instantly be there with some solutions so that's that's super compelling the next question is security and how does how do you see that leaving in because now one of the things that came up will first meeting let me back up but I get this you think about security question for a second last week ahead dupe summit we were talking with the Hadoop ecosystem Hortonworks ODP conversations etc but when you looked at kind of like reading the tea leaves it was sparked that was kind of stealing the show the subtext was smart all the spark sessions were packed the developers had was salivating over sparks like to hear that I did why why is that why are the Hadoop developers salivating over spark is it because they wanted to go faster do they see extensions any thoughts I think that I've say it two ways one is I think there was and since I did who do for quite a while I think people thought for a while Hadoop was going to be an analytics platform and it it kind of went down the path of being immoral generalized platform so you can do more than MapReduce jobs so there's been this pent-up demand for really analytics focus and spark offered that focus and the performance side I think that's the parts in Hadoop sold kind of a false dream or it didn't materialize fast but I don't think of material out of false treaty I'm saying if they promise them around yeah it well and people set those you know well the fresh maybe yeah I don't think the vendors all I think was more than well vendors you know it did to unstructured data does that unstructured data does that storing data and I didn't be able to act on it creates some interesting dynamics I mean I've worked with customers who you know started to put data in Hadoop but to have put data dupe you know we're only going to do a year's worth of data and then putting three years of data because they want to do monte pucker up my Carlo simulations against a Monty Python it's time you threw water on us and we love yours we on the cube but the problem says we're talking about before like you know our internal use we can produce you know interesting innovations in days that's going to attract audiences because now they can show their you know business people what they can do for them that's what's really driving this I mean if you gotta see XO you know CMO says you know show me what you can do you know do segmentation on my population for these products they want it in in minutes not so you know going to run it in different jobs and the over a certain period of time I was just talking with the CEOs of docusign box 18 1018 Syrian kinky was executive director and then EVP a platform that Salesforce the common thread amongst those executives was the new digital transformation has such a dynamic or impactful economic impact yes I mean dr. Sanyal using examples how literally Deutsche Telekom saved 230 million dollars on one process yes one process yes with analytics and yeah process improvements extreme it sounds funny but it's extremely low hanging fruit they haven't had technology and the economics and be able support it now we do and now you're seeing the solution developer go I think I can make a business result faster yeah and if they can show it then businesses react and I think that's the beautiful thing about what Hadoop is done I mean I brought that up earlier trying to tease that out with reality we're seeing is that that mark is continuing to grow but there's a world beyond Hadoop yep I mean Hortonworks this public company I mean IBM is massive so you got Hadoop and then sparks a beautiful extension to that that enables so much more well I think spark will go further because it's more to me is another dimension it's an integration technology so i can have sparked up to legacy systems without hadoop you know in there doing analytics in there being an avenue for doing joins on data doing analytics on unstructured and transactional data whether data pulling it all together and I think that's the again talking about multi-dimensional that's what that was hard even five years ago so any relational database that's a nightmare yeah and you're asked about security so you want to touch on yeah okay go ahead so part of the things that I like about spark is the technology is called resilient distributed data sets r dds so I read data from a source and I make it into this r DD I can work on it that gives me a great data point or a great interaction with a Cassandra datastax did a really great job of a spark driver so you think about this in businesses for a db2 or something now I know where I can put my security and my governance I can put those at certain endpoints now as i'm reading in my application writing these things out so again back to my point of an integration it's not something that i'm trying to get around a business i'm at integrating extending their life and/or capabilities that's right so I got to ask you the internal IBM question my last question is it what's the vibe like at IBM because you know I've been you know I worked at IBM way back in the day back in the 80s and the cultures changed right so much mm-hmm but there's still a huge technical group of people at IBM so I got to ask you the question with all this new cloud innovation all this new capabilities to do stuff differently what's it like for all the technical guys at IBM right now because they got to be like Hayden we can now do this we can so new capabilities are emerging what's the what's the vibe like and what are some of the things that that are low-hanging fruit that are that our game change because low-hanging fruit is game-changing today oh yes I what's the vibe eternally at idea I've internally is very hot I mean the guys and gals at this you look at cloud computing look we've done with bluemix it got is getting you know great recent press it's getting great results with customers back to this time to value piece it's new to us I mean there's only a small group that started that so now the rest of the IBM arts are going this is really cool how do we do it now you've got analytics that you know we're starting you've been you know competencies are on this now you can take the real-time aspect so yeah the five is really all those little silos you know identity system here I got to build all the software now you can gotta go horizontal yeah so you know that's kind of a new thing that's kind of exciting it's gonna be fun to watch my final question I guess is my final final question is have you been keeping track this is the sixth and final time analytics well rods great to have you on the cube you're awesome great great commentary great great insight spark in the cloud is what data bricks announce what about an on-premise i'm a customer i want i want on prem I don't necessarily want to do what's next I 40 s or other stuff oh I think you're going to see you know like hybrid models for cloud where spark as a service is there on prem i think one of the really exciting parts to me is that one the unified program model to the portability of the analytic models so let's say I start on prom because I'm worried about security and other things and then I want to move it to a cloud service well I don't have to go rewrite it I can just move the analytics over from a model standpoint so I think you're going to see this evolved very fast as people want to do either on prem or hybrid or you know dedicated cuz of the integration capabilities and the distributed nature of it that's the point yep awesome well I'll let you get the last word on the segment share what the folks who's not or aren't watching what is this all about today why is in San Francisco today IBM's announcement what's so groundbreaking about it I know you're part of it a little bit biased but share the folks why what why now what's this all about what's what's what's going on here well we think that the kind of epicenter for spark innovation is here in San Francisco amp lab with data bricks and others are doing here and we want to be a part of that and I think spark technology senator setting up is about how we can contribute and learn and you know help the community grow we think this is gonna you brought some food to the party I mean you are I said earlier beer right you bring a you know the ml yeah you got them back other wine napa valley of course you got to go to wine well craft beers good north north bay thanks so much for coming on the cube really appreciate the insight because it is a great color from an expert IBM here we're on the ground this is the cube special presentation live in San Ruby back with more with live coverage of the breakouts in the event tonight IBM spark community event here in san fran at the galvanized workspace education center we write back