>> Man: Live from Boston, Massachusetts, it's theCUBE, covering Red Hat Summit 2019. Brought to you by Red Hat. >> And we are back live in Boston as we continue our coverage here on theCUBE of Red Hat Summit 2019. It is our sixth year here at the show and this year obviously some huge announcements. A significant moment it's been for Red Hat, we heard from Jim Whitehurst a little bit ago. Stu Miniman, John Walls, we're now joined as well by Niall Fitzgerald, who is the GM of IT Application Architecture and Design at Spark NZ. Niall, good afternoon, or I guess good morning still we're in an Eastern time zone. >> Yeah it's the middle of the night in New Zealand I'd say. >> Yeah, so Spark NZ New Zealand. Tell us a little bit first off about Spark NZ. What the folks back home are doing right now, work-wise, and your role with the company. >> Yeah, so Spark is the largest provider of telecommunication services in New Zealand. All the traditional type of services you'd expect, mobile, broadband, et cetera. We came out of the traditional kind of post office, so we've a lot of heritage, and about four years ago we rebranded from Telecom New Zealand into Spark. To represent that we were changing from being a telco into much broader range of digital services. Our purpose is to help all New Zealanders win big in the digital world. >> Niall, step back for a second. Talk to our audience that might not know the telecom industry as well as you, I've been an observer and participator in the industry but you know back in the dot com boom it was like limitless bandwidth and we're gonna do all these wonderful things, and cloud and digitization, have put some new opportunities as well as stresses and strains on your industry so, you know what's going on and you said you rebranded? >> Yeah, look, I think it's well-known it's been a tough last few years for most telcos in the world. I was listening to Red Hat talking yesterday about 60 consecutive quarters or more of growth, I don't think there's any telco in the world that probably has the same story. Like most, we're facing kinda decline in all the traditional revenues like voice and text and things like that, so we're all having to kinda rebrand ourselves and deliver much higher levels of customer service. People expect the same levels of service from us that they do from Amazon, Google, and everyone else. In Spark what that means to us is we've moved into lots of new things as you said, things like ICT, we're now very big in cloud, we've recently launched a Spark Sports brand and we've got streaming right to the key events like Formula 1. We're going to stream the Rugby World Cup, which is a massive event for New Zealanders, so looking forward to seeing that and Ireland on the all blacks in the final in September this year. So yeah, lot going on. Tough times but forcing us to keep changing every year. >> And so, about these changes that you're making whether technologically based, let's just deal with that. What is that ultimately going to do for you in terms of better customer service delivery? So, you've got inherent challenges, you've talked about them at all, that the world's changing, how we use this medium, this communication opportunity is changing, and you've been just a little behind the wave, hard to keep up with it, so rapidly changing. How much of a challenge is that? And then how are you going to address this going forward? How do you stay relevant? >> Yeah I think we're lucky in one regard because if I look back about five, seven years ago we were like most traditional telcos. We had a spaghetti for want of a better description of systems, and then we had all was multiples of everything, at the time we had 19 integration layers and 10 billing systems and it wasn't uncommon. But way back in 2012 we actually embarked on a massive transformation program, and we spent five years consolidating all of that infrastructure so going into about 2017 we were very lucky in that we had a massive foundation laid already, so what that then enabled us to do was to actually push away calls from our contact centers into mobile apps, into digital adoption. We've been a big embracer of things like big data and robotic process automation as well to try and take cost out of our industry. So, I think we're quite well placed. Now that allows us to do things like innovate new products for our customers so we bundle things like Spotify and Netflix. It allows to introduce things like Spark Sports brand, which we couldn't have done five years ago before the transformation We just wouldn't have been able to enable these things with our existing kind of legacy IT estate. >> So how's open-source play into all this for you? >> Yeah open-source, I suppose our first foray into open-source was when we went to start embracing big data and automation. So we started using things like Hadoop and various other things and our entire platform is based around open-source. We changed to an IMS network recently and we started embracing things like OpenStack, and then it really took us to a new level recently when we started working on Red Hat's Fuse, and OpenShift we started implementing that. >> Okay, so the OpenStack show for many years, the last few years we saw the telcos coming in specifically for network function virtualization or NFV. Is that what you're using in that space? >> Yeah, we are. Interestingly, at this conference I've heard a lot of people talk about OpenShift and OpenStack, obviously, particularly in the telco game. We actually came out a bit differently from the application space. So we had an integration platform that we had put in through this transformation phase which had served us well, and was connecting all of our 40, 50 systems together. But it was coming up to a life cycle event, and we decided we'd look externally and see had we options beyond just upgrading it. So we started looking around, and we effectively found Fuse, and in bringing in Fuse we then brought OpenShift in, which is quite different to what I've seen from a number of other people, they're bringing in things like OpenShift and building on top of it. We did it the other way around, you know? And we did it primarily for cost reasons, you know? >> Yeah, so talk a little bit about that impact of Fuse and OpenShift, what that means. Were you already down the containerization journey, or did that help drive >> Niall: No, no some of that modernization? >> That's exactly what happened. If I'm honest we hadn't really explored containerization too much because we had come to the end of our kind of transformation journey. Open-source and containerization wasn't around when we went through that. So we kinda needed some really core reasons to move on, so, yeah effectively what happened was we looked at Fuse, I was gonna say primarily for cost, but we were looking for something that we could migrate to where it makes sense. We were looking for something that wasn't a massive lift for the people who worked in our integration already, so they could be rescaled into it, and interestingly we turned agile recently which has changed the way we look at the needs of our systems. So our old integration platform, if we needed to deploy a change we had to take an outage, which was fine when we had a centralized IT department who deployed once a month and took a two hour outage, but when you have 20 tribes all developing features in isolation and they wanna go straight through to production, if everybody took an outage then our systems wouldn't really be up very often. So one of the key things that we were looking at for our new integration platform was can we deploy hot and can we scale? So that's basically where Fuse came into us. >> Okay, so can you? >> We can and we do. Still a little bit nervous about pressing the button mid-day and doing stuff >> Right, simultaneously and thinking this has really gotta work, right? >> Yeah then normally, >> We saw it today though on the demo stage, on the keynote. You know, simultaneous operations going on. >> No, we do it, and they normally don't tell me when they're doing it they just do it and tell me it worked afterwards, but no it's actually been really successful and you can imagine connection 40 or 50 systems together is effectively the equivalent of about 2,000 API's and we managed to migrate, we're about 70% of the way through. But we've managed to migrate those without actually impacting the systems that use them and that's probably been one of our most successful IT projects that I've seen. >> It's funny, you said we were towards the end of our transformation journey, and of course I think we all understand, it is just, I might've reached a marker in my journey, but it needs to be a continuous process. And you went through an agile transformation. So bring us in a little bit. Organizationally, what happened there. Some of the good, the bad, and the ugly of agile, 'cause I mean agile's always an ongoing thing. >> It is, yeah. So about the start of last year we started to think about agile and the need to change our ways of working. And we looked at a number of models overseas, and companies like Spotify and various banks, and we settled on a model of chapter and tribes. And we took about six months in looking at what that meant for us as an organization and all of the things that we needed to change. Everything from, people's contracts to people's titles. We got rid of all complex titles and moved down to simple things like Developer, Tester, et cetera. We had to train our people in agile so we ran boot camps for over 2,000 people. We had one with 500 people attend. We had to review all of our processes and see where we had centralized things like IT governance or procurement. How do you actually manage this when you have up to 20 different people effectively, or tribes doing their own developments, so over a period of about six months we went through all of these. We started with a concept of some forerunner tribes so we could figure out how this thing actually works, you know? And get some lessons. And then on the first of July last year, about a 2,000 people in various buildings packed up their stuff in their desks and moved into a new world, into their tribes with different working spaces and different collaboration areas and all the tools that we need. So, yeah we're about nine months down that journey now and it's been good. >> How many total employees? >> We have about 5,000 in total. >> 5,000, so you had 500 at one time. 10% of your workforce in training at one time. >> That's right, yeah. Absolutely. >> How do you keep the wheels on the bus rolling? Because I mean you're asking people not only to learn new skills, but learn them in a new environment, and learn them literally in a new place. I mean that's just massive change and I think, we're human beings. We're creatures of habit to a certain extent. You had to hit a lot of bumps along the way. >> Yeah, so one of the key things we did upfront was we said the operate part of our business, which is effectively things like our contact center, our sales staff, our service desks, we will not go agile with those on the first day, because they operate in a slightly different way of working. The people in our stores, et cetera. So we had a concept of agile light and agile heavy. So we kinda parked them for a minute so that we wouldn't do exactly what you say and let the wheels fall off the trolley. And we took to people that were the IT developers, the product development staff, and all of that, which came to just over about 2,000 people, and we firstly flipped those 2,000 people and put those through bootcamp. But even as you say, scheduling the boot camps, we made sure that we always had the right people on the ground and we would schedule smaller boot camps for them later if we needed to do it, but yeah. >> So nine months in now. You talking to your peers, if they're gonna go through. Any key learnings, what were some of the most challenging things that you ran into? >> I think probably the major one is that agile at its heart is a way of working, and despite the name it's actually quite prescriptive in how you should work, you know? When you pick up the agile book it tells you all the ceremonies you need to run and the processes that you need to run as well. And I think you need to be pragmatic in how you implement it because there are so many different flavors of agile. The one flavor, even with an organization of Spark size, it doesn't work. So the tribes and squads that are building out new products compared to the tribes that are doing things like upgrading systems, they will work in different ways. So I think the first thing is be pragmatic, take the goodness and the intent of agile, but implement it in how it works for you. And there's some other practical considerations, like prior to being agile we had quite a large number of our technology partners were based offshore in India, and you know it's quite difficult to run a 10 AM stand up in New Zealand setting the priorities for the day and the sprint plans, when, you know, four members of your team are asleep in India. You know, they're missing out on all of the goodness and the collocation and the sharing, so one of the things we had anticipated that, so luckily enough we had moved a lot of those people onshore in advance of agile, you know? But it is a big cultural change for everyone in the organization, not least the leadership teams as well. >> John: Well you got through it. >> We got through it, but there's no going back. >> Absolutely, no you're in the deep end now. Well, Niall, thanks for being with us, we appreciate the time joining us here on theCUBE, and I think that an Irishman is always welcomed in Boston. >> Thank you very much! We've been enjoying the hospitality. >> Yeah the door's always open. >> Thank you very much. >> Thank you very much. Niall Fitzgerald, joing us from Spark NZ. Back with more here on theCUBE, you're watching this live at the Red Hat Summit 2019.

>> From the Palace of Fine Arts in San Francisco, it's theCUBE. Covering AT&T Spark. (upbeat music) >> Hi, I'm Maribel Lopez, the founder of Lopez Research, and I am guest hosting theCUBE at the AT&T Spark event in San Francisco. And I have the great pleasure of being with Vishy Gopalakrishnan. He is the VP of ecosystems and innovation at AT&T. And Vishy, I've known you for a long time now. I've known you through companies that are as diverse as SAP to AT&T. Could you tell us a little bit about what VP of ecosystem and innovation does and this concept of the foundry that AT&T is having? >> Sure. First of all nice to see you again, Maribel. >> Paths cross. No new people, just different business cards. >> Exactly. So ecosystem and innovation. So this organization has been around at AT&T for about seven years or so. And it was set up to fundamentally answer this question: How can AT&T systematically tap into innovation that happens outside the company and then bring it inside, and then over a period of time become as good at adopting some of those principles of innovative thinking, innovative principles of problem-solving into the company itself? So if you think about ecosystem and innovation, there are three key pillars to ecosystem and innovation. One of them is called ecosystem outreach. So this is a part of the organization that acts as the interface to the broader startup NVC community. >> Right. >> Right. So this allows us to keep on top of innovation happening across a wide variety of technology waterfronts. Networking, security, virtualization, all the way up to AR, VR, AI machine-learning, et cetera. >> It wouldn't be innovation if they weren't together, right? People try to really parse them, but true innovation comes of looking at some of the intersections of technology. >> Absolutely. And we're also agnostic in some sense about where the innovation comes from. 'Cause all we're trying to do is apply innovation to a particular business problem. And the foundry is the second component of the ecosystem and innovation organization. Think of the foundry as centers of innovation. There are six of them around the globe. Four in the US, one in Tel Aviv, Israel, and the newest one in Mexico City that we opened in March. And these foundries represent fundamentally an environment within AT&T where we can rapidly prototype new technologies, de-risk new technologies before we introduce them into the rest of the organization and actually also provide a way for us to bring proactively new, promising areas of technology to the rest of the business. So the foundries, if you will, serve as the leading edge of technology innovation within a company like AT&T. >> Well I've been in The Valley for more than 10 years now, and I came from the East Coast, and the concept of an innovation lab and innovation foundry isn't new. We've seen it come and go with established companies and with new companies. So I remember the launch of the foundry. You said it's about seven years ago, now. I can't believe it's even been that long. What have you learned in that time, and how are you making it work? Because I think everybody wants to be innovative, and they want to take, particularly established companies, these innovations and bring them back into the corporation. Can you give us a little more color and context on what you think you've done well and what surprised you? >> That's a great point to make about the relative longevity of the organization within a company like AT&T. >> And it's grown, apparently, with all the new innovation centers. >> Yes. And we've expanded to other locations outside. I think some of the lessons we've learned are that no organization stands still. >> True. >> AT&T as we know it today is different from what AT&T was seven years ago. The kinds of businesses we're in, the kinds of capabilities that we have to bring to bear, markedly different from what it was seven years ago. And the nature of the competitive waterfront is also dramatically different. So, which means that as an innovation organization, we've had to evolve almost lockstep and sometimes ahead of the organization itself. So that's been one thing that we've done, is that we've made sure that we always are aware of where the company's going, so that as we look at what kinds of innovation might apply, might be relevant, might be material for the corporation, we know that it's always grounded in what the company wants to do now, in about two years from now. >> So forget the science projects and try to get something that's practical to the business, but also a bit edgy, right? >> Yes. >> You want to be edgy. >> Yes, and it's an art and a science. We like to focus on innovation that's in context. So pure innovation is kind of interesting, but we always like to bring it back to either an internal stakeholder or an external customer as a stakeholder to sign off and be almost the kind of the voice of reason to say yes, this is interesting technology, but this is how it might or might not apply to my business problem. >> Do they ask you for things? Does the organization come to you and say, "Hey, we're looking for blah and..."? >> Absolutely. In fact, a big part of what we do as an organization is actually keep the dialogue with the internal stakeholders kind of ongoing and active, so that we always need to be aware of, from a business standpoint, what are the imperatives that a business leader is facing. 'Cause let's face it, a lot of these business leaders within a corporation as large as AT&T are running P&Ls that are pretty large. So for us to bring relevant and material innovation to them, we have to be aware of what are the two or three top, key problem areas that they're looking at. Is it cost reduction? Is it operational simplification? If it's a big part for network organization, what parts of network optimization are they most interested in? So being aware of that informs us better and in some sense helps us curate what kinds of innovative solutions we bring to them. >> Now you are talking about how you put these innovation engines around the globe, and I imagine that you are learning and gaining different things and insights from these different groups because there are phenomenally different ways people use technology depending where they are in the world. So can you share a little bit with us about what's exciting, what you're seeing in the labs today, are there geographic differences that we should be aware of as business leaders when we think of trying to roll out technologies? >> Sure. I'll give a two-part answer. One of it from areas of kind of focus for us. >> Okay. >> One as we just finished the panel on edge compute, so that's a big focus for the foundry organization, is trying to understand the use cases in which edge computing might actually give a pretty dramatic improvement in user experience, what is the role of the network edge in doing that, so working with a broad ecosystem of partners, both established and start-ups to actually make that happen. So that's one big area of focus. The other thing we're doing is... A big part of AT&T's business is actually focused on the enterprise side to AT&T business. So we have two foundry locations, one in Plano and one in Houston, that are focused exclusively on customer co-creation with our enterprise customers. For the past five years, we focused exclusively on IOT and used the Plano foundry to co-create around IOT for customers. In terms of differences across geographies, I think the most salient one is the one in Mexico City. We actually started that with the very explicit intent of innovating for emerging markets. Emerging markets have the need for high-performing, high-quality solutions. >> At a low cost. >> Exactly. So you need to deliver them at a much, much lower cost than the emerging markets actually will bear. So which means that you have to frame the problem differently, you have to go about innovation very differently, and oftentimes, you'll have to tap into the local innovation ecosystem as well. So that's a big, big part of what we're doing in Mexico as well. Trying to tap into the global network that we have as a company through all of the six foundry locations but making sure that we're tailoring it to what the local Mexican market needs. >> I'm actually very excited to see how innovation has been rolling out around the world. One of the things that comes up in every dialogue I have around innovation right now and frankly in most products is AI. Do you see a role of AI happening in the foundry today? >> Yeah we've been doing work on AI for quite some time. In fact, we've been doing a series of projects for our internal organization around applying machine-learning techniques to some very complex network optimization problems. And we're doing that for about 18 months or so. And we've been looking at even ways to apply reinforcement learning to some very classic network problems as well. As part of some of the work that we're doing around edge, we're looking at ways to do influencing at the edge. For a variety of use cases, including, for example, a public safety or a first-responder kind of a use case. So absolutely, AI and machine-learning continue to be one of the areas that we spend a lot of time on. >> Well Vishy, it's been great talking to you today here at AT&T's shape, and look forward to seeing you again soon. >> Thank you, Maribel. Likewise. >> Maribel Lopez, speaking with theCUBE. Thank you. (upbeat music)

>> From the Palace of Fine Arts in San Francisco, it's theCUBE covering AT&T Spark. (techy music) Now here's Jeff Frick. >> Hey, welcome back everybody. Jeff Frick here at theCUBE. We're at AT&T's Spark event, it's up in San Francisco at the Palace of Fine Arts. It's really all about 5G, and we're excited to be here, you know, there's been a lot of conversation about 5G for a very, very long time, and we're super excited to have the expert in the field. Maribel Lopez has been following this forever. So Maribel, first off, thanks for stopping by, thanks for hosting a few segments and great to catch up. >> Excited to be here. >> Absolutely, so 5G, you've made a funny comment before we went on. You said, "Jeff, this 5G's been going on "forever and ever and ever, but now it's finally "starting to come to reality, to fruition." >> Yeah, I got to see all the Gs: the 2G, the 3G, the 4G, now the 5G, and you know, for a couple of years we were just talking about standards, and what's really exciting to me is that now people are talking about doing production stuff, you know, not just rolling in a test van and prototype equipment, but actual things that we might be able to see deployed within the coming year. >> Right. >> People are talking about lighting up cities. AT&T announced another five cities that they were going to put, actually seven, I think, on the calendar. >> Up to a dozen, I think, now, then they had another-- >> Yes, they had seven, they added another five-- >> Seven after that, right. >> And then another seven, so we're really starting to see momentum in 5G, it's going to happen. >> Right, so there's a bunch of things with 5G that are fundamentally different than the last G. >> Right. >> And the first one, right, is it wasn't really developed just for faster voice. That was not the objective of 5G. >> Yeah. >> It's really to take advantage of IoT and this whole kind of machine to machine world in which we're in right now. >> Yeah. >> That's a fundamental difference in terms of the applications that it can open up. >> Yeah, we're seeing... To your point, I mean, we talked a lot about bandwidth before. Yes, you get more bandwidth, but you also get lower latency, and that's the thing of how fast something can travel, and that opens up a huge amount of new applications like autonomous driving. If you want a wireless connection in autonomous driving you need 5G so you have that, you know, really sharp response time to make it happen. If you're doing remote medicine, you know, 5G gives you both bandwidth, but also the latency to see if something's happening so that you can do things that are real-time in nature. So, I think it's that real-time in nature with high speed that everybody's talking about. We saw eSports and gaming listed today, and the discussion about how you could now do it on a low-end PC because between your 5G network and new software you've got this huge opportunity with the cloud to just do a whole new, different way of gaming and entertainment, so lots of great applications are coming out with 5G. >> Yeah, it's pretty interesting on that demo, because it was an NVIDIA guy talking about-- >> Yes. >> Having basically an NVIDIA data center to do all the graphic computation back in the cloud at the NVIDIA data center-- >> Yeah. >> And then delivering it to whatever kind of low-end edge device that you had, in this case a laptop. The funny thing about the latency that I thought really kind of struck home for me was they talked about when your audio and your video are slightly out of sync when you're watching a video. >> Exactly. >> When it's just off a little bit. >> Mm-hm. >> Not enough like, "Wait, this is broken," but enough to actually get nausea. >> Yeah. >> You actually have a physical reaction, so I think that was really interesting. That is what's going to go away when we have the better connectivity speeds, everything else with 5G. >> And I think that's when one of the things that's been holding back the immersive nature of new applications like VR, so that disconnect that you talked about is really important to get rid of that, and you can get rid of part of that with wireless and part of it with low latency. So, if we get the headsets a little smaller and we get more content I think we'll start to get a better vision of what's happening there. I also think we're starting to see these things come into the enterprise. You know, the enterprises are really taking 5G seriously. They're looking at doing things like their own private 5G networks in things like manufacturing and robotics, for example. >> Right, right, yeah, the private 5G, interesting, in a lot of conversation, too, about doing it for the first responders to have their own dedicated network, but one of the topics I thought interesting was the commitment to software and the commitment to opensource, and we've kind of seen the rise of the telcos and OpenStack. >> Yeah. >> We've been covering OpenStack, I think since 2013, and you could see with each and every passing year that the telco presence within the OpenStack community just increased and it really seemed to find a home, and here they dedicated a whole keynote session to AT&T's embracing of opensource. >> Yeah, opensource is actually interesting because I think it's counterintuitive to think that a large enterprise customer like AT&T would go so deep into opensource, but when you really think of it, if you want to be innovative and you want to run at, you know, what we now consider cloud speed-- >> Right. >> Digital native speed, then you need to have that concept of opensource and open APIs to build on top of so that really what you're focusing on is the part of your business that differentiates you, not on building the whole stack. So, the days of building, like, your whole stack from scratch are over, and opensource is really important, and what I found really interesting about that was the takeaway that so many companies, even competitors of each other, had all thrown in on this concept of this opensource technology so that they could basically bootstrap their innovation. >> Right, the other kind of theme that kind of came up, which I found really interesting, is if you've ever seen Jeff Bezos speak on his investment in Blue Origin. >> Yeah. >> He talks very specifically that he wants to put a platform in play-- >> Mm-hm. >> Leveraging the winnings that he's gotten from Amazon to enable future entrepreneurs to have an infrastructure in which they can build cool applications-- >> Absolutely. >> In this case for space. We heard the same message here within this kind of 5G, that the concept of, you know, kind of infinite compute, infinite bandwidth-- >> Right. >> And infinite storage asymptotically approaching zero, what applications would you build in that world, and really this constant conversation of experience, whether that be a business experience, a consumer experience-- >> Yeah. >> A first responder experience, is really what's behind kind of the excitement on this 5G conversation. >> I think there was always a disconnect of when you get data, and how quickly you can analyze that data and get it back to somebody to do something meaningful, so this whole experience is about even if you are not holding a 5G handset or some 5G thing in your hand or elsewhere, what that will do is because they've built the 5G infrastructure you get the opportunity to make 4G better for everybody. So, I think people think, "Oh, I've got to wait for 5G." It's like, "No, you're going to see the benefits "of 5G long before everybody's ubiquitously deployed, "long before everybody has 5G devices." >> Right. >> Things are just going to work better, and you can get that data faster and new experiences faster, so I'm excited for it. >> Right, and then the other piece that we hear over and over, right, is AI and machine learning, and again-- >> Absolutely, mm-hm. >> It's not AI and machine learning just for the sake of AI and machine learning. It's baked into all these other applications to make them all work better, and again, that's another big thing that we hear here at the keynotes. >> Yeah, I think the AI and machine learning is interesting because we've had it for a long time, but now everybody has access to it, right? We've got cloud services that give you algorithms, we've got massive compute, and now we've got the ability to take all the data from IoT sensors and other things and get it back to either a centralized place, or to do edge compute on it, which I think is really exciting. >> Right, so just to wrap, get your kind of your final impressions on kind of the show-- >> Yeah. >> And again, you said you'd been here for all the Gs, (laughs) so is a 5G, is this a big difference from our prior step functions? >> I think it is because of that latency that we talked about and the ability to do much more real-time, data intensive apps. So, you've always had this concept of moving to more data, but it had lower latency, it might've had a higher cost. Now we're getting that right kind of combination of cost, bandwidth, real-time nature, so I think every G gets better and 5G is just better than 4G, but in different ways, so-- >> All right, well Maribel, thanks again for stopping by, and also for helping us out guest hosting a few segments. >> Thank you. >> All right, (chuckles) she's Maribel, I'm Jeff, you're watching theCUBE. We're at AT&T Spark in San Francisco, thanks for watching. (techy music)

>> From the Palace of Fine Arts in San Francisco, it's theCUBE! Covering AT&T Spark. (bubbly music) >> Hello! I'm Maribel Lopez, the Founder of Lopez Research, and I am here today at the AT&T Spark event in San Francisco and I have great pleasure and honor of interviewing Mazin Gilbert, who is the VP of Advanced Technology and Systems at AT&T. We've been talking a lot today, and welcome Mazin. >> Thank you, Maribel for having us. >> We've been talking a lot today about 5G, 5G is like the first and foremost topic on a lot of people's mind that came to the event today, but I thought we might step back for those that aren't as familiar with 5G, and maybe we could do a little 5G 101 with Mazin. What's going on with 5G? Tell us about what 5G is and why it's so important to our future. >> 5G is not another G. (Maribel laughs) It really is a transformational and a revolution really to not to what we're doing as a company, but to society and humanity in general. It would really free us to be mobile, untethered, and to explore new experiences that we've never had before. >> Do I think of this as just faster 4G? Because we had 2G, then 3G, then 4G, is 5G something different? When you say it allows us to be mobile and untethered, don't we already have that? >> No we don't. There are a lot of experiences that are not possible to do today. So imagine that having multiple teenagers experiencing virtual reality, augmented reality, all mobile, while they are in the car all in different countries; we can't have that kind of an experience today. Imagine cars as we move towards autonomous cars, we cannot do autonomous cars today without the intelligence, the speeds and the latency with 5G, so that all cars connect and talk to each other in a split of a second. >> See, I think that's one of the real benefits of this concept of 5G. So when you talk about 5G, 5G is yes more bandwidth, but also lower latency, and that's going to allow the things that you're talking about. I know that you also mention things such as telemedicine, and FirstNet network, any other examples that you're seeing that you think are really going to add a difference to peoples lives going forward as they look at 5G? >> 5G is a key enabler in terms of how these experiences are going to really be transformed. But when you bring in 5G with the edge compute. Today, think of compute, and storage, and securing everything, is sitting somewhere, and as you're talking, that something goes to some unknown place. In the 5G era, with the edge, think of computer storage as following you. And now-- >> So you're your own data center. (laughs) >> You're pretty much your own data center. Wherever you go with every corner, there's a data center following you right there. And now add to that, we're transforming our network to be programmable with our software-defined network, and add AI into that, bringing all of this diamond together, the 5G, the edge, programming the network with software-defined, and AI, and that is what the new experiences is. This is when you'll start seeing really an autonomous world. A world in which that we're able to experience drones flying and repairing cell sites, or repairing oil tanks, without us really being involved, from being in our living room watching a movie. This is a world that is extremely fascinating, a world in which that people can interact and experience family reunion, all virtually in the same room, but they're all in different countries. >> I do think there's this breakthrough power of connectivity. We've talked about it in the next generation of telemedicine, you mentioned some of the dangerous jobs that we'd be able to use drones for, not just for sort of hovering over peoples gorgeous monuments or other things that we've seen as the initial deployments, but something that's really meaningful. Now I know the other topic that has come up quite a bit, is this topic of opensource, and you're in the advanced technology group, and sometimes I think that people don't equate the concept of opensource with large established organizations, like an AT&T, but yet, you made the case that this was foundational and critical for your innovation, can you tell us a little bit more about that? >> Opensource is really part of our DNA. If you look at the inventions of the Unix, C, C++, all originated from AT&T Labs and Bell Labs, we've always been part of that opensource community. But really in the past five years, I think opensource has moved to a completely another level. Because now we're not just talking about opensource, we're talking about open platforms, we're talking about open APIs. What that means is that, we're now into-- >> A lot of open here. (laughs) >> Everything open in here. And what that really means is that we no longer as one company, no one company in the world can make it on their own. The world-- >> K, this is a big difference. >> It's a big difference. The world is getting smaller, and companies together, for us to really drive these transformational experiences companies need to collaborate and work with each other. And this is really what opensource is, is that, think of what we've done with our software-defined network, what we called ONAP in the opensource, we started as a one company, and there was another, one of the Chinese mobile companies also had a source code in there. In the past one year, we now have a hundred companies, some of the biggest brand companies, all collaborating to building open APIs. But why the opensource and open API is important, enables collaboration, expedite innovation, we've done more in the past one year than what we could've done alone for 10 years, and that's really the power of opensource and open platforms >> I totally agree with you on this one. One of the things that we've really seen happen is as newer companies, these theoretically innovative companies have come online, cloud native companies, they've been very big on this open proponent, but we're also seeing large established companies move in the same direction, and it's allowing every organization to have that deeply innovative, flexible architecture that allows them to build new services without things breaking, so I think it's very exciting to see the breadth of companies that you had on stage talking about this, and the breadth of companies that are now in that. And the other thing that's interesting about it is they're competitors as well, right? So, there's that little bit of a edgy coopetition that's happening, but it's interesting to see that everybody feels that there's room for intense innovation in that space as well. So we've talked a little bit about opensource, we've talked about 5G, you are in advanced technology, and I think we'd be remiss to not talk about the big two letter acronym that's in the room that's not 5G, which would be AI. Tell me what's going on with AI, how are you guys thinking about it, what advice do you have for other organizations that are approaching it? Because you are actually a huge developer of AI across your entire organization, so maybe you could tee up a little bit about how that works. >> AI is transformational, and fundamental for AT&T. We have always developed AI solutions, and we were the first to deploy a AI in call centers 20 years ago. >> 20 years ago, really? >> 20 years ago. >> You were doing AI 20 years ago? >> 20 years ago. >> See, just goes to show. >> 20 years ago. I mean AI really, if you go to the source of AI, it really goes in the '40s and '50s with pioneers like Shannon and others. But the first deployment in a commercial call center, not a pilot, was really by AT&T. >> An actual implementation, yeah. >> With a service, we called it how may I help you. And the reason we put that out, because our customers were annoyed with press one for this and press two for billing, they wanted to speak naturally. And so we put the system that says "How may I help you?" and how may I help you allowed the customer to tell us in their own language, in their own words, what is it that they want from us as opposed to really dictating to them what they have to say Now today, it's really very hard for you to call any company in the world, without getting a service that uses some form of speech recognition or speech understanding. >> Thankfully. (laughs) >> But where we're applying it today and have been for the past two, three years, we're finding some really amazing opportunities that we've never imagined before. AI in its essence, is nothing more than automation leveraging data. So using your data as the oil, as the foundation, and driving automation, and that automation could be complete automation of a service, or it could be helping the human to doing their job better and faster. It could be helping a doctor in finding information about patients that they couldn't have done by themselves by processing a million records all together. We're doing the same thing at AT&T. The network is the most complex project ever to be created on the planet. And it's a complex project that changes every second of the day as people move around, and they try different devices. And so to be able to optimize that experience, is really an AI problem, so we apply it today to identify where to build the next cell sites all the way to what's the right ad to show to the customer, or, how do we really make your life easier with our services without you really calling our call center, how do I diagnose and repair your setup box before you're calling us? All of that foundation is really starting to be driven by AI technologies, very exciting. >> Well I'm actually very excited to see where AI takes us, and I'm excited to hear about what you're doing in the future. Thanks for takin' the time to come here today, >> It's my pleasure. >> And be with us on theCUBE. Thank you. >> It's always a pleasure talking to you, thank you very much. >> I'm Maribel Lopez closin' out at AT&T Spark, thank you. (bubbly music)

>> From the palace of fine arts in San Francisco it's the Cube. Covering AT&T Spark, now here's Jeff Frick. >> Hey, welcome back everybody, Jeff Frick here with the Cube. We're at the Palace of Fine Arts in San Francisco at the AT&T Spark event and it's all about 5G. 5G is this huge revolution and I haven't got a definitive number but it's something on the order of hundred X, improvement of speed and data throughput. There's a lot of excitement but one of the things that is less talked about here but it was actually up on the keynote was really the roll of opensource and AT&T talked a lot about opensource and how important it is and really redefining the company around the speed of software development versus the speed of hardware development and that's a big piece of it. We're excited to have somebody who knows all about opensource our next guest he's Jeff McCaffer. He's the Director of Opensource Progress Officer at Microsoft, did I get that right Jeff? >> No, well it said Opensource Programs Officer. >> Programs Officer. >> Yep. >> So you do all about opensource. >> Yeah. >> Well first off, welcome. >> Thank you very much Jeff, it's good to be here. >> So Microsoft, you know, no one would have ever thought, I mean, you know, I'm probably dating myself. 15 years ago with that Microsoft would be a big component of opensource but in fact they're a huge proponent of opensource. >> Absolutely, even just not so long ago you know, it was not the foremost in everybody's mind that Microsoft would be doing opensource. But now it's a core part of our company. It's a core part of how we work and our engagement with the rest of the industries. So it's really growing and it's continuing to grow. >> So how did it kind of get there and what are some of the real key components that you have to worry about in your role to managing, you know, participating in all these various communities all over the place. >> Yeah, well I mean it's been a long road but it's really the way software's happening today, you mention in the intro about the dispute of software versus hardware and software's just going so fast and you know, you can aspire to be world class but when everybody else starts there with opensource, you know it's really hard to start from zero get to there. So we're really happy to be you know, using opensource and contributing. One of the real challenges we've had going forward is the scale like simply we've got literally millions of uses of opensource across all of our products and services. And managing that, keeping track of it, engaging with those, all those communities and everything is a real big challenge. So we've been building paulo season tools and changing the culture to understand that you know, you need to engage, push fixes back, all those sorts of things. And then when we look at our releasing our software, we have thousands of opensource repositories on GitHub, thousands of developers at Microsoft working on GitHub repositories, our own and others in the community. So it's just managing all of that as being a really big challenge. >> Right and it's interesting cause the opensource projects themselves, we've seen at time and time again. You know, they fork and they go a lot of different directions. There's sometimes disagreement about direction. >> Sure. >> And prioritization, so you've got a kind of manage that within the opensource thread but as well as within, you know, where those products play a role in your products. >> Right, right and we've taken a sort of federated model in the company, we're very diverse as you know right and so my team sort of helps put guidelines in place for for project teams to run and then those project teams run their own program. How they engage with opensource, however they want to and sort of at the level they want to that matches their business requirements. So it ranges everywhere from people who are fully opensource to folks who are just you know, using a little bit of stuff here and there within their products. >> Right, what if you could speak a little to opensource and the role that it plays in employee happiness, employee retention cause you know, there's so much goodness and you see it at these shows. >> Absolutely. >> Where there's particular contributors that you know, they're rock stars in their community. They've made super important contributions. >> Yeah. >> They've managed the community and I always think back, if you're the person managing that person back at the office you know, how much time do they put into their opensource effort? >> Sure. >> How much time do they put in their company efforts? How much of their time is really the company software that's built on top of that opensource. >> Yeah. >> And how do you manage that because it is a really important piece for a lot of people's personas. >> Absolutely. >> And their self values. >> Yeah, well and there's been a lot of research that says also that high performing teams, one of the traits of high performing teams is engaging in opensource. And at the personal level like individuals, there's kind of a different set of possibilities there, you know, either you're engaging in opensource for part of your product work, right, so that's sponsored by the company. Or you might be doing some things on the side or some tangential range in between there, right? >> Right. >> And sort of all of those you need to drive to the appropriate level, the folks who are working on it day to day for their, for the company. There's some really interesting dynamics that can get setup. Super exciting for the team, sometimes it can get a little waylaid maybe but you know, you want to keep them, keep them on task. But then also the, the folks who are doing it of their own volition, like on their own time and that sort of thing. That also brings back a bunch of energy and everything into the workplace. New technologies that they'll discover in their area and they'll bring back the energy and the excitement about engaging back to the regular team. >> Right. >> So there's lots of possibilities there. >> So what brings you here, what brings you to AT&T Spark today? >> Well they invited me to speak on a panel earlier today about opensource and the future of opensource and so I had a, there were a couple of other people from Linux Foundation and from AT&T. So we had a good conversation on stage. >> Yeah it's pretty interesting how, pretty much all these projects you know, eventually get put in to the Linux Foundation. That they, you know, they've just kind of become this defacto steward for a wide variety of opensource projects. >> Yeah, well there's a number of different foundations, Linux Foundation's certainly one of the better known ones, the Eclipse Foundation, Apache. >> Right, Apache yeah, right. >> Been around lots of times doing lots of good things. So there's a ton of amazing projects out there in all of these foundations. And it's just super exciting to see them all be engaging like in this sort of cohesive right, and with a good governance model. >> Right. >> Yep. >> So I'll give you the last word, one of my favorite lines always that's opensource is opensource is free like a puppy. >> Yes, it's totally free like a puppy. >> So, you know, you're living in that world, what is one of the things about opensource that most people miss, one of the really positive attributes that most people just don't see. And then what's one of the big, you know kind of biggest, kind of ongoing challenges, that's just part of operating in this opensource world? >> Well I mean, I phrase it in challenges and opportunities, right, there are obviously lots of challenges, like I was saying with scale and managing security. And the culture change that goes around collaboration and that sort of thing. The opportunities, I think are boundless really, I mean there's, one of the most gratifying things that you can see as an opensource project, is people take your technology and use it in ways you never imagined. Right, so there's, we can think of that as our products too and we take our products and they've got opensource APIs. They've got opensource frameworks and such. And people take them and do amazing things with them that we never imagined possible. And that just, that part is really exciting and invigorating. >> Yeah, alright Jeff well thanks for taking a few minutes. >> Sure. >> Congrats on all your work and I guess we'll see you in Orlando in a month or so. >> Okay, possibly. >> Alright, he's Jeff, I'm Jeff, we're all Jeffs here and we're at the Palace of Fine Arts at AT&T Spark, thanks for watching, see you next time. (upbeat music)

>> From the Palace of Fine Arts in San Francisco, it's theCUBE, covering AT&T Spark. Now here's Jeff Frick. >> Hey, welcome back, everybody. Jeff Frick here with theCUBE. We're at the Palace of Fine Arts in San Francisco at the AT&T Spark event. It's really all about 5G and what 5G is going to enable. You know, this is a really big technology that's very, very close. I think a lot closer than most people understand. And one of the most important components of 5G is it was designed from the ground up really not so much for people-to-people communications as much as machine-to-machine communications. So we're really excited to have someone who's right in the thick of that and talk about the implications, especially another topic that we hear all the time, which is Edge computing. So it's Alicia Abella. She is the VP of Operational Automation in Program Management from AT&T Labs. Alicia, welcome. >> Thank you for having me, Jeff. >> Absolutely. So we were talking a little bit before we turn on the cameras about 5G and Edge computing. And how the two, while not directly tied together, are huge enablers of one another. I wonder if you can unpack a little bit about why is 5G such an important component to kind of the vision of Edge computing? >> Sure, absolutely. Yeah, happy to do so. So Edge computing is really about bringing processing power closer to the end device, closer to the end user, where a lot of the processing data analytics has to occur. And you want to do that because you want to be able to deliver the services and applications close to the edge, close to where the customer is, so that you can deliver on the speeds that those applications need. 5G plays a role because 5G is promising to be very fast and also very reliable and very secure. So now you've got three things to your advantage paired up with Edge to be able to deliver on a lot of these use cases that we hear a lot about when we talk about 5G, when we talk about Edge. Some example use cases are the autonomous vehicle. The autonomous vehicle is a classic example for Edge computing as well as 5G. And in fact, it illustrates a kind of continuum, because you can have processing that always has to remain in the car. Anything related to safety? That processing has to happen right on that device. The device in this case being the car. But there are other processing capabilities, like maybe updates to real-time maps. That could happen on the Edge. You still have to be near real-time, so you want to have that kind of processing and updating happening at the Edge. Then maybe you have something where you want to download some new entertainment, a movie to your car. Well, that can reside back at the data center, further away from where the device or the car is. So you've got this continuum. >> So really, what the 5G does is really open up the balance of how you can distribute that store computing and communications. It's always about latency. At the end of the day, it's always about latency. And as much as we want to get as much compute close, oh, and also, I guess power. Power and latency. >> Power and Edge actually go hand-in-hand as well. >> It's a big deal, right? >> Mhm. >> So what you're saying is, because of 5G, and the fact that now you have a much lower latency, faster connectivity port, you can now have some of that stuff maybe not at the Edge and enable that Edge device to do more, other things? >> Yes, so I often like to say that we are unleashing the device away from having it be tethered to the compute processor that's handling it and now you can go mobile. Because now what you do is, if the processing is happening on the Edge and not on the device, you save on battery life, but you also make the device more lightweight, easier to manage, easier to move around. The form factor can become smaller. So there's also an advantage to Edge computing to the device as well. >> Right. It's pretty interesting. There was an NVIDIA demo in the keynote of running a video game on the NVIDIA chips in a data center and pumping a really high resolution experience back out to the laptop screen I think is what he was using it for. And it's a really interesting use case in how when you do have these fast, reliable networks, you can shift the compute, and not just a peer compute, but the graphics, et cetera, and really start to redistribute that in lots of different ways that were just not even fathomable before. Before you had to buy the big gaming machine. You had to buy the big, giant GPU. You had to have that locally, and all that was running on your local machine. You just showed a demo where it's all running back in their data center in Santa Clara. Really opens up a huge amount of opportunity. >> That's right. So Edge computing is really distributed in nature. I mean, it is all about distribution. And distributing that compute power wherever you need it. Sprinkling it across the country of where you need it. So we've gone, there's been this pendulum shift, where we started with the mainframe, big rooms, lots of air conditioning, and then the pendulum swung over to the PC. And that client-server model. Where now you had your PC and you did your computing locally. And then it swung back the other way for Cloud computing where everything was centralized again and all that compute power was centralized. And now the pendulum is swinging back again the other way to this distributed model where now you've got your compute capabilities distributed across the country where you need it. >> Right. So interesting. I mean, networking was the last of the virtualized platform between storage and compute, and then finally networking. But if you really start to think of a world with basically infinite power, compute, infinite store, and infinite networking, basically asymptotically approaching zero pricing. Think of the world from that way. We're not there. We're never going to get to that absolute place, but it really opens up a lot of different ways to think about what you could do with that power. So I wonder if there's some other things you can share with us. At Labs, you guys are looking forward to this 5G world. What are some of the things that you see that just, wow, I would have never even thought that was even in the realm of possibility that some people are coming up with? >> Yeah. >> Any favorites? >> Oh, I think one of our favorites is certainly looking at the case of manufacturing. Even though you would think of manufacturing as very fixed, the challenge with manufacturing is that a lot of those robotics capabilities that are in the manufacturing assembly lines, for example, they're all based on wires and they can't change and upgrade what they're doing very quickly. So being able to deliver 5G, have things that are wireless, and have Edge compute capabilities that are very powerful means that they can now shift and move around their assembly lines very quickly. So that's going to help the economy. Help those businesses be able to adapt more quickly to changes in their businesses. And so that's one that is quite exciting to us. And I would say the next one that's also exciting for us would be, we talked about autonomous vehicles already, 'cause that one's kind of far out, right? >> I don't think it's as far as most people think, actually. We covered a lot of autonomous vehicle companies, and there's just so much research being done now. I don't think it's as far out as people think. >> Yes, and so I think we are definitely committed to deploying Edge compute. And in the process, from a more technical perspective, I think one of the things that we are going to be interested in doing is, and you alluded to it before, is how do you manage all of those applications and services and distribute them in a way that is economical, that we can do it at scale, that we can do it on demand? So that too is part of what's exciting about being able to deploy Edge. >> Yeah. It's pretty interesting, the manufacturing example, 'cause it came up again in the keynote to really embracing software-defined, embracing open source. And the takeaway was moving at the speed of software development, not moving at the speed of hardware development. Because software moves a lot faster. And can be more flexible. It's easy to respond to market demands, or competitive demands, or just to innovate a lot faster. So really taking that approach, and obviously a lot of conversation about you guys in the open Stack community and the open-source projects enables you and your customers to start to adapt to software-defined innovation as opposed to just pure hardware-defined innovation. >> That's right. That's right, yup. >> Alright, Alicia, I'll give you the final word. Any surprises? Oh, no, you've got a chat coming up, so why don't you give us a quick preview for what your conversation is going to be about later today? >> Yeah, thank you, Jeff. So yeah, later I'll be talking about AT&T's initiatives around encouraging women to pursue stem fields. In particular, computer science. It turns out that the number of women getting undergraduate degrees in computer science peaked in the mid-80s. And it's been going downhill since. Last year, only 17% of women were getting degrees in computer science So AT&T's mission, and what we announced today was a million dollar donation to the Girls Who Code organization. That's one of many different non-profit organizations that AT&T is involved with to make sure that we continue to encourage young women and also underrepresented minorities and others who want to get in the stem fields to get involved because technology is changing very quickly. We need people who can understand the technology, who can develop the software we talked about, and we need to get that pipeline filled up. And so we're very committed to helping the community and helping to encourage young girls to pursue degrees in stem. >> That's great. Girls Who Code is a fantastic organization. We've had 'em on. Anita Borg, I mean, there's so much good work that goes on out there, so that's a great announcement. And congratulations. >> Thank you. >> And I'm sure that's a meaningful contribution. >> Yeah, thank you. >> So Alicia, thanks for stopping by, and good luck this afternoon, and we'll see you next time. >> Thank you, Jeff. >> Alright. >> Appreciate it. >> She's Alicia, I'm Jeff. You're watching theCUBE. We're at AT&T Spark in downtown San Francisco. Thanks for watching. (upbeat electronic music)

>> From the Palace of Fine Arts in San Francisco, it's theCUBE covering AT&T Spark. (techy music) Now here's Jeff Frick. >> Hey, welcome back, everybody, Jeff Frick here with theCUBE. We're at the Palace of Fine Arts in San Francisco at the AT&T Spark event. It's all about 5G, you know we've been hearing about 5G for a long, long time, that 5G is coming, it's in cities, there's more cities that it's rolling out to, it's lots of special networks, so we're excited to be here as it becomes real, and we've got a guy who's right in the middle of the weeds, right in all the devices. He's Gordon Mansfield, the VP of converged access and device technology at AT&T, Gordon, welcome. >> Thank you. >> So, what do you think? You've probably been looking at this 5G stuff for a long, long time. It feels like we're finally getting pretty close. >> We're getting really close, you know, we're gearing up to launch our first 12 markets this year, and just this past weekend we made the first end-to-end call across our production network with a mobile form factor device, so we're real close and we're real excited. >> So, that just happened, right, this first call? >> It just happened this past weekend. So, what were some of the final hurdles to finally get that little milestone that you guys have probably been looking forward to for while? >> Yeah, so the final hurdles is really getting the device modems into, you know, that form factor device, that mobile form factor to where it can be portable, you can carry it-- >> Right. >> And make these fantastic mobile data calls, and so getting that technology, working together, communicating with the network infrastructure, that work just finished, or there's multiple stages, but a critical stage just got completed last week. We were able to take that technology straight to the field in Waco, Texas, and start demonstrating and working with it live in our production network. >> So, do you get the dog out and he can hear his master's voice when you do that first phone call? >> Well... (laughs) You know... >> The old RCA. >> It's pretty close. >> I know, nobody knows what we're even talking about, right, too old. >> (chuckles) They don't, do they? >> So, the other thing that's really interesting about 5G compared to the other, prior roll-outs is really the focus on devices, and you're in charge of devices and devices is a lot more than just handsets, right? >> Yeah. >> This was really designed for the industrial internet and IoT, and really a whole swath of device-to-device communication. How did that kind of change the way you look at your job? >> Yeah, so you know, we've been working on IoT and modules in the IoT space, but with 5G you start to enable lots of new capabilities with very high bandwidth, low latent applications, which allows us to revolutionize various vertical industries, and so now it's no longer just about the smartphone or the tablet, but it's about anything and everything that you can imagine-- >> Right. >> And so, you know, I tell people all the time, you know, when we first start talking about technology we really think about some cool things, but the reality is we barely touch the surface, and so you know, people will just begin to imagine the capabilities that 5G will unleash and you'll start to put, you know, the capabilities into everything from a refrigerator to robot arms on a manufacturing floor and all kinds of points in between. >> Right, you know it's funny, we go to a lot of tech conferences, and we were just at VMworld a couple weeks ago and you know, Michael Dell said on air that, you know, the edge will actually be bigger than the cloud, and right, it's been all about cloud for the last several years. >> Yeah. >> Now it's all about edge. Well, the key to edge is connectivity, and that's a really important piece of the 5G story. >> Absolutely, if you take your compute power and you push it further to the edge you've got to then connect, and so you can put very low-cost, low horsepower components on the edge, connect them, you know, so in a device, connect them to the edge and come up with some pretty powerful capabilities. >> Yeah, and the other interesting thing from your guys' point of view, having dealt with handsets for so long, is just the whole low power, and a lot of the edge type applications are going to be in remote areas, difficult to get to areas, difficult to plumb areas, so the whole experience with low power combined with the low latency is really a big game changer. >> That's absolutely correct, so when you take low power you can put battery devices that last years-- >> Right. >> And have them in remote locations, sensors, et cetera, and have them connect in a low-latent, high-bandwidth way to deliver, you know, anything that you can imagine. >> Right, so it feels to me that there's really not the buzz around 5G that there should be, and I don't know because we've kind of heard about it for a while and it's kind of been in extended development or people just aren't paying attention, but what's interesting, a lot of conversations in the keynotes talking about experiences. >> Mm-hm. >> Really changing the way you can think about developing applications for experiences based on this technology. We saw the NVIDIA demo where they're running NVIDIA processors in their cloud and sending it to a laptop here, where before you'd have to spend thousands of dollars on a local machine. As you look back, what are some of the things that you've seen, either in testing or conversations, that maybe people just don't have any perception of how this is going to change some of their day-to-day activities? >> So, I don't think people, you know, unfortunately we've become immune. The devices, right, the processing power that we put in devices that people carry in their pocket, they keep going up and up. The reality is at some point you've got to flatten that to... From a consumer perspective you've got to flatten that to have a device that people can afford. >> Right. >> And so, with 5G and you start putting things to the edge you start taking away some of the processing power that physically is in the phone and you put that at the edge, to where now people can have really high speed, high capabilities in a relatively low-cost device. >> That's pretty interesting, you're the first person. So, it is really this redistribution of, you know, networking, compute, and store-- >> Mm-hm. >> That's now enabled with this fast networking, where before your options were really not so great. >> Yeah, it's always a balance, but today your only option is to continue to put more and more horsepower into the device itself-- >> Right. >> More processing, compute storage, into the device. By spreading that and having some of it maintained in the network you can maintain, you can manage cost in the end user device that people carry in their pocket. >> Okay, so give you the last word, when you are at a cocktail party on the weekend talking to some people about what you do, what surprises people most about 5G once you tell them it's this new thing that's coming down the pike? >> Well, you know, look, in my job I get to see lots of cool things, and when I start describing some virtual or augmented reality, imagine walking down the street with a pair of glasses and suddenly images, right, start, you know, being fed on top of what you're really looking at. You start, you know, you can imagine a day where, you know, an advertisement may pop up in your field of view, or you know, points of interest that you might want to see, and you know, obviously we've got to control that and manage it to consumer expectations, but that's not as far away as people might imagine. >> Right, and just to recap, you're in 12 markets. >> 12 markets-- >> You're in seven, five more, and then another seven coming, right? >> That's right, so 12 by the end of '18, and seven more in early '19. We're off to a fast start and looking to grow from there. >> All right, Gordon, well congratulations on progress to date and good luck with the roll-out. >> All right, thank you. >> (chuckles) All right, he's Gordon, I'm Jeff, you're watching theCUBE. We're at AT&T Spark in San Francisco, thanks for watching. (techy music)

>> From the Palace of Fine Arts in San Francisco, it's theCUBE, covering AT&T Spark. Now here's Jeff Frick. >> Hey welcome back everybody, Jeff Frick here with theCUBE. We're at San Francisco, at the historic Palace of Fine Arts, it's a beautiful spot, it's redone, they moved Exploratorium out a couple years ago, so now it's in a really nice event space, and we're here for the AT&T Spark Event, and the conversation's all around 5G. But we're excited to have our first guest, and he's working on something that's a little bit tangential to 5G-related, but not absolutely connected, but really important work, it's Chris Sambar, he is the SVP of FirstNet at AT&T, Chris, great to see you. >> Thanks Jeff, great to be here, I appreciate it. >> Yeah, so you had a nice Keynote Presentation, talking about FirstNet. So for people I've missed, that aren't familiar, what is AT&T FirstNet? >> Sure, I'll give a quick background. As I was mentioning up there, tomorrow is the 17-year Anniversary of 9/11. So 17 years ago tomorrow, a big problem in New York City. Lots of first responders descended on the area. All of them were trying to communicate with each other, they were trying to use their radios, which they're you know, typically what you see a first responder using, the wireless networks in the area. Unfortunately challenges, it wasn't working. They were having trouble communicating with each other, their existing wireless networks were getting congested, and so the 9/11 Commission came out with a report years later, and they said we need a dedicated communications network, just for First Responders. So they spun all this up and they said, we're going to dedicate some Spectrum, 20 megahertz of D-Class Spectrum, which is really prime Spectrum. Seven billion dollars and we're going to set up this Federal entity, called the FirstNet Authority, and they're going to create a Public Safety Network across America. So FirstNet Authority spent a few years figuring out how to do it, and they landed on what we have today, which was a Public/Private Partnership, between AT&T, and Public Safety throughout America, and we're building them a terrific network across the country. It is literally a separate network so when I, I think of wireless in America, I think of four main commercial carriers, AT&T, Verizon, T-Mobile, Sprint. This is the 5th carrier, this is Public Safety's Wireless Network just for them. >> So when you say an extra network, so it's a completely separate, obviously you're leveraging infrastructure, like towers and power and those types of things. But it's a completely separate network, than the existing four that you mentioned. >> Yeah, so if you walk into our data centers throughout the country, you're going to see separate hardware, physical infrastructure that is just for FirstNet, that's the core network just for this network. On the RAN, the Radio Access Network, we've got antennas that have Band 14 on them, that's Public Safety's Band, dedicated just for them when they need it. So yeah, it's literally a physically separate network. The SIM card that goes into a FirstNet device, is a different SIM card than our commercial users would use, because it's separate. >> So one of the really interesting things about 5G, and kind of the evolution of wireless is, is taking some of the load that has been taken by like WiFi, and other options for fast, always on connectivity. I would assume radio, and I don't know that much about radio frequencies that have been around forever with communications in, in First Responders. Is the vision that the 5G will eventually take over that type of communication as well? >> Yeah, absolutely. If you look at the evolution of First Responder, and Public Safety Communications, for many years now they've used radios. Relatively small, narrow Spectrum bands for Narrow Band Voice, right, just voice communications. It really doesn't do data, maybe a little bit, but really not much. Now they're going to expand to this Spectrum, the D-Class, the D-Block Spectrum, excuse me, which is 700 megahertz, it's a low-band Spectrum, that'll provide them with Dedicated Spectrum, and then the next step, as you say, is 5G, so take the load off as Public Safety comes into the, the new Public Safety Communications space, that they've really been wanting for years and years, they'll start to utilize 5G as well on our network. >> So where are you on the development of FirstNet, where are you on the rollout, what's the sequence of events? >> The first thing we did, the award was last year in March 2017. The first thing we did was we built out the core network. When I talked about all that physical infrastructure, that basically took a year to build out, and it was pretty extensive, and about a half a billion dollars so, that was the first thing we did, that completed earlier this year. >> Was that nationwide or major metro cities or how-- >> Nationwide, everywhere in the country. >> Nationwide, okay. >> So now what we're doing is, we are putting the Spectrum that we were given, or I should say we were leased for 25 years, we're putting that Spectrum up across our towers all over the country so, that will take five years, it's a five-year build-out, tens of thousands of towers across America, will get this Public Safety Spectrum, for Public Safety, and for their use. >> Right. Will you target by GEO, by Metro area, I mean, how's it going to actually happen? That's a huge global rollout, five years is a long time. How you kind of prioritize, how are you really going to market with this? >> The Band 14 Spectrum is being rolled out in the major, the major dense areas across the country. I will tell you that by the end of the rollout, five years from now, 99% of the population of America, will have Band 14 Spectrum, so the vast majority of the population. Other areas where we don't roll it out, rural areas for example, all of the features that Public Safety wants, we call them (mumbles) and priority, which is the features to allow them to always have access to the network whenever they need it. Those features will be on our regular commercial Spectrum. So if Band 14 isn't there, the network will function exactly as if it were there for them. >> Right. Then how do you roll it out to the agencies, all the First Responders, the Fire, the Police, the EMTs, et cetera? How do they start to take advantage of this opportunity? >> Sure, so we started that earlier this year. We really started in a March-April timeframe in earnest, signing up agencies, and the uptake's been phenomenal. It's over 2500 Public Safety Agencies across America, over 150,00, and that number grows by thousands every week. That's actually a pretty old number but, they are signing up in droves. In fact, one of the problems we were having initially is, handling the volume of First Responders that wanted to sign up, and the reason is they're seeing that, whether it's a fire out in Oregon, and they need connectivity in the middle of nowhere, in a forest where there's no wireless connectivity at all, we'll bring a vehicle out there, put up an antenna and provide them connectivity. Whether it's a Fourth of July show, or a parade, or an active shooter, wherever large groups of people, combined together and the network gets congested, they're seeing that wow, my device works no matter what. I can always send a text message, I can send a video, it just works. Where it didn't work before. So they love it, and they're really, they're really signing up in droves, it's great. >> It's really interesting because it's, it's interesting that this was triggered, as part of the post 9/11 activity to make things better, and make things safer. But there was a lot of buzz, especially out here in the West with, with First Responders in the news, who were running out of band width. As you said, the Firefighters, the fire's been burning out here, it seems like forever, and really nobody thinking about those, or obviously they're probably roaming on their traditional data plan, and they're probably out there, for weeks and weeks at a time, that wasn't part of their allocation, when they figured out what plan they should be. So the timing is pretty significant, and there's clearly a big demand for this. >> Absolutely. So that example that you sight is a really good one. Two weeks ago, there was a lot in the news about a fire agency in the West, that said they were throttled by their carrier. It was a commercial carrier, and commercial carriers have terms and conditions, that sometimes they need to throttle usage, if you get to a certain level. That's how commercial networks work. >> Right, right. >> FirstNet was built with not only different technology, hardware, software, but with different terms and conditions. Because we understand that, when a First Responder responds to your house, we don't want that to be the minute in time, when their network, their plan got maxed out, and now they're getting throttled. >> Right. >> So we don't have any throttling on the FirstNet Network. So it's not only the hardware, software, technical aspects of the network, but the terms and conditions are different. It's what you would expect that a First Responder would have and want on their device, and that's what we're providing for them. >> Right, and the other cool thing that you mentioned is, we see it all the time, we go to a lot of conferences. A lot of people probably experience it at, at big events right, is that still today, WiFi and traditional LTE, has hard times in super-dense environments, where there's just tons and tons and tons of bodies I imagine, absorbing all that signal, as much as anything else, so to have a separate Spectrum in those type of environments which are usually chaotic when you got First Responders, or some of these mass events that you outlined, is a pretty important feature, to not get just completely wiped out by everybody else happening to be there at the same time. >> Exactly. I'll give you two quick examples, that'll illustrate what you just said. The first one is, on the Fourth of July, in downtown Washington D.C. You can imagine that show. It's an awesome show, but there are hundreds of thousands of people that gather around that Washington Monument, to watch the show. And the expectation is at the peak of the show, when all those people are there, you're not really going to be sending text messages, or calling people, the network's probably just not going to work very well. That's, we've all gotten used to that. >> Right, right. >> This year, I had First Responders, who were there during the event, sending me videos of the fireworks going off. Something that never would've been possible before, and them saying oh my gosh. The actually works the way it's supposed to work, we can use our phones. Then the second example, which is a really sad example. There was a recent school shooting down in Florida. You had Sheriffs, Local Police, Ambulances. You even had some Federal Authorities that showed up. They couldn't communicate with each other, because they were on different radio networks. Imagine if they had that capability of FirstNet, where they could communicate with each other, and the network worked, even though there were thousands of people that were gathering around that scene, to see what was going on. So that's the capability we're bringing to Public Safety, and it's really good for all of us. >> Do you see that this is kind of the, the aggregator of the multi-disparate systems that exist now, as you mentioned in, in your Keynote, and again there's different agencies, they've got different frequencies, they've got Police, Fire, Ambulance, Federal Agencies, that now potentially this, as just kind of a unified First Responder network, becomes the defacto way, that I can get in touch with anyone regardless of of where they come from, or who they're associated with? >> That is exactly the vision of FirstNet. In major cities across America, Police, Fire, Emergency Medical typically, are on three different radio networks, and it's very difficult for them to communicate with each other. They may have a shared frequency or two between them, but it's very challenging for them. Our goal is to sign all of them up, put them on one LTE network, the FirstNet Network, customized for them, so they can all communicate with each other, regardless of how much congestion is on the network. So that's the vision of FirstNet. >> Then that's even before you get into the 5G impacts, which will be the data impacts, whereas I think again, you showed in some of your examples, the enhanced amount of data that they can bring to bear, on solving a problem, whether it's a layout of a building for the Fire Department or drone footage from up above. We talked to Menlo Park Fire, they're using drones more and more to give eyes over the fire to the guys down on the ground. So there's a lot of really interesting applications that you can get more better data, to drive more better applications through that network, to help these guys do their job. >> Yeah, you've got it, the smart city's cameras, don't you want that to be able to stream over the network, and give it to First Responders, so they know what they're going to encounter, when they show up to the scene of whatever issue's going on in the city, of course you do, and you need a really reliable, stable network to provide that on. >> Well Chris, this is not only an interesting work, but very noble, and an important work, so appreciate all of the efforts that you're putting in, and thanks for stopping by. >> I appreciate it Jeff, it's been great talking with you. >> Alright, he's Chris, I'm Jeff, you're watching theCUBE, we're in San Francisco at the Palace of Fine Arts, at AT&T Spark. Thanks for watching, we'll see you next time. (bright music)

>> Announcer: Live from San Francisco, it's the Cube covering Spark Summit 2017, brought to you by databricks. >> Hi, we're back here where the Cube is live, and I didn't even know it Welcome, we're at Spark Summit 2017. Having so much fun talking to our guests I didn't know the camera was on. We are doing a talk with Cloudera, a couple of experts that we have here. First is Mark Grover, who's a software engineer and an author. He wrote the book, "Dupe Application Architectures." Mark, welcome to the show. >> Mark: Thank you very much. Glad to be here. And just to his left we also have Jennifer Wu, and Jennifer's director of product management at Cloudera. Did I get that right? >> That's right. I'm happy to be here, too. >> Alright, great to have you. Why don't we get started talking a little bit more about what Cloudera is maybe introducing new at the show? I saw a booth over here. Mark, do you want to get started? >> Mark: Yeah, there are two exciting things that we've launched at least recently. There Cloudera Altus, which is for transient work loads and being able to do ETL-Like workloads, and Jennifer will be happy to talk more about that. And then there's Cloudera data science workbench, which is this tool that allows folks to use data science at scale. So, get away from doing data science in silos on your personal laptops, and do it in a secure environment on cloud. >> Alright, well, let's jump into Data Science Workbench first. Tell me a little bit more about that, and you mentioned it's for exploratory data science. So give us a little more detail on what it does. >> Yeah, absolutely. So, there was private beta for Cloudera Data Science Workbench earlier in the year and then it was GA a few months ago. And it's like you said, an exploratory data science tool that brings data science to the masses within an enterprise. Previously people used to have, it was this dichotomy, right? As a data scientist, I want to have the latest and greatest tools. I want to use the latest version of Python, the latest notebook kernel, and I want to be able to use R and Python to be able to crunch this data and run my models in machine learning. However, on the other side of this dichotomy are the IT organization of the organization, where if they want to make sure that all tools are compliant and that your clusters are secure, and your data is not going into places that are not secured by state of the art security solutions, like Kerberos for example, right? And of course if the data scientists are putting the data on their laptops and taking the laptop around to wherever they go, that's not really a solution. So, that was one problem. And the other one was if you were to bring them all together in the same solution, data scientists have different requirements. One may want to use Python 2.6. Another one maybe want to use 3.2, right? And so Cloudera Data Science Workbench is a new product that allows data scientists to visualize and do machine learning through this very nice notebook-like interface, share their work with the rest of their colleagues in the organization, but also allows you to keep your clusters secure. So it allows you to run against a Kerberized cluster, allows single sign on to your web interface to Data Science Workbench, and provides a really nice developer experience in the sense that My workflow and my tools and my version of Python does not conflict with Jennifer's version of Python. We all have our own docker and Kubernetes-based infrastructure that makes sure that we use the packages that we need, and they don't interfere with each other. We're going to go to Jennifer on Altus in just a few minutes, but George first give you a chance to maybe dig in on Data Science workshop. >> Two questions on the data science side: some of the really toughest nuts to crack have been Sort of a common environment for the collaborators, but also the ability to operationalize the models once you've sort of agreed on them, and manage the lifecycle across teams, you know? Like, challenger champion, promote something, or even before that doing the ab testing, and then sort of what's in production is typically in a different language from what, you know, it was designed in and sort of integrating it with the apps. Where is that on the road map? Cause no one really has a good answer for that. >> Yeah, that's an excellent question. In general I think it's the problem to crack these days. How do you productionalize something that was written by a data scientist in a notebook-like system onto the production cluster, right? And I think the part where the data scientist works in a different language than the language that's in production, I think that problem, the best I can say right now is to actually have someone rewrite that. Have someone rewrite that in the language you're going to make in production, right? I don't see that to be the more common part. I think the more widespread problem is even when the language is production, how do you go making the part that the data scientist wrote, the model or whatever that would be, into a prodution cluster? And so, Data Science Workbench in particular runs on the same cluster that is being managed by Cloudera manager, right? So this is a tool that you install, but that is available to you as a web server, as a web interface, and so that allows you to move your development machine learning algorithms from your data science workbench to production much more easier, because it's all running on the same hardware and same systems. There's no separate Cloudera managers that you have to use to manage the workbench compared to your actual cluster. >> Okay. A tangential question, but one of the, the difficulties of doing machine learning is finding all the training data and, and sort of data science expertise to sit with the domain expert to, you know, figure out proper model of features, things like that. One of the things we've seen so far from the cloud vendors is they take their huge datasets in terms of voice, you know, images. They do the natural language understanding, speech or rather text to speech, you know, facial recognition. Cause they have such huge datasets they can train on. We're hearing noises that they'd going to take that down to the more mundane statistical kind of machine learning algorithms, so that you wouldn't be, like, here's a algorithm to do churn, you know, go to town, but that they might have something that's already kind of pre-populated that you would just customize. Is that something that you guys would tackle, too? >> I can't speak for the road map in that sense, but I think some of that problem needs to be tackled by projects like Spark for example. So I think as the stack matures, it's going to raise the level of abstraction as time goes on. And I think whatever benefits Spark ecosystem will have will come directly to distributions like Cloudera. >> George: That's interesting. >> Yeah >> Okay >> Alright, well let's go to Jennifer now and talk about Altus a little bit. Now you've been on the Cube show before, right? >> I have not. >> Okay, well, familiar with your work. Tell us again, you're the product manager for Altus. What does it do, and what was the motivation to build it? >> Yeah, we're really excited about Cloudera Altus. So, we released Cloudera Altus in its first GA form in April, and we launched Cloudera Altus in a public environment in Strata London about two weeks ago, so we're really excited about this and we are very excited to now open this up to all of the customer base. And what it is is a platform as a service offering designed to leverage, basically, the agility and the scale of cloud, and make a very easy to use type of experience to expose Cloudera capacity for, in particular for data engineering type of workloads. So the end user will be able to very easily, in a very agile manner, get data engineering capacity on Cloudera in the cloud, and they'll be able to do things like ETL and large scale data processing, productionized machine learning workflows in the cloud with this new data engineering as a service experience. And we wanted to abstract away the cloud, and cluster operations, and make the end user a really, the end user experience very easy. So, jobs and workloads as first class objects. You can do things like submit jobs, clone jobs, terminate jobs, troubleshoot jobs. We wanted to make this very, very easy for the data engineering end user. >> It does sound like you've sort of abstracted away a lot of the infrastructure that you would associate with on-prem, and sort of almost make it, like, programmable and invisible. But, um, I guess my, one of my questions is when you put it in a cloud environment, when you're on-prem you have a certain set of competitors which is kind of restrictive, because you are the standalone platform. But when you go on the cloud, someone might say, "I want to use red shift on Amazon," or Snowflake, you know, as the MPP sequel database at the end of a pipeline. And it's not just, I'm using those as examples. There's, you know, dozens, hundreds, thousands of other services to choose from. >> Yes. >> What happens to the integrity of that platform if someone carves off one piece? >> Right. So, interoperability and a unified data pipeline is very important to us, so we want to make sure that we can still service the entire data pipeline all the way from ingest and data processing to analytics. So our team has 24 different open source components that we deliver in the CDH distribution, and we have committers across the entire stack. We know the application, and we want to make sure that everything's interoperable, no matter how you deploy the cluster. So if you deploy data engineering clusters through Cloudera Altus, but you deployed Impala clusters for data marks in the cloud through Cloudera Director or through any other format, we want all these clusters to be interoperable, and we've taken great pains in order to make everything work together well. >> George: Okay. So how do Altus and Sata Science Workbench interoperate with Spark? Maybe start with >> You want to go first with Altus? >> Sure, so, we, in terms of interoperability we focus on things like making sure there are no data silos so that the data that you use for your entire data lake can be consumed by the different components in our system, the different compute engines and different tools, and so if you're processing data you can also look at this data and visualize this data through Data Science Workbench. So after you do data ingestion and data processing, you can use any of the other analytic tools and then, and this includes Data Science Workbench. >> Right, and for Data Science Workbench runs, for example, with the latest version of Spark you could pick, the currently latest released version of Spark, Spark 2.1, Spark 2.2 is being boarded of course, and that will soon be integrated after its release. For example you could use Data Science Workbench with your flavor of Spark two's version and you can run PySpark or Scala jobs on this notebook-like interface, be able to share your work, and because you're using Spark Underneath the hood it uses yarn for resource management, the Data Science Workbench itself uses Docker for configuration management, and Kubernetes for resource managing these Docker containers. >> What would be, if you had to describe sort of the edge conditions and the sweet spot of the application, I mean you talked about data engineering. One thing, we were talking to Matei Zaharia and Ronald Chin about was, and Ali Ghodsi as well was if you put Spark on a database, or at least a, you know, sophisticated storage manager, like Kudu, all of a sudden there're a whole new class of jobs or applications that open up. Have you guys thought about what that might look like in the future, and what new applications you would tackle? >> I think a lot of that benefit, for example, could be coming from the underlying storage engine. So let's take Spark on Kudu, for example. The inherent characteristics of Kudu today allow you to do updates without having to either deal with the complexity of something like Hbase, or the crappy performance of dealing HDFS compactions, right? So the sweet spot comes from Kudu's capabilities. Of course it doesn't support transactions or anything like that today, but imagine putting something like Spark and being able to use the machine learning libraries and, we have been limited so far in the machine learning algorithms that we have implemented in Spark by the storage system sometimes, and, for example new machine learning algorithms or the existing ones could rewritten to make use of the update features for example, in Kudu. >> And so, it sounds like it makes it, the machine learning pipeline might get richer, but I'm not hearing that, and maybe this isn't sort of in the near term sort of roadmap, the idea that you would build sort of operational apps that have these sophisticated analytics built in, you know, where the analytics, um, you've done the training but at run time, you know, the inferencing influences a transaction, influences a decision. Is that something that you would foresee? >> I think that's totally possible. Again, at the core of it is the part that now you have one storage system that can do scans really well, and it can also do random reads and writes any place, right? So as your, and so that allows applications which were previously siloed because one appication that ran off of HDFS, another application that ran out of Hbase, and then so you had to correlate them to just being one single application that can use to train and then also use their trained data to then make decisions on the new transactions that come in. >> So that's very much within the sort of scope of imagination, or scope. That's part of sort of the ultimate plan? >> Mark: I think it's definitely conceivable now, yeah. >> Okay. >> We're up against a hard break coming up in just a minute, so you each get a 30-second answer here, so it's the same question. You've been here for a day and a half now. What's the most surprising thing you've learned that you thing should be shared more broadly with the Spark community? Let's start with you. >> I think one of the great things that's happening in Spark today is people have been complaining about latency for a long time. So if you saw the keynote yesterday, you would see that Spark is making forays into reducing that latency. And if you are interested in Spark, using Spark, it's very exciting news. You should keep tabs on it. We hope to deliver lower latency as a community sooner. >> How long is one millisecond? (Mark laughs) >> Yeah, I'm largely focused on cloud infrastructure and I found here at the conference that, like, many many people are very much prepared to actually start taking more, you know, more POCs and more interest in cloud and the response in terms of all of this in Altus has been very encouraging. >> Great. Well, Jennifer, Mark, thank you so much for spending some time here on the Cube with us today. We're going to come by your booth and chat a little bit more later. It's some interesting stuff. And thank you all for watching the Cube today here at Spark Summit 2017, and thanks to Cloudera for bringing us these two experts. And thank you for watching. We'll see you again in just a few minutes with our next interview.

>> Narrator: Live from San Francisco, it's theCUBE, covering Spark Summit 2017. Brought to you by Databricks. >> Welcome back we're here at theCube at Spark Summit 2017. I'm David Goad here with George Gilbert, George. >> Good to be here. >> Thanks for hanging with us. Well here's the other man of the hour here. We just talked with Ali, the CEO at Databricks and now we have the Chief Architect and co-founder at Databricks, Reynold Xin. Reynold, how are you? >> I'm good. How are you doing? >> David: Awesome. Enjoying yourself here at the show? >> Absolutely, it's fantastic. It's the largest Summit. It's a lot interesting things, a lot of interesting people with who I meet. >> Well I know you're a really humble guy but I had to ask Ali what should I ask Reynold when he gets up here. Reynold is one of the biggest contributors to Spark. And you've been with us for a long time right? >> Yes, I've been contributing for Spark for about five or six years and that's probably the most number of commits to the project and lately more I'm working with other people to help design the roadmap for both Spark and Databricks with them. >> Well let's get started talking about some of the new developments that you want maybe our audience at theCUBE hasn't heard here in the keynote this morning. What are some of the most exciting new developments? >> So, I think in general if we look at Spark, there are three directions I would say we doubling down. One the first direction is the deep learning. Deep learning is extremely hot and it's very capable but as we alluded to earlier in a blog post, deep learning has reached sort of a mass produced point in which it shows tremendous potential but the tools are very difficult to use. And we are hoping to democratize deep learning and do what Spark did to big data, to deep learning with this new library called deep learning pipelines. What it does, it integrates different deep learning libraries directly in Spark and can actually expose models in sequel. So, even the business analysts are capable of leveraging that. So, that one area, deep learning. The second area is streaming. Streaming, again, I think that a lot of customers have aspirations to actually shorten the latency and increase the throughput in streaming. So, the structured streaming effort is going to be generally available and last month alone on Databricks platform, I think out customers processed three trillion records, last month alone using structured streaming. And we also have a new effort to actually push down the latency all the way to some millisecond range. So, you can really do blazingly fast streaming analytics. And last but not least is the SEQUEL Data Warehousing area, Data warehousing I think that it's a very mature area from the outset of big data point of view, but from a big data one it's still pretty new and there's a lot of use cases that's popping up there. And Spark with approaches like the CBO and also impact here in the database runtime with DBIO, we're actually substantially improving the performance and the capabilities of data warehousing futures. >> We're going to dig in to some of those technologies here in just a second with George. But have you heard anything here so far from anyone that's changed your mind maybe about what to focus on next? So, one thing I've heard from a few customers is actually visibility and debugability of the big data jobs. So many of them are fairly technical engineers and some of them are less sophisticated engineers and they have written jobs and sometimes the job runs slow. And so the performance engineer in me would think so how do I make the job run fast? The different way to actually solve that problem is how can we expose the right information so the customer can actually understand and figure it out themselves. This is why my job is slow and this how I can tweak it to make it faster. Rather than giving people the fish, you actually give them the tools to fish. >> If you can call that bugability. >> Reynold: Yeah, Debugability. >> Debugability. >> Reynold: And visibility, yeah. >> Alright, awesome, George. >> So, let's go back and unpack some of those kind of juicy areas that you identified, on deep learning you were able to distribute, if I understand things right, the predictions. You could put models out on a cluster but the really hard part, the compute intensive stuff, was training across a cluster. And so Deep Learning, 4J and I think Intel's BigDL, they were written for Spark to do that. But with all the excitement over some of the new frameworks, are they now at the point where they are as good citizens on Spark as they are on their native environments? >> Yeah so, this is a very interesting question, obviously a lot of other frameworks are becoming more and more popular, such as TensorFlow, MXNet, Theano, Keras and Office. What the Deep Learning Pipeline library does, is actually exposes all these single note Deep Learning tools as highly optimized for say even GPUs or CPUs, to be available as a estimator or like a module in a pipeline of the machine learning pipeline library in spark. So, now users can actually leverage Spark's capability to, for example, do hyper parameter churning. So, when you're building a machine learning model, it's fairly rare that you just run something once and you're good with it. Usually have to fiddle with a lot of the parameters. For example, you might run over a hundred experiments to actually figure out what is the best model I can get. This is where actually Spark really shines. When you combine Spark with some deep learning library be it BigDL or be it MXNet, be it TensorFlow, you could be using Spark to distribute that training and then do cross validation on it. So you can actually find the best model very quickly. And Spark takes care of all the job scheduling, all the tolerance properties and how do you read data in from different data sources. >> And without my dropping too much in the weeds, there was a version of that where Spark wouldn't take care of all the communications. It would maybe distribute the models and then do some of the averaging of what was done out on the cluster. Are you saying that all that now can be managed by Spark? >> In that library, Spark will be able to actually take care of picking the best model out of it. And there are different ways you an design how do you define the best. The best could be some average of some different models. The best could be just pick one out of this. The best could be maybe there's a tree of models that you classify it on. >> George: And that's a hyper parameter configuration choice? >> So that is actually building functionality in Sparks machine learning pipeline. And now what we're doing is now you can actually plug all those deep learning libraries directly into that as part of the pipeline to be used. Another maybe just to add, >> Yeah, yeah, >> Another really cool functionality of the deep learning pipeline is transfer learning. So as you said, deep learning takes a very long time, it's very computationally demanding. And it takes a lot of resources, expertise to train. But with transfer learning what we allow the customers to do is they can take an existing deep learning model as well train in a different domain and they we'd retrain it on a very small amount of data very quickly and they can adapt it to a different domain. That's how sort of the demo on the James Bond car. So there is a general image classifier that we train it on probably just a few thousand images. And now we can actually detect whether a car is James Bond's car or not. >> Oh, and the implications there are huge, which is you don't have to have huge training data sets for modifying a model of a similar situation. I want to, in the time we have, there's always been this debate about whether Sparks should manage state, whether it's database, key value store. Tell us how the thinking about that has evolved and then how the integration interfaces for achieving that have evolved. >> One of the, I would say, advantages of Spark is that it's unbiased and works with a variety of storage systems, be it Cassandra, be it Edgebase, be it HDFS, be is S3. There is a metadata management functionality in Spark which is the catalog of tables that customers can define. But the actual storage sits somewhere else. And I don't think that will change in the near future because we do see that the storage systems have matured significantly in the last few years and I just wrote blog post last week about the advantage of S3 over HDFS for example. The storage price is being driven down by almost a factor of 10X when you go to the cloud. I just don't think it makes sense at this point to be building storage systems for analytics. That said, I think there's a lot of building on top of existing storage system. There's actually a lot of opportunities for optimization on how you can leverage the specific properties of the underlying storage system to get to maximum performance. For example, how are you doing intelligent caching, how do you start thinking about building indexes actually against the data that's stored for scanned workloads. >> With Tungsten's, you take advantage of the latest hardware and where we get more memory intensive systems and now that the Catalyst Optimizer has a cost based optimizer or will be, and large memory. Can you change how you go about knowing what data you're managing in the underlying system and therefore, achieve a tremendous acceleration in performance? >> This is actually one area we invested in the DBIO module as part of Databricks Runtime, and what DBIO does, a lot of this are still in progress, but for example, we're adding some form of indexing capability to add to the system so we can quickly skip and prune out all the irrelevant data when the user is doing simple point look-ups. Or if the user is doing a scan heavy workload with some predicates. That actually has to do with how we think about the underlying data structure. The storage system is still the same storage system, like S3, but were adding actually indexing functionalities on top of it as part of DBIO. >> And so what would be the application profiles? Is it just for the analytic queries or can you do the point look-ups and updates in that sort of scenario too? >> So it's interesting you're talking about updates. Updates is another thing that we've got a lot of future requests on. We're actively thinking about how we will support update workload. Now, that said, I just want to emphasize for both use case of doing point look-ups and updates, we're still talking about in the context of analytic environment. So we would be talking about for example maybe bulk updates or low throughput updates rather than doing transactional updates in which every time you swipe a credit card, some record gets updated. That's probably more belongs on the transactional databases like Oracle or my SEQUEL even. >> What about when you think about people who are going to run, they started out with Spark on prem, they realize they're going to put much more of their resources in the cloud, but with IIOT, industrial IOT type applications they're going to have Spark maybe in a gateway server on the edge? What do you think that configuration looks like? >> Really interesting, it's kind of two questions maybe. The first is the hybrid on prem, cloud solution. Again, so one of the nice advantage of Spark is the couple of storage and compute. So when you want to move for example, workloads from one prem to the cloud, the one you care the most about is probably actually the data 'cause the compute, it doesn't really matter that much where you run it but data's the one that's hard to move. We do have customers that's leveraging Databricks in the cloud but actually reading data directly from on prem the reliance of the caching solution we have that minimize the data transfer over time. And is one route I would say it's pretty popular. Another on is, with Amazon you can literally give them just a show ball of functionality. You give them hard drive with trucks, the trucks will ship your data directly put in a three. With IOT, a common pattern we see is a lot of the edge devices, would be actually pushing the data directly into some some fire hose like Kinesis or Kafka or, I'm sure Google and Microsoft both have their own variance of that. And then you use Spark to directly subscribe to those topics and process them in real time with structured streaming. >> And so would Spark be down, let's say at the site level. if it's not on the device itself? >> It's a interesting thought and maybe one thing we should actually consider more in the future is how do we push Spark to the edges. Right now it's more of a centralized model in which the devices push data into Spark which is centralized somewhere. I've seen for example, I don't remember exact the use case but it has to do with some scientific experiment in the North Pole. And of course there you don't have a great uplink of all the data connecting transferring back to some national lab and rather they would do a smart parsing there and then ship the aggregated result back. There's another one but it's less common. >> Alright well just one minute now before the break so I'm going to give you a chance to address the Spark community. What's the next big technical challenge you hope people will work on for the benefit of everybody? >> In general Spark came along with two focuses. One is performance, the other one's ease of use. And I still think big data tools are too difficult to use. Deep learning tools, even harder. The barrier to entry is very high for office tools. I would say, we might have already addressed performance to a degree that I think it's actually pretty usable. The systems are fast enough. Now, we should work on actually make (mumbles) even easier to use. It's what also we focus a lot on at Databricks here. >> David: Democratizing access right? >> Absolutely. >> Alright well Reynold, I wish we could talk to you all day. This is great. We are out of time now. Want to appreciate you coming by theCUBE and sharing your insights and good luck with the rest of the show. >> Thank you very much David and George. >> Thank you all for watching here were at theCUBE at Sparks Summit 2017. Stay tuned, lots of other great guests coming up today. We'll see you in a few minutes.

(techno music) >> Narrator: Live from San Francisco, it's theCUBE, covering Spark Summit 2017. Brought to you by Databricks. >> You are watching the Spark Summit 2017 coverage by theCUBE. I'm your host David Goad, and joined with George Gilbert. How you doing George? >> Good to be here. >> And honored to introduce our next guest, the CTO from SnappyData, wow we were lucky to get this guy. >> Thanks for having me >> David: Jags Ramnarayan, Jags thanks for joining us. >> Thanks, thanks for having me. >> And for people who may not be familiar, maybe tell us what does SnappyData do? >> So SnappyData in a nutshell, is taking Spark, which is a computer engine, and in some sense augmenting the guts of Spark so that Spark truly becomes a hybrid database. A single data store that's capable of taking Spark streams, doing transactions, providing mutable state management in Spark, but most importantly being able to turn around, and run analytical queries on that state that is continuously merging. That's in a nutshell. Let me just say a few things, SnappyData itself is a startup that is a spun out, a spun out out of Pivotal. We've been out of Pivotal for roughly about a year, so the technology itself was to a great degree, incubated within Pivotal. It's a product called GemFire within VMware and Pivotal. So we took the guts of GemFire, which is an in-memory data base, designed for transactional low-latency, high confidence scenarios, and we are sort of fusing it, that's the key thing, fusing it into Spark, so that now Spark becomes significantly richer, as not just as a computer platform, but as a store. >> Great, and we know this is not your first Spark Summit, right? How many have you been to? Lost count? >> Boy, let's see, three, four now, Spark Summits, if I include the Spark Summit, this year, four to five. >> Great, so an active part of the community. What were you expecting to learn this year, and have you been surprised by anything? >> You know, it's always wonderful to see, I mean, every time I come to Spark, it's just a new set of innovations, right? I mean, when I first came to Spark, it was a mix of, let's talk about data frames, all of these, let's optimize my priorities. Today you come, I mean there is such a wide spectrum of amazing new things that are happening. It's just mind boggling. Right from AI techniques, structured streaming, and the real-time paradigm, and sort of this confluence that Databricks brings more to it. How can I create a confluence through a unified mechanism, where it is really brilliant, is what I think. >> Okay, well let's talk about how you're innovating at SnappyData. What are some of the applications or current projects you're working on? So number of things, I mean, GE is an investor in SnappyData. So we're trying to work with GE on the investor layer Dspace. We're working with large health care companies, also on their layer Dspace. So the part done with SnappyData is one that has a lot of high velocity streams of data emerging where the streams could be, for instance, Kafka streams driving Spark streams, but streams could also be operation databases. Your Postgres instance and your Cassandra database instance, and they're all generating continuous changes to data that's emerging in an operational world, can I suck that in and almost create a replica of that state that might be emerging in the SOQL operation environment, and still allow interactive analytics ASCIL for a number of concordant users on live data. Not cube data, not pre-aggregated data, but on live data itself, right? Being able to almost give you Google-like speeds to live data. >> George, we've heard people talking about this quite a bit. >> Yeah, so Jags, as you said upfront, Spark was conceived as sort of a general purpose, I guess, analytic compute engine, and adding DBMS to it, like sort of not bolting it on, but deeply integrating it, so that the core data structures now have DBMS properties, like transactionality, that must make a huge change in the scope of applications that are applicable. Can you desribe some of those for us? >> Yeah. The classic paradigm today that we find time and again as, the so-called smack stack, right? I mean lambda stack, now there's a smack stack. Which is really about Spark running on Mesos, but really using Spark streaming as an ingestion capability, and there is continuous state that is emerging that I want to write into Cassandra. So what we find very quickly is that the moment the state is emerging, I want to throw in a business intelligence tool on top and immediately do live dashboarding on that state that is continuously changing and emerging. So what we find is that the first part, which is the high speed drives, the ability to transform these data search, cleanse the data search, get the cleanse data into Cassandra, works really well. What is missing is this ability to say, well, how am I going to get insight? How can I ask you interesting, insightful questions, get responses immediately on that live data, right? And so the common problem there is the moment I have Cassandra working, let's say, with Spark, every time I run an analytical query, you only have two choices. One is use the parallel connector to pull in the data search from Cassandra, right, and now unfortunately, when you do analytics, you're working with large volumes. And every time I run even a simple query, all of a sudden I could be pulling in 10 gigabytes, 20 gigabytes of data into Spark to run the computation. Hundreds of seconds lost. Nothing like interactive, it's all about batch querying. So how can I turn around and say that if stuff changes in Cassandra, I can can have an immediate real-time reflection of that mutable state in Spark on which I can run queries rapidly. That's a very key aspect to us. >> So you were telling me earlier that you didn't see, necessarily, a need to replace entirely, the Cassandra in the smack stack, but to compliment it. >> Jags: That's right. >> Elaborate on that. >> So our focus, much like Spark, is all about in-memory, state management in-memory processing. And Cassandra, realistically, is really designed to say how can I scale the petabyte, right, for key value operations, semi-structured data, what have you. So we think there are a number of scenarios where you still want Cassandra to be your store, because in some sense a lot of these guys have already adapted Cassandra in a fairly big way. So you want to say, hey, leave your petabyte level wall in there, and you can essentially work with the real-time state, which could still be still many terabytes of state, essentially in main memory, that's going to work with specializing it. And we're also, I mean I can touch on this approximate query process and technology, which is other part, other key part here, to say hey, I can't really 1,000 cores, and 1,000 machines just so that you can do your job really well, so one of the techniques we are adopting, which even the Databricks guys stirred with Blink, essentially, it's an approximate query processing engine, we have our own essential approximate query processing engine, as an adjunct, essentially, to our store. What that essentially means is to say, can I take a billion records and synthesize something really, really small, using smart sampling techniques, sketching techniques, essentially statistical structures, that can be stored along with Spark and Spark memory itself, and fuse it with the Spark catalyst query engine. So that as you run your query and we can very smartly figure out, can I use the approximate data structures to answer the questions extremely quickly. Even when the data would be in petabyte volume, I have these data structures that just now taking, maybe gigabytes of storage only. So hopefully not getting too, too technical, so the Spark catalyst query optimizer, like an Oracle query optimizer, it knows about the data that it's going to query, only in your case, you're taking what catalyst knows about Spark, and extending it with what's stored in your native, also Spark native, data structures. >> That's right, exactly. So think about an optimizer always takes a query plan and says, here are all the possible plans you can execute, and here is cost estimate for these plans, we essentially inject more plans into that and hopefully, our plan is even more optimized than the plans that the Spark catalyst engine came up with. And Spark is beautiful because, the Catalyst engine is a very pluggable engine. So you can essentially augment that engine very easily. >> So you've been out in the marketplace, whether in alpha, beta, or now, production, for enough time so that the community is aware of what you've done. What are some of the areas that you're being pulled in that are, that people didn't associate Spark with? >> So more often, we land up in situations where they're looking at SAP HANA, as an example, maybe a Meme SQL, maybe just Postgres, and all of the sudden, there are these hybrid workloads, which is the Gartner term of HTAP, so there's a lot of HTAP use cases, where we get pulled into. So there's no Spark, but we get pulled into it because we just a hybrid database. That's what people look at us, essentially. >> Oh, so you pull Spark in because that's just part of your solution. >> Exactly, right. So think about Spark is not just data frames and rich API, but also it has a SQL interface, right. I can essentially execute, SQL, select SQL. Of course we augment that SQL so that now you can do what you expect from a database, which is an insert, an update, a delete, can I create a view, can I run a transaction? So all of a sudden, it's not just a Spark API but what we provide looks like a SQL database itself. >> Okay, interesting. So tell us, in the work with GE, they're among the first that have sort of educated the world that in that world there's so much data coming off devices, that we have to be intelligent about what we filter and send to cloud, we train models, potentially, up there, we run them closer to the edge, so that we get low latency analytics, but you were telling us earlier that there are alternatives, especially when you have such an intelligent database, working both at the edge and in the cloud. >> Right, so that's a great point. See what's happening with sort of a lot of these machine learning models is that these models are learned on historical data search. And quite often, especially if you look at predictive maintenance, those class of use cases, in industrial IRT, the parlance could evolve very rapidly, right? Maybe because of climate changes and let's say, for a windmill farm, there are few windmills that are breaking down so rapidly it's affecting everything else, in terms of the power generation. So being able to sort of order the model itself, incrementally and near real-time, is becoming more and more important. >> David: Wow. >> It's still a fairly academic research kind of area, but for instance, we are working very closely with the University of Michigan to sort of say, can we use some of these approximate techniques to incrementally also learn a model. Right, sort of incrementally augment a model, potential of the edge, or even inside the cloud, for instance. >> David: Wow. >> So if you're doing it at the edge, would you be updating the instance of the model associated with that locale and then would the model in the cloud be sort of like the master, and then that gets pushed down, until you have an instance and a master. >> That's right. See most typically what will happen is you have computed a model using a lot of historical data. You have typically supervised techniques to compute a model. And you take that model and inject it potentially into the edge, so that it can compute that model, which is the easy part, everybody does that. So you continue to do that, right, because you really want the data scientists to be pouring through those paradigms, looking and sort of tweaking those models. But for a certain number of models, even in the models injected in the edge, can I re-tweak that model in unsupervised way, is kind of the play, we're also kind of venturing into slowly, but that's all in the future. >> But if you're doing it unsupervised, do you need metrics that sort of flag, like what is the champion challenger, and figure out-- >> I should say that I mean, not all of these models can work in this very real-time manner. So, for instance, we've been looking at saying, can we reclassify NPC, the name place classifier, to essentially do incremental classification, or incrementally learning the model. Clustering approaches can actually be done in an unsupervised way in an incremental fashion. Things like that. There's a whole spectrum of algorithms that really need to be thought through for approximate algorithms to actually apply. So it's still an active research. >> Really great discussion, guys. We've just got about a minute to go, before the break, really great stuff. I don't want to interrupt you. But maybe switch real quick to business drivers. Maybe with SnappyData or with other peers you've talked to today. What business drivers do you think are going to affect the evolution of Spark the most? I mean, for us, as a small company, the single biggest challenge we have, it's like what one of you guys said, analysts, it's raining databases out there. And there's ability to constantly educate people how you can essentially realize a very next generation, like data pipeline, in a very simplified manner, is the challenge we are running into, right. I mean, I think the business model for us is primarily how many people are going to go and say, yes, batch related analytics is important, but incrementally, for competitive reasons, want to be playing that real-time analytics game lot more than before, right? So that's going to be big for us, and hopefully we can play a big part there, along with Spark and Databricks. >> Great, well we appreciate you coming on the show today, and sharing some of the interesting work that you're doing. George, thank you so much. and Jags, thank you so much for being on theCUBE. >> Thanks for having me on, I appreciate it. Thanks, George. And thank you all for tuning in. Once again, we have more to come, today and tomorrow, here at Spark Summit 2017, thanks for watching. (techno music)

>> Announcer: Live from San Francisco, it's theCUBE covering Spark Summit 2017, brought to you by Databricks. >> Welcome back to theCUBE, we're talking about data signs and engineering at scale, and we're having a great time, aren't we, George? >> We are! >> Well, we have another guest now we're going to talk to, I'm very pleased to introduce Matt Hunt, who's a technologist at Bloomberg, Matt, thanks for joining us! >> My pleasure. >> Alright, we're going to talk about a lot of exciting stuff here today, but I want to first start with, you're a long-time member of the Spark community, right? How many Spark Summits have you been to? >> Almost all of them, actually, it's quite amazing to see the 10th one, yes. >> And you're pretty actively involved with the user group on the east coast? >> Matt: Yeah, I run the New York users group. >> Alright, well, what's that all about? >> We have some 2,000 people in New York who are interested in finding out what goes on, and which technologies to use, and what are people working on. >> Alright, so hopefully, you saw the keynote this morning with Matei? >> Yes. >> Alright, any comments or reactions from the things that he talked about as priorities? >> Well, I've always loved the keynotes at the Spark Summits, because they announce something that you don't already know is coming in advance, at least for most people. The second Spark Summit actually had people gasping in the audience while they were demoing, a lot of senior people-- >> Well, the one millisecond today was kind of a wow one-- >> Exactly, and I would say that the one thing to pick out of the keynote that really stood out for me was the changes in improvements they've made for streaming, including potentially being able to do sub-millisecond times for some workloads. >> Well, maybe talk to us about some of the apps that you're building at Bloomberg, and then I want you to join in, George, and drill down some of the details. >> Sure. And Bloomberg is a large company with 4,000-plus developers, we've been working on apps for 30 years, so we actually have a wide range of applications, almost all of which are for news in the financial industry. We have a lot of homegrown technology that we've had to adapt over time, starting from when we built our own hardware, but there's some significant things that some of these technologies can potentially really help simplify over time. Some recent ones, I guess, trade anomaly detection would be one. How can you look for patterns of insider trading? How can you look for bad trades or attempts to spoof? There's a huge volume of trade data that comes in, that's a natural application, another one would be regulatory, there's a regulatory system called MiFID, or MiFID II, the regulations required for Europe, you have to be able to record every trade for seven years, provide daily reports, there's clearly a lot around that, and then I would also just say, our other internal databases have significant analytics that can be done, which is just kind of scraping the surface. >> These applications sound like they're oriented towards streaming solutions, and really low latency. Has that been a constraint on what you can build so far? >> I would definitely say that we have some things that are latency constrained, it tends to be not like high frequency trading, where you care about microseconds, but milliseconds are important, how long does it take to get an answer, but I would say equally important with latency is efficiency, and those two often wind up being coupled together, though not always. >> And so when you say coupled, is it because it's a trade-off, or 'cause you need both? >> Right, so it's a little bit of both, for a number of things, there's an upper threshold for the latency that we can accept. Certain architectural changes imply higher latencies, but often, greater efficiencies. Micro-batching often means that you can simplify and get greater throughput, but at a cost of higher latency. On the other hand, if you have a really large volume of things coming in, and your method of processing them isn't efficient enough, it gets too slow simply from that, and that's why it's not just one or the other. >> So in getting down to one millisecond or below, can they expose knobs where you can choose the trade-offs between efficiency and latency, and is that relevant for the apps that you're building? >> I mean, clearly if you can choose between micro-batching and not micro-batching, that's a knob that you can have, so that's one explicit one, but part of what's useful is, often when you sit down to try and determine what is the main cause of latency, you have to look at the full profile of a stack of what it's going through, and then you discover other inefficiencies that can be ironed out, and so it just makes it faster overall. I would say, a lot of what the Databricks guys in the Spark community have worked on over the years is connected to that, Project Tungsten and so on, well, all these things that make things much slower, much less efficient than they need to be, and we can close that gap a lot, I would say that from the very beginning. >> This brings up something that we were talking about earlier, which is, Matei has talked for a long time about wanting to take N 10 control of continuous apps, for simplicity and performance, and so there's this, we'll write with transactional consistency, so we're assuring the customer of exactly one's semantics when we write to a file system or database or something like that. But, Spark has never really done native storage, whereas Matei came here on the show earlier today and said, "Well, Databricks as a company "is going to have to do something in that area," and he talks specifically about databases, and he said, he implied that Apache Spark, separate from Databricks, would also have to do more in state management, I don't know if he was saying key value store, but how would that open up a broader class of apps, how would it make your life simpler as a developer? >> Right. Interesting and great question, this is kind of a subject that's near and dear to my own heart, I would say. So part of that, when you take a step back, is about some of the potential promise of what Spark could be, or what they've always wanted to be, which is a form of a universal computation engine. So there's a lot of value, if you can learn one small skillset, but it can work in a wide variety of use cases, whether it's streaming or at rest or analytics, and plug other things in. As always, there's a gap in any such system between theory and reality, and how much can you close that gap, but as for storage systems, this is something that, you and I have talked about this before, and I've written about it a fair amount too, Spark is historically an analytic system, so you have a bunch of data, and you can do analytics on it, but where's that data come from? Well, either it's streaming in, or you're reading from files, but most people need, essentially, an actual database. So what constitutes the universal system? You need file store, you need a distributive file store, you need a database with generally transactional semantics because the other forms are too hard for people to understand, you need analytics that are extensible, and you need a way to stream data in, and there's how close can you get to that, versus how much do you have to fit other parts that come together, very interesting question. >> So, so far, they've sort of outsourced that to DIY, do-it-yourself, but if they can find a sufficiently scalable relational database, they can do the sort of analytical queries, and they can sort of maintain state with transactions for some amount of the data flowing through. My impression is that, like Cassandra would be the, sort of the database that would handle all updates, and then some amount of those would be filtered through to a multi-model DBMS. When I say multi-model, I mean handles transactions and analytics. Knowing that you would have the option to drop that out, what applications would you undertake that you couldn't use right now, where the theme was, we're going to take big data apps into production, and then the competition that they show for streaming is of Kafka and Flink, so what does that do to that competitive balance? >> Right, so how many pieces do you need, and how well do they fit together is maybe the essence of that question, and people ask that all the time, and one of the limits has been, how mature is each piece, how efficient is it, and do they work together? And if you have to master 5,000 skills and 200 different products, that's a huge impediment to real-world usage. I think we're coalescing around a smaller set of options, so in the, Kafka, for example, has a lot of usage, and it seems to really be, the industry seems to be settling on that is what people are using for inbound streaming data, for ingest, I see that everywhere I go. But what happens when you move from Kafka into Spark, or Spark has to read from a database? This is partly a question of maturity. Relational databases are very hard to get right. The ones that we have have been under development for decades, right? I mean, DB2 has been around for a really long time with very, very smart people working on it, or Oracle, or lots of other databases. So at Bloomberg, we actually developed our own databases for relational databases that were designed for low latency and very high reliability, so we actually just opensourced that a few weeks ago, it's called ComDB2, and the reason we had to do that was the industry solutions at the time, when we started working on that, were inadequate to our needs, but we look at how long that took to develop for these other systems and think, that's really hard for someone else to get right, and so, if you need a database, which everyone does, how can you make that work better with Spark? And I think there're a number of very interesting developments that can make that a lot better, short of Spark becoming and integrating a database directly, although there's interesting possibilities with that too. How do you make them work well together, we could talk about for a while, 'cause that's a fascinating question. >> On that one topic, maybe the Databricks guys don't want to assume responsibility for the development, because then they're picking a winner, perhaps? Maybe, as Matei told us earlier, they can make the APIs easier to use for a database vendor to integrate, but like we've seen Splice Machine and SnappyData do the work, take it upon themselves to take data frames, the core data structure, in Spark, and give it transactional semantics. Does that sound promising? >> There're multiple avenues for potential success, and who can use which, in a way, depends on the audience. If you look at things like Cassandra and HBase, they're distributing key value stores that additional things are being built on, so they started as distributed, and they're moving towards more encompassing systems, versus relational databases, which generally started as single image on single machine, and are moving towards federation distribution, and there's been a lot with that with post grads, for example. One of the questions would be, is it just knobs, or why don't they work well together? And there're a number of reasons. One is, what can be pushed down, how much knowledge do you have to have to make that decision, and optimizing that, I think, is actually one of the really interesting things that could be done, just as we have database query optimizers, why not, can you determine the best way to execute down a chain? In order to do that well, there are two things that you need that haven't yet been widely adopted, but are coming. One is the very efficient copy of data between systems, and Apache Arrow, for example, is very, very interesting, and it's nearing the time when I think it's just going to explode, because it lets you connect these systems radically more efficiently in a standardized way, and that's one of the things that was missing, as soon as you hop from one system to another, all of a sudden, you have the semantic computational expense, that's a problem, we can fix that. The other is, the next level of integration requires, basically, exposing more hooks. In order to know, where should a query be executed and which operator should I push down, you need something that I think of as a meta-optimizer, and also, knowledge about the shape of the data, or statistics underlying, and ways to exchange that back and forth to be able to do it well. >> Wow, Matt, a lot of great questions there. We're coming up on a break, so we have to wrap things up, and I wanted to give you at least 30 seconds to maybe sum up what you'd like to see your user community, the Spark community, do over the next year. What are the top issues, things you'd love to see worked on? >> Right. It's an exciting time for Spark, because as time goes by, it gets more and more mature, and more real-world applications are viable. The hardest thing of all is to get, anywhere you in any organization's to get people working together, but the more people work together to enable these pieces, how do I efficiently work with databases, or have these better optimizations make streaming more mature, the more people can use it in practice, and that's why people develop software, is to actually tackle these real-world problems, so, I would love to see more of that. >> Can we all get along? (chuckling) Well, that's going to be the last word of this segue, Matt, thank you so much for coming on and spending some time with us here to share the story! >> My pleasure. >> Alright, thank you so much. Thank you George, and thank you all for watching this segment of theCUBE, please stay with us, as Spark Summit 2017 will be back in a few moments.

>> Announcer: Live from San Francisco it's the CUBE covering Spark Summit 2017 brought to you by Data Bricks. >> Welcome back to the CUBE, we're continuing to take about two people who are not just talking about things but doing things. We're happy to have, from Novetta, the Director of Predictive Analytics, Mr. Rob Lantz. Rob, welcome to the show. >> Thank you. >> And off to my right, George, how are you? >> Good. >> We've introduced you before. >> Yes. >> Well let's talk to the guest. Let's get right to it. I want to talk to you a little bit about what does Novetta do and then maybe what apps you're building using Spark. >> Sure, so Novetta is an advanced analytics company, we're medium sized and we develop custom hardware and software solutions for our customers who are looking to get insights out of their big data. Our primary offering is a hard entity resolution engine. We scale up to billions of records and we've done that for about 15 years. >> So you're in the business end of analytics, right? >> Yeah, I think so. >> Alright, so talk to us a little bit more about entity resolution, and that's all Spark right? This is your main priority? >> Yes, yes, indeed. Entity resolution is the science of taking multiple disparate data sets, traditional big data, and taking records from those and determining which of those are actually the same individual or company or address or location and which of those should be kept separate. We can aggregate those things together and build profiles and that enables a more robust picture of what's going on for an organization. >> Okay, and George? >> So what did you do... What was the solution looking like before Spark and how did it change once you adopted Spark? >> Sure, so with Spark, it enabled us to get a lot faster. Obviously those computations scaled a lot better. Before, we were having to write a lot of custom code to get those computations out across a grid. When we moved to Hadoop and then Spark, that made us, let's say able to scale those things and get it done overnight or in hours and not weeks. >> So when you say you had to do a lot of custom code to distribute across the cluster, does that include when you were working with MapReduce, or was this even before the Hadoop era? >> Oh it was before the Hadoop era and that predates my time so I won't be able to speak expertly about it, but to my understanding, it was a challenge for sure. >> Okay so this sounds like a service that your customers would then themselves build on. Maybe an ETL customer would figure out master data from a repository that is not as carefully curated as the data warehouse or similar applications. So who is your end customer and how do they build on your solution? >> Sure, so the end customer typically is an enterprise that has large volumes of data that deal in particular things. They collect, it could be customers, it could be passengers, it could be lots of different things. They want to be able to build profiles about those people or companies, like I said, or locations, any number of things can be considered an entity. The way they build upon it then is how they go about quantifying those profiles. We can help them do that, in fact, some of the work that I manage does that, but often times they do it themselves. They take the resolve data and that gets resolved nightly or even hourly. They build those profiles themselves for their own purpose. >> Then, to help us think about the application or the use case holistically, once they've built those profiles and essentially harmonized the data, what does that typically feed into? >> Oh gosh, any number of things really. Oh, shoot. We've got deployments in AWS in the cloud, we've got deployments, lots of deployments on premises obviously. That can go anywhere from relational databases to graph query language databases. Lots of different places from there for sure. >> Okay so, this actually sounds like everyone talks now about machine learning and forming every category of software. This sounds like you take the old style ETL, where master data was a value add layer on top, and that was, it took a fair amount of human judgment to do. Now, you're putting that service on top of ETL and you're largely automating it, probably with, I assume, some supervised guidance, supervised training. >> Yes, so we're getting into the machine learning space as far as entity extraction and resolution and recognition because more and more data is unstructured. But machine learning isn't necessarily a baked in part of that. Actually entity resolution is a prerequisite, I think, for quality machine learning. So if Rob Lantz is a customer, I want to be able to know what has Rob Lantz bought in the past from me. And maybe what is Rob Lantz talking about in social media? Well I need to know how to figure out who those people are and who's Rob Lantz and who's Robert Lantz is a completely different person, I don't want to collapse those two things together. Then I would build machine learning on top of that to say, right, now what's his behavior going to be in the future. But once I have that robust profile built up, I can derive a lot more interesting features with which to apply the machine learning. >> Okay, so you are a Data Bricks customer and there's also a burgeoning partnership. >> Rob: Yeah, I think that's true. >> So talk to us a little bit about what are some of the frustrations you had before adopting Data Bricks and maybe why you choose it. >> Yeah, sure. So the frustrations primarily with a traditional Hadoop environment involved having to go from one customer site to another customer site with an incredibly complex technology stack and then do a lot of the cluster management for those customers even after they'd already set it up because of all the inner workings of Hadoop and that ecosystem. Getting our Spark application installed there, we had to penetrate layers and layers of configuration in order to tune it appropriately to get the performance we needed. >> David: Okay, and were you at the keynote this morning? >> I was not, actually. >> Okay, I'm not going to ask you about that then. >> Ah. >> But I am going to ask you a little bit about your wishlist. You've been talking to people maybe in the hallway here, you just got here today but, what do you wish the community would do or develop, what would you like to learn while you're here? >> Learning while I'm here, I've already picked up a lot. So much going on and it's such a fast paced environment, it's really exciting. I think if I had a wishlist, I would want a more robust ML Lib, machine learning library. All the things that you can get on traditional, in scientific computing stacks moved onto a Spark ML Lib for easier access. On a cluster would be great. >> I thought several years ago ML Lib took over from Mahoot as the most active open source community for adding, really, I thought, scale out machine learning algorithms. If it doesn't have it all now, or maybe all is something you never reach, kind of like Red Queen effect, you know? >> Rob: For sure, for sure. >> What else is attracting these scale out implementations of the machine learning algorithms? >> Um? >> In other words, what are the platforms? If it's not Spark then... >> I don't think it exists frankly, unless you write your own. I think that would be the way to go. That's the way to go about it now. I think what organizations are having to do with machine learning in a distributed environment is just go with good enough, right. Whereas maybe some of the ensemble methods that are, actually aren't even really cutting edge necessarily, but you can really do a lot of tuning on those things, doing that tuning distributed at scale would be really powerful. I read somewhere, and I'm not going to be able to quote exactly where it was but, actually throwing more data at a problem is more valuable than tuning a perfect algorithm frankly. If we could combine the two, I think that would be really powerful. That is, finding the right algorithm and throwing all the data at it would get you a really solid model that would pick up on that signal that underlies any of these phenomena. >> David: Okay well, go ahead George. >> I was going to ask, I think that goes back to, I don't know if it was Google Paper, or one of the Google search quality guys who's a luminary in the machine learning space says, "data always trumps algorithms." >> I believe that's true and that's true in my experience certainly. >> Once you had this machine learning and once you've perhaps simplified the multi-vendor stack, then what is your solution start looking like in terms of broadening its appeal, because of the lower TCO. And then, perhaps embracing more use cases. >> I don't know that it necessarily embraces more use cases because entity resolution applies so broadly already, but what I would say is will give us more time to focus on improving the ER itself. That's I think going to be a really, really powerful improvement we can make to Novetta entity analytics as it stands right now. That's going to go into, we alluded to before, the machine learning as part of the entity resolution. Entity extraction, automated entity extraction from unstructured information and not just unstructured text but unstructured images and video. Could be a really powerful thing. Taking in stuff that isn't tagged and pulling the entities out of that automatically without actually having to have a human in the loop. Pulling every name out, every phone number out, every address out. Go ahead, sorry. >> This goes back to a couple conversations we've had today where people say data trumps algorithms, even if they don't say it explicitly, so the cloud vendors who are sitting on billions of photos, many of which might have house street addresses and things like that, or faces, how do you make better... How do you extract better tuning for your algorithms from data sets that I assume are smaller than the cloud vendors? >> They're pretty big. We employ data engineers that are very experienced at tagging that stuff manually. What I would envision would happen is we would apply somebody for a week or two weeks, to go in and tag the data as appropriate. In fact, we have products that go in and do concept tagging already across multiple languages. That's going to be the subject of my talk tomorrow as a matter of fact. But we can tag things manually or with machine assistance and then use that as a training set to go apply to the much larger data set. I'm not so worried about the scale of the data, we already have a lot, a lot of data. I think it's going to be getting that proof set that's already tagged. >> So what you're saying is, it actually sounds kind of important. That actually almost ties into what we hear about Facebook training their messenger bot where we can't do it purely just on training data so we're going to take some data that needs semi-supervision, and that becomes our new labeled set, our new training data. Then we can run it against this broad, unwashed mass of training data. Is that the strategy? >> Certainly we would get there. We would want to get there and that's the beauty of what Data Bricks promises, is that ability to save a lot of the time that we would spend doing the nug work on cluster management to innovate in that way and we're really excited about that. >> Alright, we've got just a minute to go here before the break, so I wanted to ask you maybe, the wish list question, I've been asking everybody today, what do you wish you had? Whether it's in entity resolution or some other area in the next couple of years for Novetta, what's on your list? >> Well I think that would be the more robust machine learning library, all in Spark, kind of native, so we wouldn't have to deploy that ourselves. Then, I think everything else is there, frankly. We are very excited about the platform and the stack that comes with it. >> Well that's a great ending right there, George do you have any other questions you want to ask? Alright, we're just wrapping up here. Thank you so much, we appreciate you being on the show Rob, and we'll see you out there in the Expo. >> I appreciate it, thank you. >> Alright, thanks so much. >> George: It's good to meet you. >> Thanks. >> Alright, you are watching the CUBE here at Spark Summit 2017, stay tuned, we'll be back with our next guest.

>> Announcer: Live from San Francisco, it's theCUBE covering Sparks Summit 2017 brought to you by Databricks. >> You are watching theCUBE at Sparks Summit 2017. We continue our coverage here talking with developers, partners, customers, all things Spark, and today we're honored now to have our next guest Dr. Jisheng Wang who's the Senior Director of Data Science at the CTO Office at Hewlett Packard Enterprise. Dr. Wang, welcome to the show. >> Yeah, thanks for having me here. >> All right and also to my right we have Mr. Jim Kobielus who's the Lead Analyst for Data Science at Wikibon. Welcome, Jim. >> Great to be here like always. >> Well let's jump into it. At first I want to ask about your background a little bit. We were talking about the organization, maybe you could do a better job (laughs) of telling me where you came from and you just recently joined HPE. >> Yes. I actually recently joined HPE earlier this year through the Niara acquisition, and now I'm the Senior Director of Data Science in the CTO Office of Aruba. Actually, Aruba you probably know like two years back, HP acquired Aruba as a wireless networking company, and now Aruba takes charge of the whole enterprise networking business in HP which is about over three billion annual revenue every year now. >> Host: That's not confusing at all. I can follow you (laughs). >> Yes, okay. >> Well all I know is you're doing some exciting stuff with Spark, so maybe tell us about this new solution that you're developing. >> Yes, actually my most experience of Spark now goes back to the Niara time, so Niara was a three and a half year old startup that invented, reinvented the enterprise security using big data and data science. So what is the problem we solved, we tried to solve in Niara is called a UEBA, user and entity behavioral analytics. So I'll just try to be very brief here. Most of the transitional security solutions focus on detecting attackers from outside, but what if the origin of the attacker is inside the enterprise, say Snowden, what can you do? So you probably heard of many cases today employees leaving the company by stealing lots of the company's IP and sensitive data. So UEBA is a new solution try to monitor the behavioral change of the enterprise users to detect both this kind of malicious insider and also the compromised user. >> Host: Behavioral analytics. >> Yes, so it sounds like it's a native analytics which we run like a product. >> Yeah and Jim you've done a lot of work in the industry on this, so any questions you might have for him around UEBA? >> Yeah, give us a sense for how you're incorporating streaming analytics and machine learning into that UEBA solution and then where Spark fits into the overall approach that you take? >> Right, okay. So actually when we started three and a half years back, the first version when we developed the first version of the data pipeline, we used a mix of Hadoop, YARN, Spark, even Apache Storm for different kind of stream and batch analytics work. But soon after with increased maturity and also the momentum from this open source Apache Spark community, we migrated all our stream and batch, you know the ETL and data analytics work into Spark. And it's not just Spark. It's Spark, Spark streaming, MLE, the whole ecosystem of that. So there are at least a couple advantages we have experienced through this kind of a transition. The first thing which really helped us is the simplification of the infrastructure and also the reduction of the DevOps efforts there. >> So simplification around Spark, the whole stack of Spark that you mentioned. >> Yes. >> Okay. >> So for the Niara solution originally, we supported, even here today, we supported both the on-premise and the cloud deployment. For the cloud we also supported the public cloud like AWS, Microsoft Azure, and also Privia Cloud. So you can understand with, if we have to maintain a stack of different like open source tools over this kind of many different deployments, the overhead of doing the DevOps work to monitor, alarming, debugging this kind of infrastructure over different deployments is very hard. So Spark provides us some unified platform. We can integrate the streaming, you know batch, real-time, near real-time, or even longterm batch job all together. So that heavily reduced both the expertise and also the effort required for the DevOps. This is one of the biggest advantages we experienced, and certainly we also experienced something like the scalability, performance, and also the convenience for developers to develop a new applications, all of this, from Spark. >> So are you using the Spark structured streaming runtime inside of your application? Is that true? >> We actually use Spark in the steaming processing when the data, so like in the UEBS solutions, the first thing is collecting a lot of the data, different account data source, network data, cloud application data. So when the data comes in, the first thing is streaming job for the ETL, to process the data. Then after that, we actually also develop the some, like different frequency like one minute, 10 minute, one hour, one day of this analytics job on top of that. And even recently we have started some early adoption of the deep learning into this, how to use deep learning to monitor the user behavior change over time, especially after user gives a notice what user, is user going to access like most servers or download some of the sensitive data? So all of this requires very complex analytics infrastructure. >> Now there were some announcements today here at Spark Summit by Databricks of adding deep learning support to their core Spark code base. What are your thoughts about the deep learning pipelines, API, that they announced this morning? It's new news, I'll understand if you don't, haven't digested it totally, but you probably have some good thoughts on the topic. >> Yes, actually this is also news for me, so I can just speak from my current experience. How to integrate deep learning into Spark actually was a big challenge so far for us because what we used so far, the deep learning piece, we used TensorFlow. And certainly most of our other stream and data massaging or ETL work is done by Spark. So in this case, there are a couple ways to manage this, too. One is to set up two separate resource pool, one for Spark, the other one for TensorFlow, but in our deployment there is some very small on-premise department which has only like four node or five node cluster. It's not efficient to split resource in that way. So we actually also looking for some closer integration between deep learning and Spark. So one thing we looked before is called the TensorFlow on Spark which was open source a couple months ago by Yahoo. >> Right. >> So maybe this is certainly more exciting news for the Spark team to develop this native integration. >> Jim: Very good. >> Okay and we talked about the UEBA solution, but let's go back to a little broader HPE perspective. You have this concept called the intelligent edge, what's that all about? >> So that's a very cool name. Actually come a little bit back. I come from the enterprise background, and enterprise applications have some, actually a lag behind than consumer applications in terms of the adoption of the new data science technology. So there are some native challenges for that. For example, collecting and storing large amount of this enterprise sensitive data is a huge concern, especially in European countries. Also for the similar reason how to collect, normally weigh developer enterprise applications. You're lack of some good quantity and quality of the trending data. So this is some native challenges when you develop enterprise applications, but even despite of this, HPE and Aruba recently made several acquisitions of analytics companies to accelerate the adoption of analytics into different product line. Actually that intelligent age comes from this IOT, which is internet of things, is expected to be the fastest growing market in the next few years here. >> So are you going to be integrating the UEBA behavioral analytics and Spark capability into your IOT portfolio at HP? Is that a strategy or direction for you? >> Yes. Yes, for the big picture that certainly is. So you can think, I think some of the Gartner Report expected the number of the IOT devices is going to grow over 20 billion by 2020. Since all of this IOT devices are connected to either intranet or internet, either through wire or wireless, so as a networking company, we have the advantage of collecting data and even take some actions at the first of place. So the idea of this intelligent age is we want to turn each of these IOT devices, the small IOT devices like IP camera, like those motion detection, all of these small devices as opposed to the distributed sensor for the data collection and also some inline actor to do some real-time or even close to real-time decisions. For example, the behavior anomaly detection is a very good example here. If IOT devices is compromised, if the IP camera has been compromised, then use that to steal your internal data. We should detect and stop that at the first place. >> Can you tell me about the challenges of putting deep learning algorithms natively on resource constrained endpoints in the IOT? That must be really challenging to get them to perform well considering that there may be just a little bit of memory or flash capacity or whatever on the endpoints. Any thoughts about how that can be done effectively and efficiently? >> Very good question >> And at low cost. >> Yes, very good question. So there are two aspects into this. First is this global training of the intelligence which is not going to be done on each of the device. In that case, each of the device is more like the sensor for the data collection. So we are going to build a, collect the data sent to the cloud, or build all of this giant pool, like computing resource to trend the classifier, to trend the model, but when we trend the model, we are going to ship the model, so the inference and the detection of the model of those behavioral anomaly really happen on the endpoint. >> Do the training centrally and then push the trained algorithms down to the edge devices. >> Yes. But even like, the second as well even like you said, some of the device like say people try to put those small chips in the spoon, in the case of, in hospital to make it like more intelligent, you cannot put even just the detection piece there. So we also looking to some new technology. I know like Caffe recently announced, released some of the lightweight deep learning models. Also there's some, your probably know, there's some of the improvement from the chip industry. >> Jim: Yes. >> How to optimize the chip design for this kind of more analytics driven task there. So we are all looking to this different areas now. >> We have just a couple minutes left, and Jim you get one last question after this, but I got to ask you, what's on your wishlist? What do you wish you could learn or maybe what did you come to Spark Summit hoping to take away? >> I've always treated myself as a technical developer. One thing I am very excited these days is the emerging of the new technology, like a Spark, like TensorFlow, like Caffe, even Big-Deal which was announced this morning. So this is something like the first go, when I come to this big advanced industry events, I want to learn the new technology. And the second thing is mostly to share our experience and also about adopting of this new technology and also learn from other colleagues from different industries, how people change life, disrupt the old industry by taking advantage of the new technologies here. >> The community's growing fast. I'm sure you're going to receive what you're looking for. And Jim, final question? >> Yeah, I heard you mention DevOps and Spark in same context, and that's a huge theme we're seeing, more DevOps is being wrapped around the lifecycle of development and training and deployment of machine learning models. If you could have your ideal DevOps tool for Spark developers, what would it look like? What would it do in a nutshell? >> Actually it's still, I just share my personal experience. In Niara, we actually developed a lot of the in-house DevOps tools like for example, when you run a lot of different Spark jobs, stream, batch, like one minute batch verus one day batch job, how do you monitor the status of those workflows? How do you know when the data stop coming? How do you know when the workflow failed? Then even how, monitor is a big thing and then alarming when you have something failure or something wrong, how do you alarm it, and also the debug is another big challenge. So I certainly see the growing effort from both Databricks and the community on different aspects of that. >> Jim: Very good. >> All right, so I'm going to ask you for kind of a soundbite summary. I'm going to put you on the spot here, you're in an elevator and I want you to answer this one question. Spark has enabled me to do blank better than ever before. >> Certainly, certainly. I think as I explained before, it helped a lot from both the developer, even the start-up try to disrupt some industry. It helps a lot, and I'm really excited to see this deep learning integration, all different road map report, you know, down the road. I think they're on the right track. >> All right. Dr. Wang, thank you so much for spending some time with us. We appreciate it and go enjoy the rest of your day. >> Yeah, thanks for being here. >> And thank you for watching the Cube. We're here at Spark Summit 2017. We'll be back after the break with another guest. (easygoing electronic music)

(lively music) >> [Announcer] Live from Boston, Massachusetts, this is the Cube, covering Sparks Summit East 2017. Brought to you by Databricks. Now, here are your hosts, Dave Vellante and George Gilbert. >> Welcome back to Sparks Summit in Boston, everybody. This is the Cube, the worldwide leader in live tech coverage. We've been here two days, wall-to-wall coverage of Sparks Summit. George Gilbert, my cohost this week, and I are going to review part two of the Wikibon Big Data Forecast. Now, it's very preliminary. We're only going to show you a small subset of what we're doing here. And so, well, let me just set it up. So, these are preliminary estimates, and we're going to look at different ways to triangulate the market. So, at Wikibon, what we try to do is focus on disruptive markets, and try to forecast those over the long term. What we try to do is identify where the traditional market research estimates really, we feel, might be missing some of the big trends. So, we're trying to figure out, what's the impact, for example, of real time. And, what's the impact of this new workload that we've been talking about around continuous streaming. So, we're beginning to put together ways to triangulate that, and we're going to show you, give you a glimpse today of what we're doing. So, if you bring up the first slide, we showed this yesterday in part one. This is our last year's big data forecast. And, what we're going to do today, is we're going to focus in on that line, that S-curve. That really represents the real time component of the market. The Spark would be in there. The Streaming analytics would be in there. Add some color to that, George, if you would. >> [George] Okay, for 60 years, since the dawn of computing, we have two ways of interacting with computers. You put your punch cards in, or whatever else and you come back and you get your answer later. That's batch. Then, starting in the early 60's, we had interactive, where you're at a terminal. And then, the big revolution in the 80's was you had a PC, but you still were either interactive either with terminal or batch, typically for reporting and things like that. What's happening is the rise of a new interaction mode. Which is continuous processing. Streaming is one way of looking at it but it might be more effective to call it continuous processing because you're not going to get rid of batch or interactive but your apps are going to have a little of each. So, what we're trying to do, since this is early, early in its life cycle, we're going to try and look at that streaming component from a couple of different angles. >> Okay, as I say, that's represented by this Ogive curve, or the S-curve. On the next slide, we're at the beginning when you think about these continuous workloads. We're at the early part of that S-curve, and of course, most of you or many of you know how the S-curve works. It's slow, slow, slow. For a lot of effort, you don't get much in return. Then you hit the steep part of that S-curve. And that's really when things start to take off. So, the challenge is, things are complex right now. That's really what this slide shows. And Spark is designed, really, to reduce some of that complexity. We've heard a lot about that, but take us through this. Look at this data flow from ingest, to explore, to process, to serve. We talked a lot about that yesterday, but this underscores the complexity in the marketplace. >> [George] Right, and while we're just looking mostly at numbers today, the point of the forecast is to estimate when the barriers, representing complexities, start to fall. And then, when we can put all these pieces together, in just explore, process, serve. When that becomes an end-to-end pipeline. When you can start taking the data in on one end, get a scientist to turn it into a model, inject it into an application, and that process becomes automated. That's when it's mature enough for the knee in the curve to start. >> And that's when we think the market's going to explode. But now so, how do you bound this. Okay, when we do forecasts, we always try to bound things. Because if they're not bounded, then you get no foundation. So, if you look at the next slide, we're trying to get a sense of real-time analytics. How big can it actually get? That's what this slide is really trying to-- >> [George] So this one was one firm's take on real-time analytics, where by 2027, they see it peaking just under-- >> [Dave] When you say one firm, you mean somebody from the technology district? >> [George] Publicly available data. And we take it as as a, since they didn't have a lot of assumptions published, we took it as, okay one data point. And then, we're going to come at it with some bottoms-up end top-down data points, and compare. >> [Dave] Okay, so the next slide we want to drill into the DBMS market and when you think about DBMS, you think about the traditional RDBMS and what we know, or the Oracle, SQL Server, IBMDB2's, etc. And then, you have this emergent NewSQL, and noSQL entrance, which are, obviously, we talked today to a number of folks. The number of suppliers is exploding. The revenue's still relatively small. Certainly small relative to the RDBMS marketplace. But, take us through what your expectations is here, and what some of the assumptions are behind this. >> [George] Okay, so the first thing to understand is the DBMS market, overall, is about $40 billion of which 30 billion goes to online transaction processing supporting real operational apps. 10 billion goes to Orlap or business intelligence type stuff. The Orlap one is shrinking materially. The online transaction processing one, new sales is shrinking materially but there's a huge maintenance stream. >> [Dave] Yeah which companies like Oracle and IBM and Microsoft are living off of that trying to fund new development. >> We modeled that declining gently and beginning to accelerate more going out into the latter years of the tenure period. >> What's driving that decline? Obviously, you've got the big sucking sound of a dup in part, is driving that. But really, increasingly it's people shifting their resources to some of these new emergent applications and workloads and new types of databases to support them right? But these are still, those new databases, you can see here, the NewSQL and noSQL still, relatively, small. A lot of it's open source. But then it starts to take off. What's your assumption there? >> So here, what's going on is, if you look at dollars today, it's, actually, interesting. If you take the noSQL databases, you take DynamoDB, you take Cassandra, Hadoop, HBase, Couchbase, Mongo, Kudu and you add all those up, it's about, with DynamoDB, it's, probably, about 1.55 billion out of a $40 billion market today. >> [Dave] Okay but it's starting to get meaningful. We were approaching two billion. >> But where it's meaningful is the unit share. If that were translated into Oracle pricing. The market would be much, much bigger. So the point it. >> Ten X? >> At least, at least. >> Okay, so in terms of work being done. If there's a measure of work being done. >> [George] We're looking at dollars here. >> Operations per second or etcetera, it would be enormous. >> Yes, but that's reflective of the fact that the data volumes are exploding but the prices are dropping precipitously. >> So do you have a metric to demonstrate that. We're, obviously, not going to show it today but. >> [George] Yes. >> Okay great, so-- >> On the business intelligence side, without naming names, the data warehouse appliance vendors are charging anywhere from 25,000 per terabyte up to, when you include running costs, as high as 100,000 a terabyte. That their customers are estimating. That's not the selling cost but that's the cost of ownership per terabyte. Whereas, if you look at, let's say Hadoop, which is comparable for the off loading some of the data warehouse work loads. That's down to the 5K per terabyte range. >> Okay great, so you expect that these platforms will have a bigger and bigger impact? What's your pricing assumption? Is prices going to go up or is it just volume's going to go through the roof? >> I'm, actually, expecting pricing. It's difficult because we're going to add more and more functionality. Volumes go up and if you add sufficient functionality, you can maintain pricing. But as volumes go up, typically, prices go down. So it's a matter of how much do these noSQL and NewSQL databases add in terms of functionality and I distinguish between them because NewSQL databases are scaled out version of Oracle or Teradata but they are based on the more open source pricing model. >> Okay and NoSQL, don't forget, stands for not only SQL, not not SQL. >> If you look at the slides, big existing markets never fall off a cliff when they're in the climb. They just slowly fade. And, eventually, that accelerates. But what's interesting here is, the data volumes could explode but the revenue associated with the NoSQL which is the dark gray and the NewSQL which is the blue. Those don't explode. You could take, what's the DBMS cost of supporting YouTube? It would be in the many, many, many billions of dollars. It would support 1/2 of an Oracle itself probably. But it's all open source there so. >> Right, so that's minimizing the opportunity is what you're saying? >> Right. >> You can see the database market is flat, certainly flattish and even declining but you do expect some growth in the out years as part of that evasion, that volume, presumably-- >> And that's the next slide which is where we've seen that growth come from. >> Okay so let's talk about that. So the next slide, again, I should have set this up better. The X-axis year is worldwide dollars and the horizontal axis is time. And we're talking here about these continuous application work loads. This new work load that you talked about earlier. So take us through the three. >> [George] There's three types of workloads that, in large part, are going to be driving most of this revenue. Now, these aren't completely, they are completely comparable to the DBMS market because some of these don't use traditional databases. Or if they do, they're Torry databases and I'll explain that. >> [Dave] Sure but if I look at the IoT Edge, the Cloud and the micro services and streaming, that's a tail wind to the database forecast in the previous slide, is that right? >> [George] It's, actually, interesting but the application and infrastructure telemetry, this is what Splunk pioneered. Which is all the torrents of data coming out of your data center and your applications and you're trying to manage what's going on. That is a database application. And we know Splunk, for 2016, was 400 million. In software revenue Hadoop was 750 million. And the various other management vendors, New Relic, AppDynamics, start ups and 5% of Azure and AWS revenue. If you add all that up, it comes out to $1.7 billion for 2016. And so, we can put a growth rate on that. And we talked to several vendors to say, okay, how much will that work load be compared to IoT Edge Cloud. And the IoT Edge Cloud is the smart devices at the Edge and the analytics are in the fog but not counting the database revenue up in the Cloud. So it's everything surrounding the Cloud. And that, actually, if you look out five years, that's, maybe, 20% larger than the app and infrastructure telemetry but growing much, much faster. Then the third one where you were talking about was this a tail wind to the database. Micro server systems streaming are very different ways of building applications from what we do now. Now, people build their logic for the application and everyone then, stores their data in this centralized external database. In micro services, you build a little piece of the app and whatever data you need, you store within that little piece of the app. And so the database requirements are, rather, primitive. And so that piece will not drive a lot of database revenue. >> So if you could go back to the previous slide, Patrick. What's driving database growth in the out years? Why wouldn't database continue to get eaten away and decline? >> [George] In broad terms, the overall database market, it staying flat. Because as prices collapse but the data volumes go up. >> [Dave] But there's an assumption in here that the NoSQL space, actually, grows in the out years. What's driving that growth? >> [George] Both the NoSQL and the NewSQL. The NoSQL, probably, is best serving capturing the IoT data because you don't need lots of fancy query capabilities for concurrency. >> [Dave] So it is a tail wind in a sense in that-- >> [George] IoT but that's different. >> [Dave] Yeah sure but you've got the overall market growing. And that's because the new stuff, NewSQL and NoSQL is growing faster than the decline of the old stuff. And it's not in the 2020 to 2022 time frame. It's not enough to offset that decline. And then they have it start growing again. You're saying that's going to be driven by IoT and other Edge use cases? >> Yes, IoT Edge and the NewSQL, actually, is where when they mature, you start to substitute them for the traditional operational apps. For people who want to write database apps not who want to write micro service based apps. >> Okay, alright good. Thank you, George, for setting it up for us. Now, we're going to be at Big Data SV in mid March? Is that right? Middle of March. And George is going to be releasing the actual final forecast there. We do it every year. We use Spark Summit to look at our preliminary numbers, some of the Spark related forecasts like continuous work loads. And then we harden those forecasts going into Big Data SV. We publish our big data report like we've done for the past, five, six, seven years. So check us out at Big Data SV. We do that in conjunction with the Strada events. So we'll be there again this year at the Fairmont Hotel. We got a bunch of stuff going on all week there. Some really good programs going on. So check out siliconangle.tv for all that action. Check out Wikibon.com. Look for new research coming out. You're going to be publishing this quarter, correct? And of course, check out siliconangle.com for all the news. And, really, we appreciate everybody watching. George, been a pleasure co-hosting with you. As always, really enjoyable. >> Alright, thanks Dave. >> Alright, to that's a rap from Sparks. We're going to try to get out of here, hit the snow storm and work our way home. Thanks everybody for watching. A great job everyone here. Seth, Ava, Patrick and Alex. And thanks to our audience. This is the Cube. We're out, see you next time. (lively music)

>> Narrator: Live from Boston, Massachusetts, this is theCUBE, covering Spark Summit East 2017. Brought to you by Databricks. Now, here are your hosts Dave Vellante and George Gilbert. >> Welcome back to Boston, everybody, this is theCUBE, the leader in live tech coverage. We're here in Boston, in snowy Boston. This is Spark Summit. Spark Summit does a East Coast version, they do a West Coast version, they've got one in Europe this year. theCUBE has been a partner with Databricks as the live broadcast partner. Our friend Bill Peterson is here. He's the head of partner marketing at MapR. Bill, good to see you again. >> Thank you, thanks for having me. >> So how's the show going for you? >> It's great. >> Give us the vibe. We're kind of windin' down day two. >> It is. The show's been great, we've got a lot of traffic coming by, a lot of deep technical questions which is-- >> Dave: Hardcore at the show-- >> It is, it is. I spend a lot of time there smiling and going, "Yeah, talk to him." (laughs) But it's great. We're getting those deep technical questions and it's great. We actually just got one on Lustre, which I had to think for a minute, oh, HPC. It was like way back in there. >> Dave: You know, Cray's on the floor. >> Oh, yeah that's true. But a lot of our customers as well. UnitedHealth Group, Wells Fargo, AMEX coming by. Which is great to see them and talk to them, but also they've got some deep technical questions for us. So it's moving the needle with existing customers but also new business, which is great. >> So I got to ask a basic question. What is MapR? MapR started in the early days of Hadoop distro, vendor, one of the big three. When somebody says to you what is MapR, what do you say? My answer today is MapR is an enterprise software company that delivers a converged data platform. That converged data platform consists of a file system, a NoSQL database, a Hadoop distribution, a Spark distribution, and a set of data management tools. And as a customer of MapR, you get all of those. You can turn 'em all on if you'd like. You can just turn on the file system, for example, if you wanted to just use the file system for storage. But the enterprise software piece of that is all the hardening we do behind the scenes on things like snapshots, mirroring, data governance, multi-tenancy, ease of use performance, all of that baked in to the solution, or the platform as we're calling it now. So as you're kind of alluding to, a year ago now we kind of got out of that business of saying okay, lead 100% with Hadoop and then while we have your attention, or if we don't, hey wait, we got all this other stuff in the basket we want to show you, we went the platform play and said we're going to include everything and it's all there and then the baseline underneath is the hardening of it, the file system, the database, and the streaming product, actually, which I didn't mention, which is kind of the core, and everything plays off of there. And that honestly has been really well-received. And it just, I feel, makes it so much easier because-- It happened here, we get the question, okay, how are you different from Cloudera or Hortonworks? And some of it here, given the nature of the attendees, is very technical, but there's been a couple of business users that I've talked to. And when I talk about us as an enterprise software company delivering a plethora of solutions versus just Hadoop, you can see the light going on sometimes in people's eyes. And I got it today, earlier, "I had no idea you had a file system," which, to me, just drives me insane because the file system is pretty cool, right? >> Well you guys are early on in investing in that file system and recovery capabilities and all the-- >> Two years in stealth writing it. >> Nasty, gnarly, hard stuff that was kind of poo-pooed early on. >> Yeah, yeah. MapR was never patient about waiting for the open source community to just figure it out and catch up. You always just said all right, we're going to solve this problem and go sell. >> And I'm glad you said that. I want to be clear. We're not giving up on open source or anything, right? Open source is still a big piece. 50% of our engineers' time is working on open source projects. That's still super important to us. And then back in November-ish last year we announced the MapR Ecosystem Packs, which is our effort to help our customers that are using open source components to stay current. 'Cause that's a pain in the butt. So this is a set of packages that have a whole bunch of components. We lead with Spark and Drill, and that was by customer request, that they were having a hard time keeping current with Spark and Drill. So the packs allow them to come up to current level within the converged data platform for all of their open source components. And that's something we're going to do at dot Level, so I think we're at 2.1 or 2 now. The dot levels will bring you up on everything and then the big ones, like the 3.0s, the 4.0s, will bring Spark and Drill current. And so we're going to kind of leapfrog those. So that's still a really important part of our business and we don't want to forget that part, but what we're trying here to do is, via the platform, is deliver all of that in one entity, right? >> So the converged data platform is relevant presumably because you've got the history of Hadoop, 'cause you got all these different components and you got to cobble 'em together and they're different interfaces and different environments, you're trying to unify that and you have unified that, right? >> Yeah, yeah. >> So what is your customer feedback with regard to the converged data platform? >> Yeah so it's a great question because for existing customers, it was like, ah, thank you. It was one of those, right, because we're listening. Actually, again, glad you said that. This week, in addition to Spark Summit we're doing our yearly customer advisory board so we've got, like a lot of vendors, we've got a 30 plus company customer advisory board that we bring in and we sit down with them for a couple of days and they give us feedback on what we should and shouldn't be doing and where, directional and all that, which is super important. And that's where a lot of this converged data platform came out of is the need for... There was just too much, it's kind of confusing. I'll give the example of streams, right? We came out with our streaming product last year and okay, I'm using Hadoop, I'm using your file system, I'm using NoSQL, now you're adding streams, this is great, but now, like MEP, the Ecosystem Packages, I have to keep everything current. You got to make it easier for me, you got to make my life easier for me. So for existing customers it's a stay current, I like this, the model, I can turn on and off what I want when I want. Great model for them, existing business. For new business it gets us out of that Hadoop-only mode, right? I kind of jokingly call us Hadoop plus plus plus plus. We keep adding solutions and add it to a single, cohesive data platform that we keep updated. And as I mentioned here, talking to new customers or new prospects, our potential new business, when I describe the model you can just see the light going on and they realize wow, there's a lot more to this than I had imagined. I got it earlier today, I thought you guys only did Hadoop. Which is a little infuriating as a marketer, but I think from a mechanism and a delivery and a message and a story point of view, it's really helped. >> More Cube time will help get this out there. (laughs) >> Well played, well played. >> It's good to have you back on. Okay, so Spark comes along a couple years ago and it was like ah, what's going to happen to Hadoop? So you guys embraced Spark. Talk more specifically about Spark, where it fits in your platform and the ecosystem generally. >> Spark, Hadoop, others as a entity to bring data into the converged data platform, that's one way to think about it. Way oversimplified, obviously, but that's a really great way, I think, to think about it is if we're going to provide this platform that anybody can query on, you can run analytics against. We talk a lot about now converged applications. So taking historical data, taking operational data, so streaming data, great example. Putting those together and you could use the Data Lake example if you want, that's fine. But putting them into a converged application in the middle where they overlap, kind of typical Venn diagram where they overlap, and that middle part is the converged application. What's feeding that? Well, Spark could be feeding that, Hadoop could be feeding that. Just yesterday we announced a Docker for containers, that could be feeding into the converged data platform as well. So we look at all of these things as an opportunity for us to manage data and to make data accessible at the enterprise level. And then that enterprise level goes back to what I was talkin' before, it's got to have all of those things, like multi-tenancy and snapshots and mirroring and data governance, security, et cetera. But Spark is a big component of that. All of the customers who came by here that I mentioned earlier, which are some really good names for us, are all using Spark to drive data into the converged data platform. So we look at it as we can help them build new applications within converged data platform with that data. So whether it's Spark data, Hadoop data, container data, we don't really care. >> So along those lines, if the focus of intense interest right now is on Spark, and Spark says oh, and we work with all these databases, data storers, file systems, if you approach a customer who's Spark first, what's the message relative to all the other data storers that they can get to through, without getting too techy, their API? >> Sure, sure. I think as you know, George, we support a whole bunch of APIs. So I guess for us it's the breadth. >> But I'm thinking of Spark in particular. If someone says specifically, I want to run Databricks, but I need something underneath it to capture the data and to manage it. >> Well I think that's the beauty of our file system there. As I mentioned, if you think about it from an architectural point of view, our file system along the bottom, or it could be our database or our streaming product, but in this instance-- >> George: That's what I'm getting at too, all three. >> Picture that as the bottom layer as your storage-- I shouldn't say storage layer but as the bottom layer. 'Cause it's not just storage, it's more than storage. Middle layer is maybe some of your open source tools and the like, and then above that is what I called your data delivery mechanisms. Which would be Spark, for example, one bucket. Another bucket could be Hadoop, and another bucket could be these microservices we're talking about. Let my draw the picture another way using a partner, SAP. One of the things we've had some success with SAP is SAP HANA sitting up here. SAP would love to have you put all your data in HANA. It's probably not going to happen. >> George: Yeah, good luck. >> Yeah, good luck, right? But what if you, hey customer, what if you put zero to two years worth of data, historical data, in HANA. Okay, maybe the customer starts nodding their head like you just did. Hey customer, what if you put two to five years worth of data in Business Warehouse. Guess what, you already own that. You've been an SAP customer for awhile, you already have it. Okay, the customer's now really nodding their head. You got their attention. To your original question, whether it's Spark or whatever, five plus years, put it in MapR. >> Oh, and then like HANA Vora could do the query. >> Drill can query across all of them. >> Oh, right including the Business Warehouse, okay. >> So we're running in the file system. That, to me, and we do this obviously with our joint SAP MapR customers, that to me is kind of a really cool vision. And to your original question, if that was Spark at the top feeding it rather than SAP, sure, right? Why not? >> What can you share with us, Bill, about business metrics around MapR? However you choose to share it, head count, want to give us gross margins by product, that's great, but-- (laughs) >> Would you like revenues too, Dave? >> We know they're very high because you're a software company, so that's actually a bad question. I've already profit-- (laughs) >> You don't have to give us top line revenues-- >> So what are you guys saying publicly about the company, its growth. >> That's fair. >> Give us the latest. >> Fantastic, number one. Hiring like crazy, we're well north of 500 people now. I actually, you want to hear a funny story? I yesterday was texting in the booth, with a candidate from my team, back and forth on salary. Did the salary negotiation on text right there in the booth and closed her, she starts on the 27th, so. >> Dave: Congratulations. >> I'm very excited about that. So moving along on that. Seven, 800 plus customers as we talk about... We just finished our fiscal year on January 31st, so we're on Feb one fiscal year. And we always do a momentum press release, which will be coming out soon. Hiring, again, like crazy, as I mentioned, executive staff is all filled in and built to scale which we're really excited about. We talk a lot about the kind of uptake of-- it used to be of the file system, Hadoop, et cetera on its own, but now in this one the momentum release we'll be doing, we'll talk about the converged data platform and the uplift we've seen from that. So we obviously can't talk revenue numbers and the like, but everything... David, I got to tell you, we've been doin' this a long time, all of that is just all moving in the right direction. And then the other example I'll give you from my world, in the partner world. Last year I rebranded our partner to the converged partner program. We're going with this whole converged thing, right? And we established three levels, elite, preferred, and affiliate with different levels there. But also, there's revenue requirements at each level, so elite, preferred, and affiliate, and there's resell and influence revenues, we have MDF funds, not only from the big guys coming to us, but we're paying out MDF funds now to select partners as well. So all of this stuff I always talk about as the maturity of the company, right? We're maturing in our messaging, we're maturing in the level of people who are joining, and we're maturing in the customers and the deals, the deal sizes and volumes that we're seeing. It's all movin' in the right direction. >> Dave: Great, awesome, congratulations. >> Bill: Thank you, yeah, I'm excited. >> Can you talk about number of customers or number of employees relative to last year? >> Oh boy. Honestly, George, I don't know off the top of my head. I apologize, I don't know the metric, but I know it's north of 500 today, of employees, and it's like seven, 800 customers. >> Okay, okay. >> Yeah, yeah. >> And a little bit more on this partner, elite, preferred, and affiliate. >> Affiliate, yeah. >> What did you call it, the converged partners program? >> Converged-- Yeah, yeah. >> What are some of the details of that? >> Sure. So the elites are invite only, and those are some of the bigger ones. So for us, we're-- >> Dave: Like, some examples. >> Cisco, SAP, AWS, others, but those are some of the big ones. And they were looking at things like resell and influence revenue. That's what I track in my... I always jokingly say at MapR, even though we're kind of a big startup now, I always jokingly say at MapR you have three jobs. You have the job you were hired for, you have your Thursday night job, and you have your Sunday night job. (Dave and George laugh) In the job that I was hired for, partner marketing, I track influence and resell revenue. So at the elite level, we're doing both. Like Cisco resells us, so this S-Series, we're in their SKU, their sales reps can go sell an S-Series for big data workloads or analytical workloads, MapR, on it, off you go. Our job then is cashing checks, which I like. That's a good job to have in this business. At the preferred level it's kind of that next tier of big players, but revenue thresholds haven't moved into the elite yet. Partners in there, like the MicroStrategies of the world, we're doing a lot with them, Tableau, Talend, a lot of the BI vendors in there. And then the affiliates are the smaller guys who maybe we'll do one piece of a campaign during the year with them. So I'll give you an example, Attunity, you guys know those guys right here? >> Sure >> Yeah, yeah. >> Last year we were doing a campaign on DWO, data warehouse offload. We wanted to bring them in but this was a MapR campaign running for a quarter, and we're typical, like a lot of companies, we run four campaigns a year and then my partner in field stuff kind of opts into that and we run stuff to support it. And then corporate marketing does something. Pretty traditional. But what I try and do is pull these partners into those campaigns. So we did a webinar with Attunity as part of that campaign. So at the affiliate level, the lower level, we're not doing a full go-to-market like we would with the elites at the top, but they're being brought into our campaigns and then obviously hopefully, we hope on the other side they're going to pull us in as well. >> Great, last question. What should we pay attention to, what's comin' up? >> Yeah, so-- >> Let's see, we got some events, we got Strata coming up you'll be out your way, or out MapR way. >> As my Twitter handle says, seat 11A. That's where I am. (laughs) Yeah, I mean the Docker announcement we're really excited about, and microservices. You'll see more from us on the whole microservices thing. Streaming is still a big one, we think, for this year. You guys probably agree. That's why we announced the MapR streaming product last year. So again, from a go-to-market point of view and kind of putting some meat behind streaming not only MapR but with partners, so streaming as a component and a delivery model for managing data in CDP. I think that's a big one. Machine learning is something that we're seeing more and more touching us from a number of customers but also from the partner perspective. I see all the partner requests that come in to join the partner program, and there's been an uptick in the machine learning customers that want to come in and-- Excuse me, partners, that want to be talking to us. Which I think is really interesting. >> Where you would be the sort of prediction serving layer? >> Exactly, exactly. Or a data store. A lot of them are looking for just an easy data store that the MapR file system can do. >> Infrastructure to support that, yeah. >> Commodity, right? The whole old promise of Hadoop or just a generic file system is give me easy access to storage on commodity hardware. The machine learning-- >> That works. >> Right. The existing machine learning vendors need an answer for that. When the customer asks them, they want just an easy answer, say oh, we just use MapR FS for that and we're done. Okay, that's fine with me, I'll take that one. >> So that's the operational end of that machine learning pipeline that we call DevOps for data scientists? >> Correct, right. I guess the nice synergy there is the whole, going back to the Docker microservices one, there's a DevOps component there as well. So, might be interesting marrying those together. >> All right, we got to go, Bill, thanks very much, good to see you again. >> All right, thank you. >> All right, George and I will be back to wrap. We're going to part two of our big data forecast right now, so stay with us, right back. (digital music) (synth music)