>> Narrator: Live from Washington D.C., it's theCUBE covering .conf2017 Brought to you by Splunk. (techno music) >> Well, welcome back to Washington D.C. We're at the Walter Washington Convention Center as we wrap up our coverage here of .conf2017. As Dave Vellante joins me, I'm John Walls here at theCUBE, coming to you live from our nation's capital. Joined by Team AWS here. With us we have rather, Ray Zhu rather, who is a senior product manager at AWS. And Roger Barga, who is the general manager of Amazon Kinesis Services. So gentlemen, thanks for being with us, we appreciate the time. >> Absolutely, thank you for the invitation. >> Dave: Oh, you're welcome. >> You bet. Alright, so let's just jump in. The streaming data thing, right? It's just blowing up. What's inspiring that popularity of the Cloud? What's kind of lit that fire and what's going to keep it burning? >> Yeah, I think over time, I think customers really do realize the value that you can get out of by collecting, analyzing, and reacting to data in real time. Cause that really provides a very differentiated experience to their customers, you know, for example you're able to analyze your user behavior data in real time, provide them with a much more engaging experience, much more relevant content. You're able to diagnosis your service, understand your law of data issues in real time, so that when you have an issue, you can fix that right away. So that really provides a very different customer experience. So I think our customers are realizing the value of real time processing, which is why we think streaming data is gaining more and more popularity. And this is why Cloud is all the good stuff that Cloud can offer and tell the customers. It's highly scalable, so you don't need to worry about if it's going to scale later on when I scale my business. It's a matter of sort of like click of a button. We scale the infrastructure for you and we got all the resource ready for you to go on streaming data. We got super, it's very cost effective, right? So that cause we price at very low. As we keep improving the efficiency of running the service, we reduce our cost structure, we return that back to our customers as a price cut. The third thing which I think is super important is agility, right cause you don't need to set up an infrastructure, install any software, make all the configurations. Starting up a Kinesis Stream is like 15 seconds on the average console, you're done. And it really allows the developers, the customers, to move fast and purely focus their resources and effort on the things that really differentiate their customer experience. >> So very AWS like, we love AWS, we're a customer, it's our favorite Cloud. We'll go on record of saying that, you know? (laughs) We're loyal to you guys. Crowd, our Crowd Chat App runs on it, basically run our whole company on Amazon, where we can. >> Roger: Great. >> In 2013, we got the preview of Kinesis. It was a lot of buzz. It was kind of before the whole streaming meme took over. We were talkin' about real time at the time, but so maybe you can take us through the evolution of Kinesis and where we are today. >> I'd be happy to. You know, when we first built Kinesis Stream, what the company was trying to do, is we had all of the AWS billing and metering records coming from all of our services, our EC2 incidences. This was a lot of data that had to be captured. And the way we were doing it was in batch. We were storing this data in S3 buckets. We were starting large EMR jobs up at the end of day actually to aggregate them by the customer account. So say this was your bill for the end of the day. But we had customers that said actually I'd like to know what I'm spending every hour, every few minutes. And frankly that batch processing wasn't scaling. So we had to innovate and create Kinesis Streams as a real time system that was constantly aggregating all of the billing and metering records that were coming in from our customer's accounts. Totalling them in near real time and we presented our customers with a new experience of billing and insights into their billing and even forecasts of what they were spending at any given time. But we had other teams that immediately looked at Kinesis and said hey, we're dealing with real time streaming data and our customers want it delivered and aggregated and provided, so Cloud watch logs and Cloud watch metrics built on top of us. And this was the start of something which continues to this day. Other services are looking at, and even customers, are looking at a Kinesis Stream and saying, that's a really useful abstraction that we can build a new service, a new experience for our customers. And today we have over a dozen AWS and Amazon retail services that build on top of Kinesis Streams as a fundamental abstraction to offer new experiences and new insights as three events. Cloud watch events, there's a host of services, which underneath Kinesis is running, but they're offering unique value building on top of it. Which is why Kinesis today is considered a foundational service and we can't build an AWS region without Kinesis being there for all these other services to build on top of. So that's been exciting to see that kind of adoption, different uses for this fundamental abstraction called a Kinesis Stream. And you know, it's also, and we can talk later about how it's transforming analytics, which is really exciting as well. >> Well, that's a great topic. I mean, why don't we talk about that. And one of the things that we've noted about AWS, and other Cloud providers, is obviously simplicity and delivering as a service is critical. We all know about the complexity of, for instance, the Hadoop Ecosystem And the challenges that a lot of customers have. Delivering that as a service has dramatically simplified their lives. That's why you see so many people going to the Cloud. We've always predicted that is what happened. Maybe talk about that a little bit. And then we can get into the analytics discussion. >> Yeah, so again, customers are always looking at ways to actually get insights into their data to better support their customers, to better understand what's going on in their business. And of course, Hadoop had managed EMR, had been a great benefit, cause customers could move their developers into the analytics that they want to do and not worry about this undifferentiated heavy lifting of operating these services. And the same is true for Kinesis Streams. But we're seeing customers, and if you stop for a moment and think about this, data never loses it's value. It always has it's historical value for machine learning, for understanding trends over time, but the insights that data has are actually very, very perishable and they can actually turn to zero within an hour if you can't extract those insights. That's the unique area where Kinesis Streams has kept adding value to our customers. Giving 'em the ability to get instant insights into what's going on in their business, their customers, their business processes, so they can take action and improve a customer experience, or capitalize on an opportunity. So what we're seeing and the role, I believe, that streaming data, at large, plays is about giving customers real time insights and then business opportunity to improve how they run their business. >> So. >> Go ahead, please. So who's using it? I mean or what's the if there's a sweet spot or a sweet spot for an industry or vertical to use that, I mean, in terms of whether it's in a minute, an hour, or whatever, what would that be? >> Yeah, so today, I'm really pleased to see, because we have watched this evolution since 2014, but today in virtually every market segment, where data is being continuously generated, we have customers that are actually taking advantage of the real time insights that they can get out of that data virtually every market segment. I'll pick a couple of examples which are kind of fun. One is Amazon Game Studios, near and dear to our heart. Now typically games are written, they're completely developed end to end. They're shipped in a box, made available to customers, and they hope that game and the engagement has the outcome that they want. Amazon Games Studios is actually writing that game in near real time ahead of their customers, so they release a new level of the game. They will actually watch the engagement. They'll look at how customers are dying, surviving, how long they're playing. And is it traveling in the direction they want? They stream all of the multi, all of the game data from their players in real time. And they build dashboards so they can see exactly how game play is going. And if they don't like it or they think they can make an improvement, they'll get right online, change the game itself, and re-deploy the game, so the customer experience is actually, within minutes it's being evolved. Another customer I like to talk about is Hertz Publishing. We all like to read. When Hertz started making the transition of their magazines, Cosmopolitan, Car and Driver, from print to digital form, they instrumented it so they could actually watch how long was a customer reading an article, how were their comments trending in Twitter and in Facebook. So they could actually get a sense of engagement with an article. Whether the article should be rebroadcast to other digital channels, other magazines. Should they change the article? Double down and write a new one. So again, they're engagement and then the business metrics by which they measure engagement and readers, readership have all increased because they have that intimate understanding of what's happening in real time. So again, every market segment, where there's data continuously generated, customers are using this to provide a better experience. >> That phrase undifferentiated heavy lifting we first heard it widely in the tech community in 2012 in Andy Jassy's keynote at Reinvent and it's become sort of a mantra. It probably was one well before that inside of AWS. And often times AWS doesn't talk about TCL but it's not the main reason why people go to the Cloud. You emphasized that a lot. And there's all this debate. Oh a cheaper on prem, oh no, Cloud is cheaper. But this idea of essentially eliminating labor that is doing that non-differentiated heavy lifting is something that you guys have really lived and popularized. We see that labor cost shifting from provisioning luns into other areas, up the stack, if you will. Application, digital business, analytics, et cetera. What are you guys seeing, in terms of how organizations, I mean, there's two types of organizations, right, the Cloud native guys who obviously didn't have the resources, but then enterprises that are bringing their business to the Cloud. Where are they shifting that undifferentiated heavy lifting labor towards? >> To. And they are in fact moving it up stream. We think about it very abstractly. You know, operating servers doesn't really bring any special IP that that company possesses to bear. It is about, you know, just managing servers, managing the software on it, figuring our how to scale. These are problems which we are able to take away. And we've often worked with customers and showed them the value of moving to our managed servers. And the excitement from the leadership, from their customers, is like wonderful. That project we couldn't, we aren't able to fund, if we can just onboard here, onto Kinesis for example, or any one of our managed services, then we can immediately move and get that fund project that we really wanted to fund, it would actually be unique value as move them over to that. So they're actually moving upstream as you said. And they're actually leveraging their unique understanding of their industry, their customer, to go ahead and add value there. So it is a distribution and I think in a very productive way. >> I want to ask about the data pipeline. So one of the values that AWS brings is simplification. When I look, however, at the data pipeline, it's very rich. If I look at the number of data services, Kinesis, Aurora, DYNAMO dv, EBS, S3, Glacier, each of these has a programming interface that is, I use the word primitive not in pejorative way but >> Roger: Yes, yes. >> But a deep level, low level. And so the data pipeline gets increasingly complex. There's probably a benefit of that, because I get access to the primitives, but it increases complexity. First of all, is that a fair assertion on my part? And how are your customers dealing with that? >> Be happy to take that one, yeah? >> Sure. >> Okay. >> Yep, so I think from our perspective all these different capabilities and technologies by customer choice. We build these services because our customers ask for them. And we order a wide variety so that people can choose for the developers who want to have full control over the entire staff, they have access to these lower level services. You know as you mentioned a few, DYNAMO dv, Kinesis Stream, S3, but we also build an abstraction layer on top of these different services. We also have a different set of customers asking for simplicity, just doing a specific type of things. I want you guys to take care of all the complexities, I just want that functionality. The example would be services like Kinesis Files, Kinesis Analytics, which is the abstraction layer we put on top. So for customers who are looking for simplicity, we also have these kind of capability for them. So I think at the end of the day, it's customer choice and demand. That's why we have this rich functionality and capabilities at AWS. >> So you guys have already solved that problem essentially, the one that I was sort of putting forth. >> So I won't say, I like Ray's answer. It's about listening to the customer. Cause in many cases if we would have, if we said, hey, we're going to go build a monolithic service that simplifies this, we would potentially disappoint many other customers. Say actually I really do want to have that low level control. >> Right. >> I'm used to having that. But when we hear customers asking for something which we can then translate to a service, we'll build a new service. And we will actually up level it and actually build a simpler abstraction for a targeted audience. So for us it's all about listening to the customers, build what they want, and if it means that we're going to actually bring two or three of our services together to work in concert for our customer, we'd do that in a heartbeat. >> Yeah that low level control also allows you to be presumably maybe not more agile but more responsive to the market demand. Because if you did build that monolithic service, you would essentially be locking yourselves in to a fossilized set of functions and services that you can't easily respond to market conditions. Is that a fair way to think about it? >> That is a fair statement, because basically our customers can look at these API's and together for these various services, realize how to use these API's in concert to get an end and done. And should we have precise feedback on a specific service, we can add a new API or tailor it over time. So it does give us a great deal of agility in working on these individual services. >> So Ray, you're a product guy and you're talking about listening to customers, right? And coming up with products, it's what you do. What are you hearing now? Where do people want to go now? Because I assume you've been in the market place for four years now with this, evolution is (clears throat), excuse me, perpetual, constant, so where do you want to take it? What's the next level or what's percolating in the back of your mind right now? >> Yeah, I think people always looking for different type of tools that they're familiar with or they want to use to analyze these data in real time and provide a differentiated customer experience. A concrete example I want to give is actually why we're here. At the Splunk Conference is at Kinesis we have a service called Kinesis Firehose. Based on customer demand when we launched Kinesis Streams, customers wanted to make sure they had access to data sooner than they used to do, but they want to use the tools they're familiar with. And apparently there's a diverse set of tools different customers want to use. We started with S3 for data lay, kind of storage, we used Reshift as a data warehouse. And overtime we heard from customers say, hey, we want you to use Splunk analyze the data. But we would like to use Kinesis Firehose and suggest a solution. Can you guys do something about it? So actually the two teams got together. We thought it's a strong customer value proposition, great capability for other customers. So we start this partnership. We're here actually earlier this day, today, we made the announcement actually, Kinesis Firehose is going to support Splunk as data of redestinations. And this integration is not in beta program. It's open for public sign up. Just go to the Kinesis Files website. You can sign up, get early access. So basically from today, you can use Kinesis Firehose in real time streaming (mumbles) service to get real data into your Splunk cluster. We're super excited about it. >> And okay, and I can access those Splunk services through the market place or what's the way in which I bring Splunk to? >> Good question. For this integration actually we're just a different version of Splunk. You can run Splunk on AWS using ECT extensions. You can access through the market place. You can have your, you can use native Splunk Cloud, which manage all the servers for you. You can also use Splunk on print in that regard. >> Okay. What have you guys learned since the orig, the first reinvent? I mean, I think, and again, I don't mean this as a pejorative but AWS is pretty dogmatic in its view of the world as you you are very strict (laughs) about your philosophy. But at the same time, as you learn about the enterprise, you've evolved. What have you learned about enterprise customers in that five, seven year journey of really getting intense with the enterprise? >> Yeah, that's a good question. But again, we're dogmatic about we always listen to our customers. We will never deviate from that. It's part of our culture. And the customers need to tell us where they want to go. And I'll tell you when we first started with Kinesis, just to answer your question, it was about low latency. We want to get that answer really fast, cause our ad tech customers are some of our very early customers, so it really was about that that extremely low latency response. As even our customers have started to look at Kinesis as a fundamental abstraction on which to put all of their business data in and now they're telling their customers well you should, if their IT customers within their company, if you want any business data, attach to the stream and pull it out. So now we're seeing less emphasis on low latency and to end processing, but increase request I want to be able to attach a dozen consumers, because this stream is actually supporting my entire enterprise. I want to have security. So we recently released encryption at rest. Our customers are asking for support for a VPC flow logs, which we hope to be talking with you about very soon. So now it's becoming actually very mainstream to actually, for the enterprise, and they want all the enterprise ready features, all the certifications, Fed Rep, Hippa, et cetera. So now we're actually seeing the Kinesis Stream itself being put into the enterprise as a fundamental building block for how they're going to run their business and how they're going to build their applications within the business. >> So that philosophy, I mean, you are customer driven first and there's a lot a, Andy Jassy says, there's a lot of ways to compete. You can be competitive oriented, but we're customer oriented. And I, it's clear, you guys do that. At the same time, customers sometimes don't know what they want, so you have to be good at decoding. >> Roger: Yes. >> If you listen to all your customers, you know, five years ago, they say, well we're not going to put any data in there. Sensitive data in the Cloud. Now everybody has sort of gotten over that. You said, alright, well we have to make it more secure. We have to get, you know, whatever certified, et cetera, et cetera. There's an art to this, listening to customers, isn't there? >> It gets back to one of our leadership principles of we always work customer backwards. We need to understand what they want, what experience they'd like to have. We have to anchor everything on that. But there is this element of invent and simplify. Because our customers may guess at what a solution is, but let's make sure we really understand what they want, what they need, the constraints under which that solution must offer. Then we go back to our engineering teams and other teams and we invent and simplify on their behalf. And we're not done there. We actually then bring these back to customers and in fact, why we're here today, we've spent two days talking to customers but even before this collaboration with Splunk began, we actually brought customers in and it turned out, their customers were often our customers. So we started talking, what is the problem? And we started with the very clear problem stain. And once both of our teams, we've loved working with Splunk, they work very customer backwards, like we do. And together once we understood this is the problem we are trying to address, and we had no preconception about how we're going to do it, but we worked backwards on what it would take to actually get that experience for our customers. And we're actually here beta testing it. And we're going to have a very aggressive two or three month beta test with customers, did we get it right? And we'll refine as well before we actually release it to the customer. So again, that working with the customer, work customer backwards. But invent and simplify on their behalf. Because many Splunk customers weren't aware of Firehose until we explained it to them as a potential solution. They're like ah, that will do it, thank you. >> So very outcome driven. I mean, I know you guys write press releases before you sometimes launch products. Sort of as you say, that's what you mean by working backwards, right? >> Roger: Yes, yes it is. It really is. >> Ray: You're good listeners. >> So far it's worked. (laughter) >> It's always fun at the company, when somebody says I have a customer, the entire room gets quiet and we all start listening. It's actually fun to see that, because that's the magic word. I have a customer and we all want to listen. What do they want? What are they challenged with? Cause that's where the innovation starts from which is exciting to be part of that. >> It's been a great formula, no doubt about that. >> It has, it has. >> Thank you both for being here. Didn't realize it was a big day. So congratulations >> Thank you. >> on your announcement as well. >> Absolutely. >> Ray, Roger, good to see you. >> It's great talking with you. >> Alright, you're watching theCUBE live here from Washington D.C. .conf2017. (techno music)