>> Narrator: Live from San Francisco, it's The Cube, covering Flink Forward, brought to you by Data Artisans. (upbeat music) >> Hi this is George Gilbert, we are at Flink Forward the conference for the Apache Flink community, sponsored by Data Artisans, which is the company commercializing Flink, and we have with us now Enrico Canzonieri, wait a minute I didn't get that right, Canzonieri, Sorry, Enrico from Yelp. And he's going to tell us how sort of Flink has taken Yelp by storm over the past year. Why don't we start off with where you were last year in terms of your data pipeline and what challenges you were facing. >> Yeah sure, so we had a Python company in the sense we developed most of our software in Python, so until last year we had most of our stream processing was depending on Python. We had developed and announced framework that was doing Python processing, and that was really what we had, there was no Flink running, most of the applications were built around a very simple interface that was process message function, that was what we expected developers to use, so no real obstruction there. >> Okay so, in other words, it sounds like you had a discrete task, request response or a batch, and then hand it off to the next function, and is that what the pipeline looked like? >> The pipeline was more of a streaming pipeline where we had a Kafka topic and input and we had these developers who would write this process message function where each message would be individually processed, there was a kind of the semantic of the pipeline there. And then we will get the result of that processing task into another Kafka topic and then get another processing function on top of that So we could have very easily, two or three processing tasks all connected by Kafka topics. Obviously there were like big limitations of that kind of architecture, especially when you want to do something more advanced that can include windowing, aggregation, or especially state management. >> Well Kafka has several layers of abstraction and I guess you'd have to go pretty low level to get the windowing, all the windowing and state management capabilities. >> Yeah it becomes really hard you basically have to implement by yourself unless you're using a (mumbles) on the conference platform or you're using what they call Kafka streams, but we are not using that. >> Oh, not? Okay. >> Obviously we were trying to implement that on top of what Python or Simple Flame work from zero. >> So tell us how the choice of Flink, sort of where did it hit your awareness and where did you start mapping it into this pipeline that was Python based? >> Yeah so we had the really I think two main use cases, this was last year, that we were struggling to get right and to really get working. The first one was a connector and the challenge there was to aggregate that locally, scale it to hundreds of streams, and then once we aggregated that locally, upload at our own S3. So that was one application, we were really struggling to get that work because in the framework we had, we had no real obstruction for windowing so we had this process message function where it was trying to implement all of that. And also because we were using a very low level Kafka code primitives, getting scalability was not as straight forward. So that was one application that was pretty challenging, the other one was really purer state full application, where we needed to retain the state forever. It was doing a windowed join across streams, so obviously the challenges in that case are even harder more because you have to implement a state management from the ground up. >> And all the time semantics. >> Yeah we had basically no even time semantics. We were not supporting that. >> Okay. >> So we looked at Flink because of even time support so then we could actually do even time processing. State management support, already implemented, it's way different than implementing it from the ground up. And then obviously the abstractions so the streaming primitives, you have windows, you have a nice interface that you can use that makes developers who are writing code it becomes easier for them. >> So let's start with the state management, help us walk through like, what capabilities in state management does Flink have relative to the lowest level abstraction you were using in Kafka or perhaps what spark structured streaming might provide. >> Yeah so I think the nice features in streams are really around the fact that the state management is implemented and fully supports the clusterized approach of Flink. So for example if you're using Kafka, Flink already, in the Kafka connector, Flink already provides a way to represent the Kafka state the state of a Kafka consumer. It also, for open editors, if you have a flat map or you have a window, state for Windows is already fully supported so if you are cumulating events, in your window, you don't really know what to do, then nothing special, these states will be automatically maintained by the Flink framework. That means that if Flink is taking a snapshot so a check point or a save point, all the state that was there will get stored in the check point and then will be able to recover. >> For the full window. >> Yeah. >> It's cause it understands the concept of the window when it does a check point. >> Yeah because there's native support in Flink for that. >> And what's the advantage of having state be integrated with the compute as opposed to compute and then some sort of API to a separate state manager? >> It's definitely like call to clarity, it's a big simplification of how you implement your code, your streaming application. Cause in the end if for every simple string application you need to go ahead and implement or define, implement basically the way your state gets stored, that really makes a very complex application, especially on the maintain, for maintenance. So in Flink you kind of focus on the business logic we actually did some tuning on the state manager that was necessary, but the tuning that we did applies in the same way across all the applications we built. Then users want to build an application, they focus on the business logic that they want, and they have, I would say the state is more kind of declarative, you say you want this map, you need this list and this state as part of the state and Flink will take care of actually making sure that that gets into the check point. >> So the sort of semantics of state management are built in at the compute layer, as opposed to going down to an API for a separate service in other implementations. >> Yeah, yeah. >> Okay. All right we have just a minute left, tell us about some of the things you're looking forward to doing with Flink, and are they similar to what the DA platform that's coming out from Data Artisans or do you have still a whole bunch of things on the data pipeline that you want to accomplish with just the core functionality? >> Yeah we definitely, I will say one of the features that we are really excited about is the stream sequel. I see a lot of potential there for new applications, we actually use a stream sequel at Yelp and we deploy that as a service so it makes it easier for users to deploy and to develop stream plus string applications. We definitely are planning to expand our Flink deployment into just new apps. Especially one of the things that we try to do especially building reusable components, and trying to deploy the reusable components are very coupled with the way we think about our data pipeline. >> Okay so would it be fair to say that can you look at the DA platform and say for companies that are not quite as sophisticated as you, that this is going to make it easier for you know main stream companies to build and deploy, operate? >> I see good potential there, I was looking at the presentation in the morning I like the integration we culminated for sure, since like that's where kind of the current trend for application deployment is going. So yeah I definitely see potential, I think for Yelp we clearly have a complex enough deployment and service integration that won't probably be a good fit for us. But probably companies that are approaching the route of Flink, now and we'll probably have an already existing deployment they may probably give it a try. >> Okay, all right Enrico we got to end it there but that was very helpful and thanks for stopping by. >> Thanks for having me here. >> Okay. And this is George Gilbert we are at Flink Forward, the Data Artisan's conference for the Apache Flink community, and we will be right back after this short break. (upbeat music)