>> Live from Las Vegas, it's theCUBE. Covering AWS re:Invent 2018. Brought to you by Amazon Web Services, Intel. And their ecosystem partners. >> It's good to have you back here on theCUBE as we continue our day three coverage of AWS re:Invent. This is our 7th year at this show by the way, and it was just a little itty bitty thing some seven years ago. It's going to almost 40 thousand plus this year, and I think most of them are still here enjoying day three. Rebecca Knight with John Walls, and we're now joined by Dan Potter, who is the vice president of product management, and marketing at Attunity. Dan, good to see ya. >> Great to be back at theCUBE. >> You're a CUBE alumn, >> I am a CUBE alumn. >> we should say. >> Yes I am. >> And you were with Rebecca last year, two Bostonians, so again, I'll try to interject when I can, right? (Dan laughs) >> You don't speak our language fella. We can translate. >> It's alright Dan, it's okay. >> We were saying, before we got started here, you go to a lot of shows, right? >> Yes. >> And so, every one has it's own personality, it has it's own rhythm, it's own vibe. I mean, how would you characterize what you're seeing here? Especially, here we are day three, and it's still alive and thriving. >> It is absolutely overwhelming. This is my 3rd. Every year it grows. But I just seem to spend my days going from hotel to hotel, you know, to try to hit the sessions you want to, I feel like I'm always in an Uber, it's just so big, and the keynotes, there are so many new solutions that they're rolling out, it's just, the scale is so impressive. >> So what keeps you coming back? I mean, is it the chance to see so many customers in one place? Is it to hear the dizzying number of announcements from Andy Jassy? >> So I loved Andy's presentation, and the keynote this morning was great. For us, all of our customers are moving to the Cloud. I mean, Amazon really is the pioneer of people, and their transformation to the Cloud, and the success that customers are having with the Amazon platform is just astounding, and to see, over the last few years, how organizations have overcome some of the technical barriers, some of the perceptual, the regulatory barriers, they're all gone now, and this wave of movement to Amazon, and to Google and to Azure, it's real and it's happening and it's only accelerating. So it's exciting for us, you know, we're a vendor of data integration solutions. So we help customers move their data into the Cloud, and it's been great business for us, but it's been really fun connecting with our customers who we've gone through multiyear journeys with them as they're moving to the Cloud. So it's fun to see the success that they're having now with all the new technologies in the Amazon stack, it's great. >> So I want to ask you about the trends in the marketplace, what you're seeing, what you're hearing, as you've said, the security, the regulatory, the concerns are pretty much gone now. >> They are. >> They've had this aha moment. >> The Cloud is where I want to be. >> Yes. >> So what else are you seeing? >> Well, things continue to change, so if you look over the last few years, if you look at what's happening, all of those barriers are removed, but the technology stack it's still very much in motion in a positive way. New products are being introduced. Like today if you look at the announcement of a managed Kafka service. So one of the big trends we see is the move for realtime analytics, and to empower realtime analytics you need realtime data movement infrastructure, and Kafka is becoming an integral part of our customers data integration fabric. So that trend to realtime analytics, and having services like Kafka now on the Amazon platform, really important. >> And you got to Hadoop play, right? I mean you're working with Hadoop, you're working with Kafka as you point out. >> For sure. >> Yeah so. >> Well Hadoop's a great example of some of the changes that have happened over the last few years. Five years ago it was all Hadoop, and then all of a sudden the data lake strategy was Hadoop, and S3, and now it's Hadoop, S3, and it's Snowflake. There's so many different technologies that are really purposed to solve very particular pain points, this is the excitement for customers to be able to have this array of different technologies, and done right, if they have an architecture that supports them in moving that data where and when it's needed in whatever time frame, and structuring that information so it's analytics ready, that's the value, and that's some of the real innovations that you've seen over the last few years as this has all started to mature. >> Yeah, well I mean, take me through the data decision if you will. When am I going to leave on-prem? When am I going to move into the public cloud? As the volume of data grows, right? We're talking about trillions of processes within seconds. That's a big nut to crack for a lot of people. Why do I leave put on my Legacy system? Why do I move over? How reliable is it? What's the latency factor here? How do I make sure everybody gets to it, who needs to get to it, if it's over here, and over here? >> Exactly. >> So take us through that. >> So there are two big use cases that we see. One is analytics workloads. The Cloud is a perfect place for analytics. It allows you to create a very large data lake, bring in all kinds of hetero-genius data, bring it together, perform realtime transformation, and deliver analytics ready data to a wide variety of different business users and use cases. So the Cloud is really well purpose fit for analytics. If you look at all of the innovations that you've seen this week, a lot in AI and machine learning, a lot in realtime analytics. I mean this is the elasticity of the Cloud, and the storage capabilities, and the cost benefits of being able to store lots of information, and to be able to run different processes, analytic processes, when you need those, scale up and scale down, perfect fit for analytics. So that one is an absolute no brainer. We see a lot of people, this is the 1st choice for them as they're moving their analytic processes. The 2nd one we see is customers who have core transactional systems, like mainframe systems, you see this a lot in finance, big banks, insurance companies, these are 20 year old. >> I don't want to leave the mainframe, right? >> Not only are they not leaving the mainframe, but they're continuing to invest in the mainframe, and the mainframe is optimized for those transaction processing systems. But what they're not optimized for is how do I build new customer facing, web based applications, mobile applications, and the Cloud is the perfect environment to do that. So the way that we marry those two things together, and the big trend here is this is where realtime synchronization of data comes in. Every time there's an update on that mainframe system we can move that changed data to the Cloud in realtime. So if you're a bank, and you want to provide a web based interface, let me check my account balance, I need a realtime view, but you don't want to write that application against the mainframe, it's too expensive, the processing of a mainframe is too expensive. So if I can replicate that data into the Cloud, and I've got this whole modern array of tools in the Cloud, and I can take modern approaches, like microservices architectures, so I can have different optimized smaller databases that are purposed for different types of mobile apps, or web apps, that's the other trend that we're seeing. So that's kind of bridging that Legacy gap, and to your question of, what data do I leave on-prem? And what do I move to the Cloud? Those core transaction processing systems, they may never move to the Cloud, or in our lifetime we may not see those. Other databases, other applications are lift and shift moving to the Cloud. So things that are a more modern architecture we're seeing a lot of lift and shift directly to the Cloud. But it's going to be a mix for some time. >> So I understand you have a new launch of Attunity for data lakes on AWS, >> We do, yeah. >> tell our viewers a little more about that. >> So this is exciting. So I'll step back for a moment. We provide realtime data integration, and we move that changed data from on-prem into the Cloud. Moving the data is the 1st step, and it's an absolute requirement. But what really needs to happen in order to get the value from your data lake and cloud, you need to be able to not just move the data but shape that data, and make it purpose fit and analytics ready. So if our use case is analytics, and I want to be able to shape this data into a data mart, or I want to create an operational data store for realtime reporting or I'm a data scientist, I need a historic data store on a subset of information. Those are the analytic ready data sets that need to be created, and we're doing that end-to-end data pipeline. So realtime data movement, shaping that data, making it analytics ready and fully automating that process. So it's a streaming data pipeline process that is really leveraging the best of your core transactional systems, mainframe, SAP, Oracle, Legacy apps, files, and moving that to the Cloud in realtime so you can take advantage of all the wonderful capabilities on the Amazon platform. >> So you've been talking a lot about the changes in the data integration space, and sort of what we're seeing. What are your biggest challenges, and biggest opportunities as you're looking to 2019? >> So the biggest challenge is that there's a lot of moving parts, ya know? If you look at, again, you look at the last five years, and how many things have changed as an enterprise architect, they must scratch their head every morning and say what else Is going to change? I thought we had this figured out. So it's a challenge for us because there's a lot of different targets to support. Different clouds, Multicloud, multiple technologies, but that's also the opportunity. The opportunity here is that for us to play that role, and to help customers move data where and when they need it to whatever technology, we're completely agnostic. So if a new technology comes up, like a Snowflake. Great cloud data warehouse built on top of S3. We've seen a lot of customer interest in that, and that's been recent, the last two years out of nowhere. But very large enterprise customers have said, I want to jump on Snowflake. So for them to very quickly say, alright, now I'm going to point my data in addition to Hadoop on-prem, I'm going to point it into the Amazon Cloud, load it into Snowflake, automatically build out that data warehouse for me, and let's get real value. That's the opportunity and the excitement for us. It's never stale, there's always lots of work to do, and the types of impact that it's having on our customers, again it's really transformative to watch them go from the traditional monolithic, slow, traditional warehousing processes to more dynamic, realtime spinning up data marts for business users very very quickly so business users can have better insights, faster, make better decisions quicker, that has the impact that these organizations have been looking for, and that's why they're investing so much in the Cloud, so they can have that business impact, and we're really starting to see that. >> It's almost good new, bad news, right? The good news is things will always change. >> Yeah. >> The bad news is things will always change. >> Absolutely. >> But that's what makes it fun. Every year you come here and it's just, there's a buzz. (Rebecca laughs) There's always something exciting, and there's been some great announcements over the last few days, including ours, and it's been fun. >> It has been fun. >> Alright, Dan thanks for being with us. >> Happy to be here. >> Great to have you once again on theCUBE. >> Thanks for having me. >> See you soon I hope, down the road. >> I hope. >> Dan Potter joining us here on theCUBE. Back with more from AWS re:Invent after a short break.

>> Live from San Jose in the heart of Silicon Valley, it's theCUBE, covering DataWorks Summit 2018, brought to you by Hortonworks. >> Welcome back to theCUBE's live coverage of DataWorks here in sunny San Jose, California. I'm your host Rebecca Knight along with my co-host James Kobielus. We're joined by Dan Potter. He is the VP Product Management at Attunity and also Ali Bajwah, who is the principal partner solutions engineer at Hortonworks. Thanks so much for coming on theCUBE. >> Pleasure to be here. >> It's good to be here. >> So I want to start with you, Dan, and have you tell our viewers a little bit about the company based in Boston, Massachusetts, what Attunity does. >> Attunity, we're a data integration vendor. We are best known as a provider of real-time data movement from transactional systems into data lakes, into clouds, into streaming architectures, so it's a modern approach to data integration. So as these core transactional systems are being updated, we're able to take those changes and move those changes where they're needed when they're needed for analytics for new operational applications, for a variety of different tasks. >> Change data capture. >> Change data capture is the heart of our-- >> They are well known in this business. They have changed data capture. Go ahead. >> We are. >> So tell us about the announcement today that Attunity has made at the Hortonworks-- >> Yeah, thank you, it's a great announcement because it showcases the collaboration between Attunity and Hortonworks and it's all about taking the metadata that we capture in that integration process. So we're a piece of a data lake architecture. As we are capturing changes from those source systems, we are also capturing the metadata, so we understand the source systems, we understand how the data gets modified along the way. We use that metadata internally and now we're built extensions to share that metadata into Atlas and to be able to extend that out through Atlas to higher data governance initiatives, so Data Steward Studio, into the DataPlane Services, so it's really important to be able to take the metadata that we have and to add to it the metadata that's from the other sources of information. >> Sure, for more of the transactional semantics of what Hortonworks has been describing they've baked in to HDP in your overall portfolios. Is that true? I mean, that supports those kind of requirements. >> With HTP, what we're seeing is you know the EDW optimization play has become more and more important for a lot of customers as they try to optimize the data that their EDWs are working on, so it really gels well with what we've done here with Attunity and then on the Atlas side with the integration on the governance side with GDPR and other sort of regulations coming into the play now, you know, those sort of things are becoming more and more important, you know, specifically around the governance initiative. We actually have a talk just on Thursday morning where we're actually showcasing the integration as well. >> So can you talk a little bit more about that for those who aren't going to be there for Thursday. GDPR was really a big theme at the DataWorks Berlin event and now we're in this new era and it's not talked about too, too much, I mean we-- >> And global business who have businesses at EU, but also all over the world, are trying to be systematic and are consistent about how they manage PII everywhere. So GDPR are those in EU regulation, really in many ways it's having ripple effects across the world in terms of practices. >> Absolutely and at the heart of understanding how you protect yourself and comply, I need to understand my data, and that's where metadata comes in. So having a holistic understanding of all of the data that resides in your data lake or in your cloud, metadata becomes a key part of that. And also in terms of enforcing that, if I understand my customer data, where the customer data comes from, the lineage from that, then I'm able to apply the protections of the masking on top of that data. So it's really, the GDPR effect has had, you know, it's created a broad-scale need for organizations to really get a handle on metadata so the timing of our announcement just works real well. >> And one nice thing about this integration is that you know it's not just about being able to capture the data in Atlas, but now with the integration of Atlas and Ranger, you can do enforcement of policies based on classifications as well, so if you can tag data as PCI, PII, personal data, that can get enforced through Ranger to say, hey, only certain admins can access certain types of data and now all that becomes possible once we've taken the initial steps of the Atlas integration. >> So with this collaboration, and it's really deepening an existing relationship, so how do you go to market? How do you collaborate with each other and then also service clients? >> You want to? >> Yeah, so from an engineering perspective, we've got deep roots in terms of being a first-class provider into the Hortonworks platform, both HDP and HDF. Last year about this time, we announced our support for acid merge capabilities, so the leading-edge work that Hortonworks has done in bringing acid compliance capabilities into Hive, was a really important one, so our change to data capture capabilities are able to feed directly into that and be able to support those extensions. >> Yeah, we have a lot of you know really key customers together with Attunity and you know maybe a a result of that they are actually our ISV of the Year as well, which they probably showcase on their booth there. >> We're very proud of that. Yeah, no, it's a nice honor for us to get that distinction from Hortonworks and it's also a proof point to the collaboration that we have commercially. You know our sales reps work hand in hand. When we go into a large organization, we both sell to very large organizations. These are big transformative initiatives for these organizations and they're looking for solutions not technologies, so the fact that we can come in, we can show the proof points from other customers that are successfully using our joint solution, that's really, it's critical. >> And I think it helps that they're integrating with some of our key technologies because, you know, that's where our sales force and our customers really see, you know, that as well as that's where we're putting in the investment and that's where these guys are also investing, so it really, you know, helps the story together. So with Hive, we're doing a lot of investment of making it closer and closer to a sort of real-time database, where you can combine historical insights as well as your, you know, real-time insights. with the new acid merge capabilities where you can do the inserts, updates and deletes, and so that's exactly what Attunity's integrating with with Atlas. We're doing a lot of investments there and that's exactly what these guys are integrating with. So I think our customers and prospects really see that and that's where all the wins are coming from. >> Yeah, and I think together there were two main barriers that we saw in terms of customers getting the most out of their data lake investment. One of them was, as I'm moving data into my data lake, I need to be able to put some structure around this, I need to be able to handle continuously updating data from multiple sources and that's what we introduce with Attunity composed for Hive, building out the structure in an automated fashion so I've got analytics-ready data and using the acid merge capabilities just made those updates much easier. The second piece was metadata. Business users need to have confidence that the data that they're using. Where did this come from? How is it modified? And overcoming both of those is really helping organizations make the most of those investments. >> How would you describe customer attitudes right now in terms of their approach to data because I mean, as we've talked about, data is the new oil, so there's a real excitement and there's a buzz around it and yet there's also so many high-profile cases of breeches and security concerns, so what would you say, is it that customers, are they more excited or are they more trepidatious? How would you describe the CIL mindset right now? >> So I think security and governance has become top of minds right, so more and more the serveways that we've taken with our customers, right, you know, more and more customers are more concerned about security, they're more concerned about governance. The joke is that we talk to some of our customers and they keep talking to us about Atlas, which is sort of one of the newer offerings on governance that we have, but then we ask, "Hey, what about Ranger for enforcement?" And they're like, "Oh, yeah, that's a standard now." So we have Ranger, now it's a question of you know how do we get our you know hooks into the Atlas and all that kind of stuff, so yeah, definitely, as you mentioned, because of GDPR, because of all these kind of issues that have happened, it's definitely become top of minds. >> And I would say the other side of that is there's real excitement as well about the possibilities. Now bringing together all of this data, AI, machine learning, real-time analytics and real-time visualization. There's analytic capabilities now that organizations have never had, so there's great excitement, but there's also trepidation. You know, how do we solve for both of those? And together, we're doing just that. >> But as you mentioned, if you look at Europe, some of the European companies that are more hit by GDPR, they're actually excited that now they can, you know, really get to understand their data more and do better things with it as a result of you know the GDPR initiative. >> Absolutely. >> Are you using machine learning inside of Attunity in a Hortonworks context to find patterns in that data in real time? >> So we enable data scientists to build those models. So we're not only bringing the data together but again, part of the announcement last year is the way we structure that data in Hive, we provide a complete historic data store so every single transaction that has happened and we send those transactions as they happen, it's at a big append, so if you're a data scientist, I want to understand the complete history of the transactions of a customer to be able to build those models, so building those out in Hive and making those analytics ready in Hive, that's what we do, so we're a key enabler to machine learning. >> Making analytics ready rather than do the analytics in the spring, yeah. >> Absolutely. >> Yeah, the other side to that is that because they're integrated with Atlas, you know, now we have a new capability called DataPlane and Data Steward Studio so the idea there is around multi-everything, so more and more customers have multiple clusters whether it's on-prem, in the cloud, so now more and more customers are looking at how do I get a single glass pane of view across all my data whether it's on-prem, in the cloud, whether it's IOT, whether it's data at rest, right, so that's where DataPlane comes in and with the Data Steward Studio, which is our second offering on top of DataPlane, they can kind of get that view across all their clusters, so as soon as you know the data lands from Attunity into Atlas, you can get a view into that across as a part of Data Steward Studio, and one of the nice things we do in Data Steward Studio is that we also have machine learning models to do some profiling, to figure out that hey, this looks like a credit card, so maybe I should suggest this as a tag of sensitive data and now the end user, the end administration has the option of you know saying that okay, yeah, this is a credit card, I'll accept that tag, or they can reject that and pick one of their own. >> Will any of this going forward of the Attunity CDC change in the capture capability be containerized for deployment to the edges in HDP 3.0? I mean, 'cause it seems, I mean for internetive things, edge analytics and so forth, change data capture, is it absolutely necessary to make the entire, some call it the fog computing, cloud or whatever, to make it a completely transactional environment for all applications from micro endpoint to micro endpoint? Are there any plans to do that going forward? >> Yeah, so I think what HDP 3.0 as you mentioned right, one of the key factors that was coming into play was around time to value, so with containerization now being able to bring third-party apps on top of Yarn through Docker, I think that's definitely an avenue that we're looking at. >> Yes, we're excited about that with 3.0 as well, so that's definitely in the cards for us. >> Great, well, Ali and Dan, thank you so much for coming on theCUBE. It's fun to have you here. >> Nice to be here, thank you guys. >> Great to have you. >> Thank you, it was a pleasure. >> I'm Rebecca Knight, for James Kobielus, we will have more from DataWorks in San Jose just after this. (techno music)

>> Announcer: Live from Midtown Manhattan, it's theCUBE, covering Big Data New York City 2017. Brought to you by SiliconANGLE Media and its ecosystem sponsor. >> Okay, welcome back, everyone, to our live special CUBE coverage in New York City in Manhattan, we're here in Hell's Kitchen for theCUBE's exclusive coverage of our Big Data NYC event and Strata Data, which used to be called Strata Hadoop, used to be Hadoop World, but our event, Big Data NYC, is our fifth year where we gather every year to see what's going on in big data world and also produce all of our great research. I'm John Furrier, the co-host of theCUBE, with Peter Burris, head of research. Our next guest, Itamar Ankorion, who's the Chief Marketing Officer at Attunity. Welcome back to theCUBE, good to see you. >> Thank you very much. It's good to be back. >> We've been covering Attunity for many, many years. We've had many conversations, you guys have had great success in big data, so congratulations on that. But the world is changing, and we're seeing data integration, we've been calling this for multiple years, that's not going away, people need to integrate more. But with cloud, there's been a real focus on accelerating the scale component with an emphasis on ease of use, data sovereignty, data governance, so all these things are coming together, the cloud has amplified. What's going on in the big data world, and it's like, listen, get movin' or you're out of business has pretty much been the mandate we've been seeing. A lot of people have been reacting. What's your response at Attunity these days because you have successful piece parts with your product offering? What's the big update for you guys with respect to this big growth area? >> Thank you. First of all, the cloud data lakes have been a major force, changing the data landscape and data management landscape for enterprises. For the past few years, I've been working closely with some of the world's leading organizations across different industries as they deploy the first and then the second and third iteration of the data lake and big data architectures. And one of the things, of course, we're all seeing is the move to cloud, whether we're seeing enterprises move completely to the cloud, kind of move the data lakes, that's where they build them, or actually have a hybrid environment where part of the data lake and data works analytics environment is on prem and part of it is in the cloud. The other thing we're seeing is that the enterprises are starting to mix more of the traditional data lake, the cloud is the platform, and streaming technologies is the way to enable all the modern data analytics that they need, and that's what we have been focusing on on enabling them to use data across all these different technologies where and when they need it. >> So, the sum of the parts is worth more if it's integrated together seems to be the positioning, which is great, it's what customers want, make it easier. What is the hard news that you guys have, 'cause you have some big news? Let's get to the news real quick. >> Thank you very much. We did, today, we have announced, we're very excited about it, we have announced a new big release of our data integration platform. Our modern platform brings together Attunity Replicate, Attunity Compose for Hive, and Attunity Enterprise Manager, or AEM. These are products that we've evolved significantly, invested a lot over the last few years to enable organizations to use data, make data available, and available in the real time across all these different platforms, and then, turn this data to be ready for analytics, especially in Hive and Hadoop environments on prem and now also in the cloud. Today, we've announced a major release with a lot of enhancements across the entire product line. >> Some people might know you guys for the Replicate piece. I know that this announcement was 6.0, but as you guys have the other piece part to this, really it's about modernization of kind of old-school techniques. That's really been the driver of your success. What specifically in this announcement makes it, you know, really work well for people who move in real time, they want to have good data access. What's the big aha for the customers out there with Attunity on this announcement? >> That's a great question, thank you. First of all is that we're bringing it all together. As you mentioned, over the past few years, Attunity Replicate has emerged as the choice of many Fortune 100 and other companies who are building modern architectures and moving data across different platforms, to the cloud, to their lakes, and they're doing it in a very efficient way. One of the things we've seen is that they needed the flexibility to adapt as they go through their journey, to adapt different platforms, and what we give them with Replicate was the flexibility to do so. We give them the flexibility, we give them the performance to get the data and efficiency to move only the change of the data as they happen and to do that in a real-time fashion. Now, that's all great, but once the data gets to the data lake, how do you then turn it into valuable information? That's when we introduced Compose for Hive, which we talked about in our last session a few month ago, which basically takes the next stage in the pipeline picking up incremental, continuous data that is fed into the data lake and turning those into operational data store, historical data stores, data store that's basically ready for analytics. What we've done with this release that we're really excited about is putting all of these together in a more integrated fashion, putting Attunity Enterprise Manager on top of it to help manage larger scale environments so customers can move faster in deploying these solutions. >> As you think about the role that Attunity's going to play over time, though, it's going to end up being part of a broader solution for how you handle your data. Imagine for a second the patterns that your customers are deploying. What is Attunity typically being deployed with? >> That's a great question. First of all, we're definitely part of a large ecosystem for building the new data architecture, new data management with data integration being more than ever a key part of that bigger ecosystem because as all they actually have today is more islands with more places where the data needs to go, and to your point, more patterns in which the data moves. One of those patterns that we've seen significantly increase in demand and deployment is streaming. Where data used to be batch, now we're all talking about streaming. Kafka has emerged as a very common platform, but not only Kafka. If you're on Amazon Web Services, you're using Kinesis. If you're in Azure, you're using Azure Event Hubs. You have different streaming technologies. That's part of how this has evolved. >> How is that challenge? 'Cause you just bring up a good point. I mean, with the big trend that customers want is they want either the same code basis on prem and that they have the hybrid, which means the gateway, if you will, to the public cloud. They want to have the same code base, or move workloads between different clouds, multi-cloud, it seems to be the Holy Grail, we've identified it. We are taking the position that we think multi-cloud will be the preferred architecture going forward. Not necessarily this year, but it's going to get there. But as a customer, I don't want to have to rebuild employees and get skill development and retraining on Amazon, Azure, Google. I mean, each one has its own different path, you mentioned it. How do you talk to customers about that because they might be like, whoa, I want it, but how do I work in that environment? You guys have a solution for that? >> We do, and in fact, one of the things we've seen, to your point, we've seen the adoption of multiple clouds, and even if that adoption is staged, what we're seeing is more and more customers that are actually referring to the term lock-in in respect to the cloud. Do we put all the eggs in one cloud, or do we allow ourselves the flexibility to move around and use different clouds, and also mitigate our risk in that respect? What we've done from that perspective is first of all, when you use the Attunity platform, we take away all the development complexity. In the Attunity platform, it is very easy to set up. Your data flow is your data pipelines, and it's all common and consistent. Whether you're working on prem, whether you work on Amazon Web Services, on Azure, or on Google or other platforms, it all looks and feels the same. First of all, and you solve the issue of the diversity, but also the complexity, because what we've done is, this is one of the big things that Attunity is focused on was reducing the complexity, allowing to configure these data pipelines without development efforts and resources. >> One of the challenges, or one of the things you typically do to take complexity out is you do a better job of design up front. And I know that Attunity's got a tool set that starts to address some of of these things. Take us a little bit through how your customers are starting to think in terms of designing flows as opposed to just cobbling together things in a bespoke way. How is that starting to change as customers gain experience with large data sets, the ability, the need to aggregate them, the ability to present them to developers in different ways? >> That's a great point, and again, one of the things we've focused on is to make the process of developing or configuring these different data flows easy and modular. First, while in Attunity you can set up different flows in different patterns, and you can then make them available to others for consumption. Some create the data ingestion, or some create the data ingestion and then create a data transformation with Compose for Hive, and with Attunity Enterprise Manager, we've now also introduced APIs that allow you to create your own microservices, consuming and using the services enabled by the platform, so we provide more flexibility to put all these different solutions together. >> What's the biggest thing that you see from a customer standpoint, from a problem that you solve? If you had to kind of lay it out, you know the classic, hey, what problem do you solve? 'Cause there are many, so take us through the key problem, and then, if there's any secondary issues that you guys can address customers, that seems the way conversation starts. What are key problems that you solve? >> I think one of the major problems that we solve is scale. Our customers that are deploying data lakes are trying to deploy and use data that is coming, not from five or 10 or even 50 data sources, we work at hundreds going on thousands of data sources now. That in itself represents a major challenge to our customers, and we're addressing it by dramatically simplifying and making the process of setting those up very repeatable, very easy, and then providing the management facility because when you have hundreds or thousands, management becomes a bigger issue to operationalize it. We invested a lot in a management facility for those, from a monitoring, control, security, how do you secure it? The data lake is used by many different groups, so how do we allow each group to see and work only on what belongs to that group? That's part it, too. So again, the scale is the major thing there. The other one is real timeliness. We talked about the move to streaming, and a lot of it is in order to enable streaming analytics, real-time analytics. That's only as good as your data, so you need to capture data in real time. And that of course has been our claim to fame for a long time, being the leading independent provider of CDC, change data capture technology. What we've done now, and also expanded significantly with the new release, version six, is creating universal database streaming. >> What is that? >> We take databases, we take databases, all the enterprise databases, and we turn them into live streams. When you think, by the way, by the most common way that people have used, customers have used to bring data into the lake from a database, it was Scoop. And Scoop is a great, easy software to use from an open source perspective, but it's scripting and batch. So, you're building your new modern architecture with the two are effectively scripting and batch. What we do with CDC is we enable to take a database, and instead of the database being something you come to periodically to read it, we actually turn it into a live feed, so as the data changes in the database, we stream it, we make it available across all these different platforms. >> Changes the definition of what live streaming is. We're live streaming theCUBE, we're data. We're data streaming, and you get great data. So, here's the question for you. This is a good topic, I love this topic. Pete and I talk about this all the time, and it's been addressed in the big data world, but it's kind of, you can see the pattern going mainstream in society globally, geopolitically and also in society. Batch processing and data in motion are real time. Streaming brings up this use case to the end customer, which is this is the way they've done it before, certainly store things in data lakes, that's not going to go away, you're going to store stuff, but the real gain is in motion. >> Itamar: Correct. >> How do you describe that to a customer when you go out and say, hey, you know, you've been living in a batch world, but wake up to the real world called real time. How do you get to them to align with it? Some people get it right away, I see that, some people don't. How do you talk about that because that seems to be a real cultural thing going on right now, or operational readiness from the customer standpoint? Can you just talk through your feeling on that? >> First of all, this often gets lost in translation, and we see quite a few companies and even IT departments that when you talk, when they refer to real time, or their business tells them we need real time, what they understand from it is when you ask for the data, the response will be immediate. You get real time access to the data, but the data is from last week. So, we get real time access, but for last week's data. And that's what we try to do is to basically say, wait a second, when you mean real time, what does real time mean? And we start to understand what is the meaning of using last week's data versus, or yesterday's data, over the real time data, and that makes a big difference. We actually see that today the access, the availability, the availability to act on the real time data, that's the frontier of competitive differentiation. That's what makes a customer experience better, that's what makes the business more operationally efficient than the competition. >> It's the data, not so much the process of what they used to do. They're version of real time is I responded to you pretty quickly. >> Exactly, the other thing that's interesting is because we see it with, again, change of the capture becoming a critical component of the modern data architecture. Traditionally, we used to talk about different type of tools and technology, now CDC itself is becoming a critical part of it, and the reason is that it serves and it answers a lot of fundamental needs that are now becoming critical. One is the need for real-time data. The other one is efficiency. If you're moving to the cloud, and we talked about this earlier, if you're data lake is going to be in the cloud, there's no way you're going to reload all your data because the bandwidth is going to get in the way. So, you have to move only the delta. You need the ability to capture and move only the delta, so CDC becomes fundamental both in enabling the real time as well the efficient, the low-impact data integration. >> You guys have a lot of partners, technology partners, global SIs, resellers, a bunch of different partnership levels. The question I have for you, love to get your reaction and share your insight into is, okay, as the relationship to the customer who has the problem, what's in it for me? I want to move my business forward, I want to do digital business, I need to get up my real-time data as it's happening. Whether it's near real time or real time, that's evolution, but ultimately, they have to move their developers down a certain path. They'll usually hire a partner. The relationship between partners and you, the supplier to the customer, has changed recently. >> That's correct. >> How is that evolving? >> First of all, it's evolving in several ways. We've invested on our part to make sure that we're building Attunity as a leading vendor in the ecosystem of they system integration consulting companies. We work with pretty much all the major global system integrators as well as regional ones, boutique ones, that focus on the emerging technologies as well as get the modern analytic-type platforms. We work a lot with plenty of them on major corporate data center-level migrations to the cloud. So again, the motivations are different, but we invest-- >> More specialized, are you seeing more specialty, what's the trend? >> We've been a technology partner of choice to both Amazon and Microsoft for enabling, facilitating the data migration to the cloud. They of course, their select or preferred group of partners they work with, so we all come together to create these solutions. >> Itamar, what's the goals for Attunity as we wrap up here? I give you the last word, as you guys have this big announcement, you're bringing it all together. Integrating is key, it's always been your ethos in the company. Where is this next level, what's the next milestone for you guys? What do you guys see going forward? >> First of all, we're going to continue to modernize. We're really excited about the new announcement we did today, Replicate six, AEM six, a new version of Compose for Hive that now also supports small data lakes, Aldermore, Scaldera, EMR, and a key point for us was expanding AEM to also enable analytics on the data we generate as data flows through it. The whole point is modernizing data integration, providing more intelligence in the process, reducing the complexity, and facilitating the automation end-to-end. We're going to continue to solve, >> Automation big, big time. >> Automation is a big thing for us, and the point is, you need to scale. In order to scale, we want to generate things for you so you don't to develop for every piece. We automate the automation, okay. The whole point is to deliver the solution faster, and the way we're going to do it is to continue to enhance each one of the products in its own space, if it's replication across systems, Compose for Hive for transformations in pipeline automation, and AEM for management, but also to create integration between them. Again, for us it's to create a platform that for our customers they get more than the sum of the parts, they get the unique capabilities that we bring together in this platform. >> Itamar, thanks for coming onto theCUBE, appreciate it, congratulations to Attunity. And you guys bringing it all together, congratulations. >> Thank you very much. >> This theCUBE live coverage, bringing it down here to New York City, Manhattan. I'm John Furrier, Peter Burris. Be right back with more after this short break. (upbeat electronic music)

>> Narrator: Live from San Jose in the heart of Silicon Valley, it's the CUBE covering DataWorks Summit 2017 brought to you by Hortonworks. >> Hey, welcome back to the CUBE live from the DataWorks Summit day 2. We've been here for a day and a half talking with fantastic leaders and innovators, learning a lot about what's happening in the world of big data, the convergence with Internet of Things Machine Learning, artificial intelligence, I could go on and on. I'm Lisa Martin, my co-host is George Gilbert and we are joined by a couple of guys, one is a Cube alumni, Itamar Ankorion, CMO of Attunity, Welcome back to the Cube. >> Thank you very much, good to be here, thank you Lisa and George. >> Lisa: Great to have you. >> And Arvind Rajagopalan, the Director of Technology Services for Verizon, welcome to the Cube. >> Thank you. >> So we were chatting before we went on, and Verizon, you're actually going to be presenting tomorrow, at the DataWorks summit, tell us about building... the journey that Verizon has been on building a Data Lake. >> Oh, Verizon is over the last 20 years, has been a large corporation, made up of a lot of different acquisitions and mergers, and that's how it was formed in 20 years back, and as we've gone through the journey of the mergers and the acquisitions over the years, we had data from different companies come together and form a lot of different data silos. So the reason we kind of started looking at this, is when our CFO started asking questions around... Being able to answer One Verizon questions, it's as simple as having Days Payable, or Working Capital Analysis across all the lines of businesses. And since we have a three-major-ERP footprint, it is extremely hard to get that data out, and there was a lot of manual data prep activities that was going into bringing together those One Verizon views. So that's really what was the catalyst to get the journey started for us. >> And it was driven by your CFO, you said? >> Arvind: That's right. >> Ah, very interesting, okay. So what are some of the things that people are going to hear tomorrow from your breakout session? >> Arvind: I'm sorry, say that again? >> Sorry, what are some of the things that the people, the attendees from your breakout session, are going to learn about the steps and the journey? >> So I'm going to primarily be talking about the challenges that we ran into, and share some around that, and also talk about some of the factors, such as the catalysts and what drew us to sort of moving in that direction, as well as getting to some architectural components, from high-level standpoint, talk about certain partners that we work with, the choices we made from an architecture perspective and the tools, as well as to kind of close the loop on, user adoption and what users are seeing in terms of business value, as we start centralizing all of the data at Verizon from a backoff as Finance and Supply Chains standpoint. So that's kind of what I'm looking at talking tomorrow. >> Arvind, it's interesting to hear you talk about sort of collecting data from essentially backoff as operational systems in a Data Lake. Were there... I assume that the state is sort of more refined and easily structured than the typical stories we hear about Data Lakes. Were there challenges in making it available for exploration and visualization, or were all the early-use cases really just Production Reporting? >> So standard reporting across the ERP systems is very mature and those capabilities are there, but then you look at across-ERP systems and we have three major ERP systems for each of the lines of businesses, when you want to look at combining all of the data, it's very hard, and to add to that, you pointed on self-service discovery, and visualization across all three datas, that's even more challenging, because it takes a lot of heavy lift, to normalize all of the data and bring it into one centralized platform, and we started off the journey with Oracle, and then we had SAP HANA, we were trying to bring all the data together, but then we were looking at systems in our non-SAP ERP systems and bringing that data into a SAP-kind of footprint, one, the cost was tremendously high, also there was a lot of heavy lift and challenges in terms of manually having to normalize the data and bring it into the same kind of data models. And even after all of that was done, it was not very self-service oriented for our users and Finance and Supply Chain. >> Let me drill into two of those things. So it sounds like the ETL process of converting it into a consumable format was very complex, and then it sounds like also, the discoverability, like where a tool, perhaps like Elation, might help, which is very, very immature right now, or maybe not immature, it's still young. Is that what was missing, or why was the ETL process so much more heavyweight than with a traditional data warehouse? >> The ETL processes, there's a lot of heavy lifting there involved, because of the proprietary data structures of the ERP systems, especially SAP is... The data structures and how the data is used across clustered and pool tables, is very proprietary. And on top of that, bringing the data formats and structures from a PeopleSoft ERP system which are supporting different lines of businesses, so there are a lot of customization that's gone into place, there are specific things that we use in the ERPs, in terms of the modules and how the processes are modeled in each of the lines of businesses, complicates things a lot. And then you try and bring all these three different ERPs, and the nuances that they have over the years, try and bring them together, it actually makes it very complex. >> So tell us then, help us understand, how the Data Lake made that easier. Was it because you didn't have to do all the refinement before it got there. And tell us how Attunity helped make that possible. >> Oh absolutely, so I think that's one of the big things, why we picked the Hortonworks as one of our key partners in terms of buidling out the Data Lake, it just came on greed, you aren't necessarily worried about doing a whole lot of ETL before you bring the data in, and it also provides with the tools and the technologies from a lot other partners. We have a lot of maturity now, better provided self-service discovery capabilities for ad hoc analysis and reporting. So this is helpful to the users because now they don't have to wait for prolonged IT development cycles to model the data, do the ETL and build reports for the to consume, which sometimes could take weeks and months. Now in a matter of days, they're able to see the data they're looking for and they're able to start the analysis, and once they start the analysis and the data is accessible, it's a matter of minutes and seconds looking at the different tools, how they want to look at it, how they want to model it, so it's actually being a huge value from the perspective of the users and what they're looking to do. >> Speaking of value, one of the things that was kind of thematic yesterday, we see enterprises are now embracing big data, they're embracing Hadoop, it's got to coexist within our ecosystem, and it's got to inter-operate, but just putting data in a Data Lake or Hadoop, that's not the value there, it's being able to analyze that data in motion, at rest, structured, unstructured, and start being able to glean or take actionable insights. From your CFO's perspective, where are you know of answering some of the questions that he or she had, from an insights perspective, with the Data Lake that you have in place? >> Yeah, before I address that, I wanted to quickly touch upon and wrap up George's question, if you don't mind. Because one of the key challenges, and I do talk about how Attunity helped. I was just about to answer the question before we moved on, so I just want to close the loop on that a little bit. So in terms of bringing the data in, the data acquisition or ingestion is key aspect of it, and again, looking at the proprietary data structures from the ERP systems is very complex, and involves a multi-step process to bring the data into a strange environment, and be able to put it in the swamp bring it into the Lake. And what Attunity has been able to help us with is, it has the intelligence to look at and understand the proprietary data structures of the ERPs, and it is able to bring all the data from the ERP source systems directly into Hadoop, without any stops, or staging data bases along the way. So it's been a huge value from that standpoint, I'll get into more details around that. And to answer your question, around how it's helping from a CFO standpoint, and the users in Finance, as I said, now all the data is available in one place, so it's very easy for them to consume the data, and be able to do ad hoc analysis. So if somebody's looking to, like I said earlier, want to look at and calculate base table, as an example, or they want to look at working capital, we are actually moving data using Attunity, CDC replicate product, we're getting data in real-time, into the Data Lake. So now they're able to turn things around, and do that kind of analysis in a matter of hours, versus overnight or in a matter of days, which was the previous environment. >> And that was kind of one of the things this morning, is it's really about speed, right? It's how fast can you move and it sounds like together with Attunity, Verizon is really not only making things simpler, as you talked about in this kind of model that you have, with different ERP systems, but you're also really able to get information into the right hands much, much faster. >> Absolutely, that's the beauty of the near real-time, and the CDC architecture, we're able to get data in, very easily and quickly, and Attunity also provides a lot of visibility as the data is in flight, we're able to see what's happening in the source system, how many packets are flowing through, and to a point, my developers are so excited to work with a product, because they don't have to worry about the changes happening in the source systems in terms of DDL and those changes are automatically understood by the product and pushed to the destination of Hadoop. So it's been a game-changer, because we have not had any downtime, because when there are things changing on the source system side, historically we had to take downtime, to change those configurations and the scripts, and publish it across environments, so that's been huge from that standpoint as well. >> Absolutely. >> Itamar, maybe, help us understand where Attunity can... It sounds like there's greatly reduced latency in the pipeline between the operational systems and the analytic system, but it also sounds like you still need to essentially reformat the data, so that it's consumable. So it sounds like there's an ETL pipeline that's just much, much faster, but at the same time, when it's like, replicate, it sounds like that goes without transformations. So help us sort of understand that nuance. >> Yeah, that's a great question, George. And indeed in the past few years, customers have been focused predominantly on getting the data to the Lake. I actually think it's one of the changes in the fame, we're hearing here in the show and the last few months is, how do we move to start using the data, the great applications on the data. So we're kind of moving to the next step, in the last few years we focused a lot on innovating and creating the solutions that facilitate and accelerate the process of getting data to the Lake, from a large scope of systems, including complex ones like SAP, and also making the process of doing that easier, providing real-time data that can both feed streaming architectures as well as batch ones. So once we got that covered, to your question, is what happens next, and one of the things we found, I think Verizon is also looking at it now and are being concomitant later. What we're seeing is, when you bring data in, and you want to adopt the streaming, or a continuous incremental type of data ingestion process, you're inherently building an architecture that takes what was originally a database, but you're kind of, in a sense, breaking it apart to partitions, as you're loading it over time. So when you land the data, and Arvind was referring to a swamp, or some customers refer to it as a landing zone, you bring the data into your Lake environment, but at the first stage that data is not structured, to your point, George, in a manner that's easily consumable. Alright, so the next step is, how do we facilitate the next step of the process, which today is still very manual-driven, has custom development and dealing with complex structures. So we actually are very excited, we've introduced, in the show here, we announced a new product by Attunity, Compose for Hive, which extends our Data Lake solutions, and what Compose of Hive is exactly designed to do, is address part of the problem you just described, where's when the data comes in and is partitioned, what Compose for Hive does, is it reassembles these partitions, and it then creates analytic-ready data sets, back in Hive, so it can create operational data stores, it can create historical data stores, so then the data becomes formatted, in a matter that's more easily accessible for users, who want to use analytic tools, VI-tools, Tableau, Qlik, any type of tool that can easily access a database. >> Would there be, as a next step, whether led by Verizon's requirements or Attunity's anticipation of broader customer requirements, something where, there's a, if not near real-time, but a very low latency landing and transformation, so that data that is time-sensitive can join the historical data. >> Absolutely, absolutely. So what we've done, is focus on real-time availability of data. So when we feed the data into the Data Lake, we fit it into ways, one is directly into Hive, but we also go through a streaming architecture, like Kafka, in the case of Hortonworks, can also fit also very well into HDF. So then the next step in the process, is producing those analytic data sets, or data source, out of it, which we enable, and what we do is design it together with our partners, with our inner customers. So again when we work on Replicate, then we worked on Compose, we worked very close with Fortune companies trying to deal with these challenges, so we can design a product. In the case of Compose for Hive for example, we have done a lot of collaboration, at a product engineering level, with Hortonworks, to leverage the latest and greatest in Hive 2.2, Hive LLAP, to be able to push down transformations, so those can be done faster, including real-time, so those datasets can be updated on a frequent basis. >> You talked about kind of customer requirements, either those specific or not, obviously talking to telecommunications company, are you seeing, Itamar, from Attunity's perspective, more of this need to... Alright, the data's in the Lake, or first it comes to the swamp, now it's in the Lake, to start partitioning it, are you seeing this need driven in specific industries, or is this really pretty horizontal? >> That's a good question and this is definitely a horizontal need, it's part of the infrastructure needs, so Verizon is a great customer, and we even worked similarly in telecommunications, we've been working with other customers in other industries, from manufacturing, to retail, to health care, to automotive and others, and in all of those cases it's on a foundation level, it's very similar architectural challenges. You need to ingest the data, you want to do it fast, you want to do it incrementally or continuously, even if you're loading directly into Hadoop. Naturally, when you're loading the data through a Kafka, or streaming architecture, it's a continuous fashon, and then you partition the data. So the partitioning of the data is kind of inherent to the architecture, and then you need to help deal with the data, for the next step in the process. And we're doing it both with Compose for Hive, but also for customers using streaming architectures like Kafka, we provide the mechanisms, from supporting or facilitating things like schema unpollution, and schema decoding, to be able to facilitate the downstream process of processing those partitions of data, so we can make the data available, that works both for analytics and streaming analytics, as well as for scenarios like microservices, where the way in which you partition the data or deliver the data, allows each microservice to pick up on the data it needs, from the relevant partition. >> Well guys, this has been a really informative conversation. Congratulations, Itamar, on the new announcement that you guys made today. >> Thank you very much. >> Lisa: Arvin, great to hear the use case and how Verizon really sounds quite pioneering in what you're doing, wish you continued success there, we look forward to hearing what's next for Verizon, we want to thank you for watching the CUBE, we are again live, day two, of the DataWorks summit, #DWS17, before me my co-host George Gilbert, I am Lisa Martin, stick around, we'll be right back. (relaxed techno music)

>> Voiceover: From Orlando, Florida, it's theCube, covering Sapphire now, headline sponsored by SAP, Hana, the Cloud, the leader in Platform as a service, with support from Console Inc, the cloud internet company, now here are your hosts, John Furrier, and Peter Burris. >> Hey welcome back everyone, we are here live at SAP Sapphire in Orlando, Florida, this is theCube, Silicon Angle Media's flagship program, we go out to the events and extract the scene of the noise, I'm John Furrier with my co-host Peter Burris, our next guest is Matt Hayes, VP of SAP Business, Attunity, welcome to theCube. >> Thank you, thank you so much. >> So great to have you on, get the update on Attunity. You've been on theCube many times, you guys have been great supporters of theCube, appreciate that, and want to get a little update, so obviously Attunity, it's all about big data, Hana is a big data machine, it does a lot of things fast, certainly analystics being talked about here, but how do you guys fit in with SAP, what's your role here? How does it fit? >> Sure sure, well I think this is our ninth of tenth time here at Sapphire, we've been in the ecosystem for quite some time, our Gold Client solution is really designed to help SAP customers move data from production to non-production systems, and now, more throughout the landscape, or the enterprise even, so as SAP's evolved, we've evolved with SAP and a lot of our customers get a lot of value by taking real-life production data out of their production system, and moving that to non-production systems, training, sandbox, test environments. Some customer's use it for troubleshooting, you know, you have a problem with some data in production, you can bring that into a non-production system and test that, and some scrambling capabilities as well. Most SAP customers have a lot of risk if their copying the production data into non-production systems that are less secure, less regulated, so some of the data scrambling or obfuscation techniques that we have make it so that that data can safely go into those non-production systems and be protected. >> What's been your evolution? I mean obviously you mentioned you guys been evolving with SAP, so what is the current evolution? What's the highlight, what's the focus? >> So, obviously Hana has been the focus for quite some time and it still is, more and more of our customer's are moving to Hana, and adopting that technology, less so with S4, because that's kind of a newer phase, so a lot of people are making the two step approach of going to Hana, and then looking at S4, but Cloud as well, we can really aid in that Cloud enablement, because the scrambling. When we can scramble that sensitive data, it helps customer's feel comfortable and confident that they can put vendor and customer and other sensitive data in a Cloud based environment. >> And where are you guys winning? So what's the main thrust of why you guys are doing business in the SAP ecosystem. >> So with SAP you're always looking to do things better. And when you do things better, it results in cost savings on your project, and if you could save money on your project and do things smarter, you free up peoples time to focus on the fun projects, to focus on Hana, to focus on Cloud, and with our software, with our technology, by copying that data and providing real production data in the development and sandbox environments, we're impacting and improving the change control processes, we're impacting and improving the testing processes within companies, we're enabling some automation of some of those processes. >> Getting things up and running faster in the POC or Development environment? Real data? >> Yeah because you can be more nimble if you have real production data that you're working with while you're prototyping, you can make changes faster, you can be more confident in what you're promoting to production, you can be avoiding having a bad transport or a bad change going into the production environment and impact your business. So if you're not having to worry about that kind of stuff, you can worry about the fun stuff. You can look at Hana, you can look at Cloud, you can look at some of the newer technologies that SAP is providing. >> So, you guys grew up and matured, as you said, you've grown as SAP has grown, SAP used to be regarded as largely an applications company, now SAP, you know the S4, Hana platform, is a platform, and SAP's talking about partnerships, they're talking about making this whole platform even more available, accessible, to new developers through the Apple partnership etcetera, creates a new dynamic for you guys who have historically been focused on being able to automate the movement of data, certain data, certain processes, how are you preparing to potentially have to accommodate an accelerated rate of digitization as a consequence of all these partners, now working at SAP as a platform? >> That's a great question, and it's actually, it aligns with Attunity's vision and direction as well, so SAP, like you said, used to be an applications company, now it's an applications company with a full platform integrated all the way around, and Attunity is the same way, we came to Attunity through acquisition, and bringing our SAP Gold Client technology, but now we're expanding that, we're expanding it so that we can provide SAP data to other parts of the enterprise, we can combine data, we can combine highly structured SAP data with unstructured data, such as IOT Data, or social media streams in Hadoop, so the big data vision for Attunity is what's key, and right now we're in the process of blending what we do with SAP, with big data, which happens to align with SAP's platform. You know SAP is obviously helping customers move to Hana on the application side, but there's a whole analytics realm to it, that's even a bigger part of SAP's business right now, and that's kind of where we fit in. We're looking at those technologies, we're looking at how we can get data in and out of Hadoop, SAP Data in and out of Hadoop, how we can blend that with non SAP Data, to provide business value to SAP customers through that. >> Are you guys mainly focused on Fren, or are you also helping customer's move stuff into and out of Clouds and inside a hybrid cloud environment? >> Both actually, most SAP customer's are on Premise, so most of our focus is on Premise, we've seen a lot of customers move to the Cloud, either partial or completely. For those customers, they can use our technology the exact same way, and Attunity's replication software works on Prem and in the Cloud as well. So Cloud is definitely a big focus. Also, our relationship with Amazon, and Red Shift, there's a lot of Cloud capabilities and needs for moving data between on Premise and the Cloud, and back and forth. >> As businesses build increasingly complex workloads, which they clearly are, from a business stand point, they're trying to simplify the underlying infrastructure and technology, but they're trying to support increasingly complex types of work. How do you anticipate that the ecosystems ability to be able to map this on to technology is going to impact the role that data movement plays. Let me be a little bit more specific, historically, there were certain rules about how much data could be moved and how much work could be done in a single or a group of transactions. We anticipate that the lost art of data architecture across distances, more complex applications, it's going to become more important, are you being asked by your customers to help them think through, in a global basis, the challenges of data movement, as a set of flows within the enterprise, and not just point to point types of integration? >> I think we're starting to see that. I think it's definitely an evolving aspect of what's going on as, some low level examples that I can share with you on that are, we have some large global customers that have regional SAP environments, they might run one for North America, one for South America, Europe, and Asia-Pacific. Well they're consolidating them, some of those restrictions have been removed and now they're working on consolidating those regional instances into one global SAP instance. And if they're using that as a catalyst to move to Hana, that's really where you're getting into that realm where you're taking pieces that used to have to be distributed and broken up, and bringing them together, and if you can bring the structured enterprise application data on the SAP side together, now you can start moving towards some of the other aspects of the data like the analytics pieces. >> But you still have to worry about IOT, which is where are we going to process the data? Are we going to bring it back? Are we going to do it locally? You're worrying about sources external to your business, how you're going to move them in so that their intellectual property is controlled, my intellectual property is controlled, there's a lot of work that has to go in to thinking about the role that data movement is going to play within business design. >> Absolutely, and I actually think that that's part of the pieces that need to evolve over the next couple of years, it's kind of like the first time that you were here and heard about Hana, and here we are eight years later, and we understand the vision and the roadmap that that's played. That's happening now too, when you talk to SAP customers, some of them have clearly adopted the Hadoop technology and figured out how to make that work. You've got SAP Vora technology to bring data in and out of Hana from Hadoop, but that stuff is all brand new, we're not talking to a lot of customers that are using those. They're on the roadmap, they're looking at ways to do it, how to do it, but right now it's part of the roadmap. I think what's going to be key for us at Attunity is really helping customers blend that data, that IOT data, that social media stream data, with structured data from SAP. If I can take my customer master out of SAP and have that participate with IOT data, or if I can take my equipment master data out of SAP and combine that with Vlog data, IOT Data, I can start really doing predictive analytics, and if I can do those predictive analytics, with that unstructured data, I can use that to automate features within my enterprise application, so for example, if I know a part's going to fail, between 500 and 1000 hours of use, then I can proactively create maintenance tickets, or service notifications or something, so we can repair the device before it actually breaks. >> So talk about the, for the folks out there who want to kind of know the Attunity story a bit more, take a minute to explain kind of where you fit in, and where you, where SAP hands off to you, and where you fit specifically because big data management, there's are important technologies, but some say, well doesn't SAP have that? So where's the hand off? Where do you guys sister up against these guys the best? How should customers, or potential customers, know when to call you and what not. >> So, I often refer to SAP as a 747 Jumbo Jet right? So it's the big plane, and it's got everything in it. Anything at all, and all that you need to do, you could probably do it somewhere inside of SAP. There's an application for it, there's a platform for it, there's now a database for it, there's everything. So, a lot of customers work only in that realm, but there's a lot of customers that work outside of that too, SAP's an important part of the enterprise landscape, but there's other pieces too. >> People are nibbling at the solution, not fully baked out SAP. >> Right, right. >> You do one App. >> Yeah, and SAP's great at providing tools for example, to load data into Hana, there's a lot of capability to take non-SAP source data and bring it into Hana. But, what if you want to move that data around? What if you wanted to do some things different with it? What if you wanted to move some data out and back in? What if you want to, you know there's just a lot of things you want to be able to do with the data, and if you're all in on the SAP side, and you're all into the Hana platform, and that's what you're doing, you've probably got all the pieces to do that. But if you've got some pieces that are outside of that, and you need it all to play together, that's where Attunity comes in great, because Attunity has that, we're impartial to that, we can take data and move it around wherever, of course SAP is a really important part of our play in what we do, but we need to understand what the customers are doing, and everyday we talk to customers that are always looking, >> Give an example, give it a good example of that, customer that you've worked with, use a case. >> Yeah, let's see, most of my examples are going to be SAP centric, >> That's okay. >> We've got a couple of customers, I don't know if I can mention their names, where they come to us and say, "Hey we've got all this SAP data, and we might have 30 different SAP systems and we need all of that SAP data to pull together for us to be able to analyze it, and then we have non-SAP data that we want to partner with that as well. There might be terra-data, there might be Hadoop, might be some Oracle applications that are external that touch in, and these companies have these complex visions of figuring out how to do it, so when you look at Attunity and what we provide, we've got all these great solutions, we've got the replication technology, we've got the data model on the SAP side to copy the SAP data, we now have the data warehouse automation solution with Compose that keeps finding niche ways to work in, to be highly viable. >> But the main purpose is moving data around within SAP, give or take the Jumbo Jet, or 737. >> Well sometimes you just got to go down to the store and buy a half gallon of milk, right? And you're not going to jump on a Jumbo Jet to go down and get the milk. >> Right. >> You need tooling that makes it easy to get it. >> Got milk, it's the new slogan. Got data. >> Well there you go, the marketing side now. >> Okay so, vibe of the show, what's your take at SAP here, you've been here nine years, you've been looking around the landscape, you guys have been evolving with it, certainly it's exciting now. You're hearing really concrete examples of SAP showing some of the dashboards that McDermott's been showing every year, I remember when the iPad came out, "Oh the iPad's the most amazing thing", of course analytics is pretty obvious. That stuffs now coming to fruition, so there's a lot of growth going on, what's your vibe of the show? You seeing that, can you share any color commentary? Hallway conversations? >> Yeah, Sapphire's, you know, you get everything. You know it's like you said, the half gallon of milk, well we're at the supermarket right now, you need milk, you need eggs, you need flowers, whatever you need is here. >> The cake can be baked, if you have all the ingredients, Steve Job's says "put good frosting on it". (laughs) That's a UX. >> Lots of butter and lots of sugar. But yeah there's so many different focuses here at Sapphire, that it's a very broad show and you have an opportunity, for us it's a great opportunity to work with our partners closer, and it's also a good opportunity to talk to out customers, and certain levels within our customers, CIO's, VIP's. >> They're all together, they're all here. >> Right exactly, and you get to hear what their broader vision is, because every day we're talking to customers, and yeah we're hearing their broader vision, but here we hear more of it in a very confined space, and we get to map that up against our roadmap and see what we're doing and kind of say, yeah we're on the right track, I mean we need to be on the right track in two fronts. First and foremost with our customers, and second of all with SAP. And part of our long term success has been watching SAP and saying "okay, we can see where they're going with this, we can see where they're going with this, and this one they're driving really fast on, we've got to get on this track, you know, Hana. >> So the folks watching that aren't here, any highlights that you'd like to share? >> Wow, well you guys said yourself, Reggie Jackson was here the other night, that was pretty fantastic. I'm a huge baseball fan, go Cubby's, but it was fun to see Reggie Jackson. >> Park Ball, you know you had a share of calamities, I'm a Red Sox's man. >> Yeah you're wounds have been healed though (laughs). >> We've had the Holy Water been thrown from Babe Ruth. It was great that Reggie though was interesting, because we talk about a baseball concept that was about the unwritten rules, we saw Batista get cold-cocked a couple of days ago, and it brought up this whole unwritten rules, and we kind of had a tie in to business, which is the rules are changing, certainly in the business that we're in, and he talked about the unwritten rules of Baseball and at the end he said, "No, they aren't unwritten rules, they're written" And he was hardcore like MLB should not be messing with the game. >> Yeah. >> I mean Batista got fined, I think, what, five games? Was that the key mount? >> Yeah, yup. >> Didn't he get one game, and the guy that punched him got eight. >> That's right, he got it, eight games, that's right. So okay, MLB's putting pressure on them for structuring the game, should we let this stuff go? We came in late, second base, okay, what's your take on that? >> Well I mean as a Baseball fan I love the unwritten rules, I love the fact that the players police the game. >> Well that's what he was talking about, in his mind that's exactly what he was saying. That the rules amongst the players for policing the game are very, very well understood, and if Baseball tries to legislate and take it out of the players hands, it's going to lead to a whole bunch of chaotic behavior, and it's probably right. >> Yeah, and you've already got replay, and what was it, the Met's guy said he misses arguing with the umpires, and the next day he got thrown out (laughs). >> Probably means he wanted to get thrown out, needed a day off. What's going on with Attunity, what's next for you guys? What's next show, what's put on the business,. >> So, show-wise this is one of our most important shows of the year, events of the year, well I'll always be a tech-head, tech-heads are very targeted audience for us, we have a new version of Gold Client that's out a bit later this month, more under the hood stuff, just making things faster, and aligning it better with Hana and things like that, but we're really focused on integrating the solutions at Attunity right now. I mean you look at Attunity and Attunity had grown by acquisition, the RepliWeb acquisition in '11, and the acquisition of my company in 2013, we've added Compose, we've added Visibility, so now we've got this breath of solutions here and we're now knitting them together, and they're really coming together nicely. The Compose product, the data warehouse automation, I mean it's a new concept, but every time we show it to somebody they love it. You can't really point it at a SAP database, cause the data mile's too complex, but for data warehouse's of applications that have simple data models where you just need to do some data warehousing, basic data warehouses, it's phenomenal. And we've even figured out with SAP how we can break down certain aspects of that data, like just the financial data. If we just break down the financial data, can we create some replication and some change data capture there using the replicate technology and then feed it into Compose, provide a simple data warehouse solution that basic users can use. You know, you've got your BW, you've got your business objects and all that, but there's always that lower level, we're always talking to customers where they're still doing stuff like downloading contents of tables into spreadsheets and working with it, so Compose kind of a niche there. The visibility being able to identify what data's being used and what's not used, we're looking at combining that and pointing that at an SAP system and combining that with archiving technology and data retention technologies to figure out how we can tell a customer, alright here's your data retention policies, but here's where you're touching and not touching your data, and how can we move that around and get that out. >> Great stuff Matt, thanks for coming on theCube, appreciate that, if anything else I got to congratulate you on your success and, again, it's early stages and it's just going to get bigger and bigger, you know having that robust platform, and remember, not everyone runs their entire business on SAP, so there's a lot of other data warehouses coming round the corner. >> Yeah that's for sure, and we're well positioned and well aligned to deal with all types of data, me as an SAP guy, I love working with SAP data, but we've got a broader vision, and I think our broader visions really align nicely with what our customers want. >> Inter-operating the data, making it work for you, Got Data's new slogan here on theCube, we're going to coin that, 'Got Milk', 'Got Data'. Thanks to Peter Burris, bringing the magic here on theCube, we are live in Orlando, you're watching theCube. (techno music) >> Voiceover: There'll be millions of people in the near future that will want to be involved in their own personal well-being and wellness.

>> From the SiliconANGLE Media office in Boston, Massachusetts, it's theCUBE! Now here's your host, Stu Miniman! >> I'm Stu Miniman and this is a CUBE Conversation from our Boston area studio. Going to dig in to discuss the data catalog and to help me do that, I want to welcome to the program first-time guest Joe DosSantos who is the global Head of Data Management Strategy at Qlik. Joe, thank you so much for joining us. >> Good to be here Stu. >> All right so the data catalog, let's start there. People, in general, know what a catalog is. well maybe some of the millenniums might not know as much as those of us that been in the industry a little bit longer might have. So start there and help level set us. >> So our thinking is that there are lots of data assets around and people can't get at them. And just like you might be able to go to Amazon and shop for something, and you go through a catalog or you go to the library and you can see what's available, we're trying to approximate that same kind of shopping experience for data. You should be able to see what you have, you should be able to look for things that you need, you should be able to find things you didn't even know were available to you. And then you should be able to be able to put them into your cart in a secure way. >> So Joe, the step one is, I've gathered my data lake, or whatever oil or water analogy we want to use for gathering the data on, and then we've usually got analytic tools and lots of things there but this is a piece of that overall puzzle, do I have that right? >> That's exactly right so, if you think about what are the obstacles to analytics, there are studies out there that say less than one percent of analytics data is actually being analyzed. We're having a trouble with the pipelines to get data into the hands of people who can do something meaningful with it. So what is meaningful? Could be data science, could be natural language, which maybe if you have an Alexa at home or you just ask a question and that information is provided right back to you. So somebody wants to do something meaningful with data but they can't get it. Step one is go retrieve it, so our Attunity solution is really about how do we start to effectively build pipelines to go retrieve data from the source? The next step though is how do I understand that data? Cataloging isn't about just having a whole bunch of boxes on a shelf, it's being able to describe the contents of those shelves, it's being able to know that I need that thing. If you were to go into an Amazon.com experience and you say I'm going on a fishing trip and you're looking for a canoe, it'll offer you a paddle, it'll offer you lifejackets. It guides you through that experience. We want data to be the same way, this guided trip through the data that's available to you in that environment. >> Yes, it seems like - metadata is something we often talk about but it seems like even more than that. >> It really is, metadata is a broad term. If you want to know about your data, you want to know where it came from. I often joke that there are three things you want to know about data: what is it, where did it come from and who can have access to it under what circumstances. Now those are really simple concepts but they're really complex under the covers. What is data? Well, is this private information, is this person identifiable information, is a tax ID, is it a credit card? I come from TD Bank and we were very preoccupied with the idea of someone getting data that they shouldn't. You don't want everyone running around with credit cards, how do I recognize a credit card, how do I protect a credit card? So the idea of cataloging is not just available for everything, it's security. I'm going to give you an example of what happens when you walk into a pharmacy. If you walk into a pharmacy and you want a pack of gum or shampoo you walk up to the shelf and you grab it, it's carefully marked in the aisles, it's described but it's public, it's easy to get, there aren't any restrictions. If you wanted chewing tobacco or cigarettes you would need to present somebody with an ID who need to say that you are of age, who would need to validate that you are authorized to see that and if you wanted Oxycontin, you'd best have a prescription. Why isn't data like that, why don't we have rules that stipulate what kind of data belong in what kind of category and who can have access to it? We believe that you can, so a lot of impediments to that are about availability and visibility but also about security and we believe that once you've provisioned that data to a place then the next step is understanding clearly what it is, and who can have access to it so that you can provision it downstream to all of these different analytic consumers that need it. >> Yeah, data security is absolutely front and center, it's the conversation at board levels today, so the catalog, is it a security tool or it works with kind of your overall policies and procedures? >> So you need to have a policy. One of the fascinating things that exists in a lot of companies is you ask people please give me the titles of the columns that constitute personally identifiable information, you'll get blank stares. So if you don't have a policy, you don't have a construct, you're hopelessly lost. But as soon as you write that down now you can start building rules around that. You can know who can have access to what under what circumstances. When I was at TD we took care to try and figure out what the circumstances were that allowed people to do their job. If you're in marketing you need to understand the demographic information, you need to be able to distribute a marketing list that actually has people's names and addresses on it. Do you need their credit card number, probably not. We started to work through these scenarios of understanding what the nature of data was on a must-have basis and then you don't have to ask for approval every single time. If you go to Amazon you don't ask for approval to buy the canoe, you just know whether it's in stock, if it's available and if it's in your area. Same thing with data, we want to remove all of the friction associated with that just because the rules are in place. >> Okay, so now that I have the data what do I do with it? >> Well this is actually really an important part of out Qlik story. So Qlik is not trying to lock people into a Qlik visualization scenario. Once you have data what we're trying to do is to say that discovery might happen across lots of different platforms. Maybe you're a Tableau user, I don't know why, but there are Tableau users - no in fact we did use Tableau at TD - but if you wanted provision data and discover things and comparable BI tools, no problem. Maybe you want to move that into a machine learning type of environment, you have TensorFlow, you have H2O libraries doing predictive modeling, you have R and Python, all of those things are things that you might want to do, in fact these days a lot of times people don't want analytics and visualizations, they want to ask the questions, do you have an Amazon Alexa in your house? >> I have an Alexa and a Google Home. >> That's right so you don't want a fancy visualization, you want the answer to a question so a catalog enables that, a catalog helps you figure out where the data is that asks a question. So when you ask Alexa what's the capital of Kansas it's going through the databases that it has that are neatly tagged and cataloged and organized and it comes back with Topeka. >> Yeah. >> I didn't want to stump you there. >> Thank you Joe, boy, I think back in the world, there are people, ontological studies as to how I put these things together. As a user I'm guessing, using a tool like this, I don't need to have to figure how to set all this up, there's got to be way better tools and things like that just like in the discussion of metadata, most systems today do that for me or at least a lot of it but how much do I as a customer customize stuff and how much does it do it for me? >> So when you and I have a conversation we share a language and if I say where do you live you know that living implies a house, implies an address and you've made that connection. And so effectively all businesses have their own terminology and ontology of how they speak and what we do is, if we have that ontology described to us we will enforce those rules so we are able to then discover the data that fits that categorization of data. So we need the business to define that force and again a lot of this is about processing procedure. Anyone who works in technology knows that very little of the technological problems are actually about technology, they're about process and people and psychology. What we're doing is if someone says I care deeply and passionately about customers and customers have addresses and these are the rules around them, we can then apply those rules. Imagine the governance tools are there to make laws, we're like the police, we enforce those laws at time of shopping in that catalog metaphor. >> Wow Joe, my mind is spinning a little bit because one of the problems you have if you work for a big customer, you'd have different parts of the company that would all want the same answer but they'd ask it in very different ways and they don't speak the same language so does a catalog help with that? >> Well it does and it doesn't. I think that we are moving to a world in which for a lot of questions, truth is in the eye of the beholder. So if you think about a business that wants to close the books, you can't have revenue that was maybe three million, maybe four million. But if you want to say what was the effectiveness of the campaign that we ran last night? Was it more effective with women or men - why? Anytime someone asks a question like why, or I wonder if, these are questions that invite investigation, analysis and we can come to the table with different representations of that data, it's not about truth, it's about how we interpret that. So one of the peculiar and difficult things for people to wrap their arm around is in the modern data world with data democratization, two people can go in search of the same question and get wildly different answers. That's not bad, that's life, right? So what's the best movie that's out right now? There's no truth, it's a question of your tastes and what you need to be able to do is, as we move to a democratized world is, what were the criteria that were used? What was the data that was used? And so we need those things to be cited but the catalog is effectively the thing that puts you in touch with the data that's available. Think about your college research projects. You wrote a thesis or a paper, you were meant to draw a conclusion, you had to go to the library and get the books that you needed. And maybe, hopefully, no one had ever combined all of those ideas from those books to create the conclusion that you did. That's what we're trying to do every single day in the businesses of the world in 2019. >> Yeah it's a little scary in the world of science most things don't come down to a binary answer, there's the data to prove it and what we understand today might not be - if we look and add new data to it it could change. Bring in some customer examples as to what they're doing, how this impacts it and I wish brings more certainty into our world. >> Absolutely, so I come from TD Bank and I was the Vice President of Information Management Technology there, and we used Data Catalyst to catalog a very large data lake so we had a Hadoop data lake that was six petabytes, had about 200 different applications in it. And what we were able to do was to allow self service to those data assets in that lake. So imagine you're just looking for data and instead of having to call somebody or get a pipeline built and spend the next six months getting data, you go to a portal, you grab that data. So what we were able to do was to make it very simple to reduce that. We usually think that it takes about 50% of your time in an analysis context to find the data, to make the data useful, what if that was all done for you? So we created a shopping experience for that at an enterprise level. What was the goal - well at TD, we were all about legendary customer experience so we found very important were customer interactions and their experiences, their transactions, their web Qliks, their behavioral patterns and if you think about it what any company is looking to do is to catch a customer in the act of deciding and what are those critical things that people decide? In a bank it might be when to buy a house, when you need mortgages and you need potentially loans and insurance. For a healthcare company it might be when they change jobs, for a hospital it might be when the weather changes. And everybody's looking for an advantage to do that and you can only get that advantage if you're creative about recognizing those moments through analytics and then acting in real time with streaming to do something about that moment. >> All right so Joe one of the questions I have is is there an aspect of time when you go into this because I understand if I ask questions based on the data that I have available today but if I'd asked that two weeks before that it would be some different data and if I kept watching it, it would do that and so I've got certain apps I use like when's the best time to buy a ticket, when is the best time to do that, how does that play in? >> So there are two different dimensions to this, the first is what we call algorithmic decay. If you're going to try and develop an algorithm you don't want the data shifting under your feet as you do things because all of a sudden your results will change if you're not right and the sad reality is that most humans are not very original so if I look at your behavior for the past ten years and if I look at the past twenty it won't be necessarily different from somebody else, so what we're looking to do is catch mass patterns, that's the power of big data, to look at a lot of patterns to figure out the repeatability in most patterns. At that point you're not really looking for the data to change, then you go to score it and this is where the data changes all the time. So think about big data as looking at a billion rows and figuring out what's going on. The next thing would be traditionally called fast data which is now based on an algorithm - this event just happened, what should I do? That data is changing under your feet regularly, you're looking to stream that data, maybe with a change data capture tool like Attunity, you're looking to get that into the hands of people in applications to make decisions really quickly. Now what happens over time is people's behaviors change - only old people are on Facebook now right, you know this, so demographics change and the things that used to be very predictive fail to be and there has to be capability in an industry, in an enterprise to be able deal with those algorithms as they start to decay and replace them with something fresher. >> All right Joe, how do things like government compliance fit into this? >> So governance is really at the core of the catalog. You really need to understand what the rules are if you want to have an effective catalog. We don't believe that every single person in a data democratized world should have access to every single data element. So you need to understand what is this data, how should I protect it and how should I think about the overall protection of this data and the use of this data. This is a really important governance principle to figure out who can have access to these data sets under what circumstances. Again nothing to do with technology but the catalog should really enforce your policy and a really good catalog should help to enforce the policies that you're coming up with, with who should have access to that data under what circumstances. >> Okay so Joe this is a pretty powerful tool, how do customers measure that they're getting adoption, that they're getting the results that they were hoping to when they roll this out? >> No one ever woke up one day and said boy would it be great if I stockpiled petabytes of data. At the end of the day, >> I know some storage companies that say that. >> They wish the customers would say that but at the end of the day you have data for analytics value and so what is analytics value? Maybe it's about a predictive algorithm. Maybe it's about a vizualisation, maybe its about a KPI for your executive suite. If you don't know, you shouldn't start. What we want to start to do is to think about use cases that make a difference to an enterprise. At TD that was fundamentally about legendary customer experience, offering the next best action to really delight that customer. At SunLife that was about making sure that they had an understand from a customer support perspective about their consumers. At some of our customers, at a healthcare company it was about faster discovery of drugs. So if you understand what those are you then start from the analytical outcome to the data that supports that and that's how you get started. How can I get the datasets that I'm pretty sure are going to drive the needle and then start to build from there to make me able to answer more and more complex questions. >> Well great those are some pretty powerful use cases, I remember back in the early Hadoop days it was like let's not have the best minds of our time figuring out how you can get better ad clicks right? >> That's right it's much easier these days. Effectively Hadoop really allows you to do, what big data really allows you to do is to answer questions more comprehensively. There was a time when cost would prevent you from being able to look at ten years worth of history, those cost impediments are gone. So your analytics are can be much better as a result, you're looking at a much broader section of data and you can do much richer what-if analysis and I think that really the secret of any good analytics is encouraging the what-if kind of questions. So you want in a data democratized world to be able to encourage people to say I wonder if this is true, I wonder if this happened and have the data to support that question. And people talk a lot about failing fast, glibly, what does that mean? Well I wonder if right now women in Montana in summertime buy more sunglasses. Where's the data that can answer that question? I want that quickly to me and I want in five minutes to say boy Joe, that was really stupid. I failed and I failed fast but it wasn't because I spent the next six weeks looking for the data assets, it's because I had the data, got analysis really quickly and then moved on to something else. The people that can churn through those questions fastest will be the ones that win. >> Very cool, I'm one of those people I love swimming into data always seeing what you can learn. Customers that want to get started, what do you recommend, what are the first steps? >> So the first thing is really about critical use case identification. Again no one wants to stockpile data so we need to start to think about how the data is going to affect an outcome and think about that user outcome. Is it someone asking in natural language a question of an application to drive a certain behavior? Is it a real time decision, what is the thing that you want to get good at? I've mentioned that TD wanted to be good about customer experience and offer development. If you think about what Target did there's a notorious story about them being able to predict pregnancy because they recognized that there was an important moment, there was a behavioral change in consumers that would overall change how they buy. What's important to you, what data might be relevant for that, anchor it there, start small, go start to operationalize the pipes that get you the data that you need and encourage a lot of experimentation with these data assets that you've got. You don't need to create petabytes of data. Create the data sets that matter and then grow from use case to use case. One of our customers SunLife did a wonderful job of really trying to articulate seven or eight key use cases that would matter and built their lake accordingly. First it was about customer behavior then it was employee behavior. If you can start to think about your customers and what they care about there's a person out there that cares about customer attrition. There's a person out there that cares about employee attrition, there's a person out there that cuts costs about cost of delivery of goods. Let's figure out what they need and how to use analytics to drive that and then we can start to get smart about the data assets that can really cause that analytics to explode. >> All right well Joe, really appreciate all the updates on the catalogs there, data at the center of digital transformation for so many customers and illuminating some key points there. >> Happy to be here. >> All right thank you so much for watching theCUBE, I'm Stu Miniman. (upbeat music)

>> Announcer: Live from San Jose, California, it's the CUBE, covering Big Data Silicon Valley 2017. >> Good afternoon everyone. This is George Gilbert. We're at Silicon Valley Big Data in conjunction with Strata and Hadoop World. We've been here every year for six years and I'm pleased to bring with us today a really interesting panel, with our friends from Attunity, Itamar Ankorion. We were just discussing, has an Israeli name, but some of us could be forgiven for thinking Italian or Turkish. Itamar is CMO of Attunity. We have Chris Murphy who is from a very large insurance company that we can't name right now, and then Martin Lidl from Deloitte. We're going to be talking about their experience building a data lake, a high value data lake, and some of the technology choices they made, including how Attunity fits in that. Maybe kicking that off, Chris, perhaps you can tell us what the big objectives were for the data lake, in terms of what outcomes were you seeking. >> Okay, I'd start off by saying there wasn't any single objective. It was very much about putting in a key enterprise component that would facilitate many, many things. When I look at it now and I look back, with wisdom hopefully, I see it as trying to put in data as a service within the company. Very much we built it as an operational data lake first and foremost, because we wanted to generate value for the company. I very much convey to people that this was something that was worth investing in on an ongoing basis, and then on the back of that of course, once you've actually pulled all the data together and started to curate it and make it available, then you can start doing the research work as well. We were trying to get the best of both worlds from that perspective. >> Let me follow up with that just really quickly. It sounds like if you're doing data as a service, it's where central IT as a function created a platform on which others would build applications and you had to make that platform mature at a certain level, not just the software but the data itself. Then at that point, did you show prototype applications to different departments and business units, or how did the uptake, you know, how organically did that move? >> Not so much, it was very much a fast delivering, agile, set of projects working together, so we actually had, and we used to call it the holy trinity of the projects we were doing. We had putting in a new customer portal that would be getting all of its data from the data lake, putting in a new CRM system getting all of its data from the data lake and talking to the customer portal, and then of course at the back behind that, the data lake itself feeding all the data to these systems. We weren't developing in parallel to to those projects, but of course those were not small projects. Those were sizable beasts, but side by side with that, we were still able to use the data lake to do some proof of concept work around analytics. Interestingly, one of the first things we used the data lake for, in terms of on the analytics side, was actually meeting a government regulatory requirement, where they needed us to get an amount of data together for them very quickly. When I say quickly, I mean within two weeks. We went to our typical suppliers and said, "How long will this take?" About three months, they thought. In terms of actually using the data lake, we pulled the data together in about two days and most of the delays were due to the lack of strict requirements, where we were just figuring out exactly what people wanted, and that really helped demonstrate the benefit of having a data lake at base. >> So Martin, tell us how Deloitte, you know, with its sort of deep bench of professional services skills, could help make that journey easier for Chris and for others. >> There were actually a number of areas where we engaged ... We were all the way from the very beginning, engaged in working on the business case creation and really when it sort of came to life was when we brought our technology people actually in to work out a road map of how to deal with it. As Chris said, there were many moving parts, therefor many teams within Deloitte that were engaged with different areas of specialization, so from a big development perspective on the one hand to sales force, CRM in the background, and then obviously my team of sort of data ninjas that came in and built the data lake. What we also did is actually we partnered with other third parties on the testing side, so that we covered, really, the full life cycle there. >> If I were to follow up with that, it sounds like because there were other systems being built out in parallel that depended on this, you probably had less, fewer degrees of freedom in terms of what the data had to look like when you were done. >> I think that's true, to a degree, but when you look at that every model that we deployed, it was very much agile delivery and we, during the liberation phase, we were working together very closely across these three teams, right? So there was a certain amount of, well not freedom in terms of what to deliver in the end, but to come to an agreement as to what good will look like at the end of a sprint or for a release, so there were no surprises as such. Still, through the flexible architecture that we had built and the flexible model that we had delivering, we could also respond to changes very quickly, so if the product owner changed priority or made priority calls and changed priority items in the backlog, we could quite quickly respond to this. >> So Itamar, maybe you can help us understand how Attunity added value, that other products couldn't really do and how it made the overall pipeline more performant. >> Okay, absolutely. The project that again, this Fortune 100 company was putting together, was an operational data lake. It was very important for them to get data from a lot of different data sources, so they can merge it together for analytic purposes, and also get data in real time so they can support real time analytics using information that is very fresh. That data in many financial services and insurance companies came from the mainframe, so multiple systems on the mainframe as well as other systems, and they needed an efficient way to get the data ingested into their data lake, so that's where Attunity came in, as part of the overall data lake architecture, to support an incremental, continuous, universal data ingestion process. Attunity replicate lends itself to being able to load the data directly into the data lake, into Hadoop, in this case, or also if they opt to use Kafka or go through mechanisms like Kafka and others, so it provided a lot of flexibility architecturally to capture data as it changes, in their many different databases and feed that into the data lake so it can be used for different types of analytics. >> So just to drill down on that one level, 'cuz many of us would assume that, you know, the replication log that Attunity sort of models itself after, would be similar to the event log that Kafka works, sort of models itself after. Is it that if you use Kafka you have to modify the source systems, and therefor it puts more load on them, where as with Attunity you are sort of piggybacking on what's already happening, and so you don't add to the load on those systems? >> Okay, great question. Let me clarify. >> Okay. First of all, Kafka is a great technology that we're seeing more and more customers adopt as part of their overall big data management architectures. It's a public subscribe basically infrastructure that allows you to scale up the messaging of data and storage of data as events, as messages, so you can easily move it around and process it also in a more real time streaming fashion. Attunity complements Kafka and is actually very well integrated with it, as well as other streaming type of ingestion data processing technologies. What Attunity brings to the picture here is primarily the key function of technology, CDC, change data capture, which is the ability, the technology to capture the data as it changes, in many different databases. Do that in a manner that has very little impact, if any, on the source system and the environment, and deliver it in real time. So what Attunity does in a sense, we turn the databases to be live feeds that then can stream, either directly, either we can take it directly into platforms such as Hive, HDFS, or we can feed it into Kafka for further processing integration through Kafka integration. So again, it's very complementary in that sense. >> Okay. So maybe give us, Chris, a little more color on the before and after state, you know, before these multiple projects happened, and then the data lake as sort of a data foundation for these other systems that you're integrating. What business outcomes changed, and how did they change? >> Oof, that's a tough question. I've been asked many flavors of that question before and the analogy I always come back to is it's like we were moving from candle power to electricity. There's no single use case that shows this is why you need a data lake. It was many, many things they wanted to do. In the before picture, again that was always just very challenging, so like many companies, we've outsourced the mainframe support operation and running of our system to third parties, and we were constrained by that. You know, we were in that crazy situation where we couldn't get to our own data. By implementing the data lake, we've broken down that barrier. We now have things back in our control. I mentioned before that POC we did with the regulatory reporting, again, three months ... Two days. It was night and day in terms of what we were now able to do. >> Many banks are beginning to say that their old business model was get the customers' checking account and then, you know, upsell, cross sell, to all these other related products or services. Is something happening like that with insurance, where if you break down the data silos, it's easier to sell other services? >> There will be, is probably the best way to put it. We're not there yet, and you know it's a road, right? It's a long journey and we're doing it in stages, so I think we've done what? Three different releases on the data lake to date? That's very much on the plan. We want to do things like nudges to demonstrate to the customers how there are products that could be a very good fit for them, because once you understand your customer, you understand what their gaps are, what their needs, what their wants are. Again, very much in the roadmap, just not at that part of the map yet. >> So help us maybe understand some of the near term steps you want to take on that roadmap towards that nirvana. >> So, those >> And what the role Attunity as a vendor might play, and Deloitte, you know as a professional service organization, to help get you there. >> So Attunity was obviously was all about getting the data there as efficiently as possible. Unfortunately like many things, in your first iteration it's still, our data lake is still running on a batch basis, but we'd like to evolve that as time goes by. In terms of actually making use of the lake, one of the key things that we were doing on that was actually implementing a client matching solution, so we didn't actually have a MDM system in place for managing our customers. We had 12 different policy admin systems in place. Customers could be coming to us being enrolled, they could be a beneficiary, they could be the policy holder, they could be a power of attorney, and we could talk to someone on the phone and not really understand who they were. You get them into the data lake, you start to build up that 360 view about who people are, then you start to understand what can I do for this person. That was very much the journey we're going on. >> And Martin, have you worked with ... Are you organized by industry line and is there a sort of capability maturity level where you know, you can say, okay, you have to master these skills and at that skill level then you can do these richer business offerings? >> Yeah, absolutely. First of all, yes, we are organized by industry groups and we have sort of a common model across industry store that describe what you just said. When we talk about inside strength in organization, this is really where you are sort of moving to on the maturity curve, as you become more mature in using your analytical capabilities and turning data from just data into information, into a real asset you can actually monetize, right? Where we went with Chris' organization and actually there's many other life insurers, is actually sort of the first step on this journey, right? What Chris described around for the first time being able to see a customer centric view and see what a customer has in terms of product, and therefor what they don't have, right? And where there's opportunities for cross selling, this is sort of a first step into becoming more proactive, right? There's actually a lot more that can follow on after that, but yeah, we've got maturity models that we assess against and we sort of gradually move people, organizations to the right place for them, because it's not going to be right for every organization to be an inside driven organization, to make this huge investment, to get there, but most companies will benefit will benefit from nudging them in that direction. >> Okay, and on that note we're going to have to leave it here. I will say that I think that there's a session at 2:30 today with the Deloitte and the unnamed insurance team talking in greater depth about the case study, with Attunity. On that, we'll be taking a short break. We'll be back at Big Data Silicon Valley. This is George Gilbert and we'll see you in a few short minutes.