from the cube studios in Palo Alto in Boston connecting with thought leaders all around the world this is a cube conversation hi everybody this is David on tape with the cube and welcome to the special digital presentation we're really digging into how IBM is operationalizing and automating the AI and data pipeline not only for its clients but also for itself and with me is Julie Lochner who looks after offering management and IBM's data and AI portfolio Julie great to see you again okay great to be here thank you talk a little bit about the role you have here at IBM sure so my responsibility in offering management in the data and AI organization is really twofold one is I lead a team that implements all of the back-end processes really the operations behind anytime we deliver a product from the data AI team to the market so think about all of the release cycle management pricing product management discipline etc the other roles that I play is really making sure that um we are working with our customers and making sure they have the best customer experience and a big part of that is developing the data ops methodology it's something that I needed internally from my own line of business execution but it's now something that our customers are looking for to implement in their shops as well well good I really want to get into that and so let's let's start with data ops I mean I think you know a lot of people are familiar with DevOps not maybe not everybody's familiar with the data Ops what do we need to know about data well I mean you bring up the point that everyone knows DevOps and and then in fact I think you know what data Ops really does is bring a lot of the benefits that DevOps did for application development to the data management organizations so when we look at what is data ops it's a data management it's a it's a data management set of principles that helps organizations bring business ready data to their consumers quickly it takes it borrows from DevOps similarly where you have a data pipeline that associates a business value requirement I have this business initiative it's gonna drive this much revenue or this much cost savings this is the data that I need to be able to deliver it how do I develop that pipeline and map to the data sources know what data it is know that I can trust it so ensuring that it has the right quality that I'm actually using the data that it was meant for and then put it to use so in in history most dated management practices deployed a waterfall like methodology or implementation methodology and what that meant is all the data pipeline projects were implemented serially and it was dawn based on potentially a first-in first-out program management office with a DevOps mental model and the idea of being able to slice through all of the different silos that's required to collect the data to organize it to integrate it to validate its quality to create those data integration pipelines and then present it to the dashboard like if it's a Cognos dashboard for a operational process or even a data science team that whole end-to-end process gets streamlined through what we're calling data ops methodology so I mean as you well know we've been following this market since the early days of a dupe and people struggle with their data pipelines it's complicated for them there's a raft of tools and and and they spend most of their time wrangling data preparing data improving data quality different roles within the organization so it sounds like you know to borrow from from DevOps data OPS's is all about REME lining that data pipeline helping people really understand and communicate across end to end as you're saying but but what's the ultimate business outcome that you're trying to drive so when you think about projects that require data to again cut cost to automate a business process or drive new revenue initiatives how long does it take to get from having access to the data to making it available that duration for every time delay that is spent wasted trying to connect to data sources trying to find subject matter experts that understand what the data means and can verify its quality like all of those steps along those different teams and different disciplines introduces delay in delivering high quality data fast so the business value of data Ops is always associated with something that the business is trying to achieve but with a time element so if it's for every day we don't have this data to make a decision we're either making money or losing money that's the value proposition of data ops so it's about taking things that people are already doing today and figuring out the quickest way to do it through automation through workflows and just cutting through all of the political barriers that often happens when these data's cross different organizational boundaries yeah so speed time to insights is critical but to in and then you know with DevOps you're really bringing together the skill sets into sort of you know one super dev or one super ops it sounds with data ops it's really more about everybody understanding their role and having communication and line-of-sight across the entire organization it's not trying to make everybody a superhuman data person it's the whole it's the group it's the team effort really it's really a team game here isn't it well that's a big part of it so just like any type of practice there's people aspects process aspects and technology right so people process technology and while you're you're describing it like having that super team that knows everything about the data the only way that's possible is if you have a common foundation of metadata so we've seen a surgeons in the data catalog market and last you know six seven years and what what the what that the innovation in the data catalog market has actually enabled us to be able to drive more data ops pipelines meaning as you identify data assets you've captured the metadata you capture its meaning you capture information that can be shared whether they're stakeholders it really then becomes more of a essential repository for people to really quickly know what data they have really quickly understand what it means in its quality and very quickly with the right proper authority like privacy rules included put it to use for models you know dashboards operational processes okay and and we're gonna talk about some examples and one of them of course is ibm's own internal example but but help us understand where you advise clients to start I want to get into it where do I get started yeah I mean so traditionally what we've seen with these large data management data governance programs is that sometimes our customers feel like this is a big pill to swallow and what we've said is look there's an opera there's an opportunity here to quickly define a small project align it to a high-value business initiative target something that you can quickly gain access to the data map out these pipelines and create a squad of skills so it includes a person with DevOps type programming skills to automate an instrument a lot of the technology a subject matter expert who understands the data sources and its meaning a line of business executive who can translate bringing that information to the business project and associating with business value so when we say how do you get started we've developed a I would call it a pretty basic maturity model to help organizations figure out where are they in terms of the technology where are they in terms of organizationally knowing who the right people should be involved in these projects and then from a process perspective we've developed some pretty prescriptive project plans that help you nail down what are the data elements that are critical for this business business initiative and then we have for each role what their jobs are to consolidate the datasets map them together and present them to the consumer we find that six-week projects typically three sprints are perfect times to be able to in a timeline to create one of these very short quick win projects take that as an opportunity to figure out where your bottlenecks are in your own organization where your skill shortages are and then use the outcome of that six-week sprint to then focus on filling in gaps kick off the next project and iterate celebrate the success and promote the success because it's typically tied to a business value to help them create momentum for the next one all right that's awesome I want to now get into some examples I mean or you're we're both massachusetts-based normally you'd be in our studio and we'd be sitting here face-to-face obviously with kovat 19 in this crisis we're all sheltering in place you're up in somewhere in New England I happen to be in my studio believe it but I'm the only one here so relate this to kovat how would data ops or maybe you have a concrete example in in terms of how it's helped inform or actually anticipate and keep up-to-date with what's happening with building yeah well I mean we're all experiencing it I don't think there's a person on the planet who hasn't been impacted by what's been going on with this coded pandemic crisis so we started we started down this data obscurity a year ago I mean this isn't something that we just decided to implement a few weeks ago we've been working on developing the methodology getting our own organization in place so that we could respond the next time we needed to be able to you know act upon a data-driven decision so part of step one of our journey has really been working with our global chief data officer Interpol who I believe you have had an opportunity to meet with an interview so part of this year journey has been working with with our corporate organization I'm in the line of business organization where we've established the roles and responsibilities we've established the technology stack based on our cloud pack for data and Watson knowledge catalog so I use that as the context for now we're faced with a pandemic crisis and I'm being asked in my business unit to respond very quickly how can we prioritize the offerings that are gonna help those in critical need so that we can get those products out to market we can offer a you know 90-day free use for governments and Hospital agencies so in order for me to do that as a operations lead for our team I needed to be able to have access to our financial data I needed to have access to our product portfolio information I needed to understand our cloud capacity so in order for me to be able to respond with the offers that we recently announced you know you can take a look at some of the examples with our Watson citizen assistant program where I was able to provide the financial information required for us to make those products available for governments hospitals state agencies etc that's a that's a perfect example now to to set the stage back to the corporate global chief data office organization they implemented some technology that allowed us to ingest data automatically classify it automatically assign metadata automatically associate data quality so that when my team started using that data we knew what the status of that information was when we started to build our own predictive models and so that's a great example of how we've partnered with a corporate central organization and took advantage of the automated set of capabilities without having to invest in any additional resources or headcount and be able to release products within a matter of a couple of weeks and in that automation is a function of machine intelligence is that right and obviously some experience but but you couldn't you and I when we were consultants doing this by hand we couldn't have done this we could have done it at scale anyways it is it machine intelligence an AI that allows us to do this that's exactly right and as you know our organization is data and AI so we happen to have the a research and innovation teams that are building a lot of this technology so we have somewhat of an advantage there but you're right the alternative to what I've described is manual spreadsheets it's querying databases it's sending emails to subject matter experts asking them what this data means if they're out sick or on vacation you have to wait for them to come back and all of this was a manual process and in the last five years we've seen this data catalog market really become this augmented data catalog and that augmentation means it's automation through AI so with years of experience and natural language understanding we can comb through a lot of the metadata that's available electronically we can comb through unstructured data we can categorize it and if you have a set of business terms that have industry standard definitions through machine learning we can automate what you and I did as a consultant manually in a matter of seconds that's the impact the AI is had in our organization and now we're bringing this to the market and it's a it's a big part of where I'm investing my time both internally and externally is bringing these types of concepts and ideas to the market so I'm hearing first of all one of the things that strikes me is you've got multiple data sources and data lives everywhere you might have your supply chain data and your ERP maybe that sits on Prem you might have some sales data that's sitting in the SAS store in a cloud somewhere you might have you know a weather data that you want to bring in in theory anyway the more data that you have the better insights that you can gather assuming you've got the right data quality but so let me start with like where the data is right so so it sits anywhere you don't know where it's gonna be but you know you need it so that that's part of this right is being able to read it quickly yeah it's funny you bring it up that way I actually look a little differently it's when you start these projects the data was in one place and then by the time you get through the end of a project you find out that it's a cloud so the data location actually changes while we're in the middle of projects we have many or coming even during this this pandemic crisis we have many organizations that are using this as an opportunity to move to SAS so what was on Prem is now cloud but that shouldn't change the definition of the data it shouldn't change its meaning it might change how you connect to it um it might also change your security policies or privacy laws now all of a sudden you have to worry about where is that data physically located and am I allowed to share it across national boundaries right before we knew physically where it was so when you think about data ops data ops is a process that sits on top of where the data physically resides and because we're mapping metadata and we're looking at these data pipelines and automated workflows part of the design principles are to set it up so that it's independent of where it resides however you have to have placeholders in your metadata and in your tool chain where we oughta mating these workflows so that you can accommodate when the data decides to move because of corporate policy change from on-prem to cloud then that's a big part of what data Ops offers it's the same thing by the way for DevOps they've had to accommodate you know building in you know platforms as a service versus on from the development environments it's the same for data ops and you know the other part that strikes me and listening to you is scale and it's not just about you know scale with the cloud operating model it's also about what you're talking about is you know the auto classification the automated metadata you can't do that manually you've got to be able to do that in order to scale with automation that's another key part of data Ops is it not it's well it's a big part of the value proposition and a lot of a part of the business base right then you and I started in this business you know and Big Data became the thing people just move all sorts of data sets to these Hadoop clusters without capturing the metadata and so as a result you know in the last 10 years this information is out there but nobody knows what it means anymore so you can't go back with the army of people and have them query these data sets because a lot of the contact was lost but you can use automated technology you can use automated machine learning with natural under Snatcher Alang guaa Jing to do a lot of the heavy lifting for you and a big part of data ops workflows and building these pipelines is to do what we call management-by-exception so if your algorithms say you know 80% confident that this is a phone number and your organization has a you know low risk tolerance that probably will go to an exception but if you have a you know a match algorithm that comes back and says it's 99 percent sure this is an email address right and you I have a threshold that's 98% it will automate much of the work that we used to have to do manually so that's an example of how you can automate eliminate manual work and have some human interaction based on your risk threshold now that's awesome I mean you're right the no schema on right said I throw it into a data leg the data link becomes the data swap we all know that joke okay I want to understand a little bit and maybe you have some other examples of some of the use cases here but there's some of the maturity of where customers are I mean it seems like you got to start by just understanding what data you have cataloging it you're getting your metadata act in order but then you've got a you've got a data quality component before you can actually implement and get yet to insight so you know where our customers on the on the maturity model do you have any other examples that you can share yeah so when we look at our data ops maturity model we tried to simplify it I mentioned this earlier that we try to simplify it so that really anybody can get started they don't have to have a full governance framework implemented to take advantage of the benefits data ops delivers so what we did we said if you can categorize your data ops programs into really three things one is how well do you know your data do you even know what data you have the second one is and you trust it like can you trust its quality can you trust its meeting and the third one is can you put it to use so if you really think about it when you begin with what data do you know right the first step is you know how are you determining what data you know the first step is if you are using spreadsheets replace it with a data catalog if you have a department line of business catalog and you need to start sharing information with the departments then start expanding to an enterprise level data catalog now you mentioned data quality so the first step is do you even have a data quality program right have you even established what your criteria are for high quality data have you considered what your data quality score is comprised of have you mapped out what your critical data elements are to run your business most companies have done that for they're they're governed processes but for these new initiatives and when you identify I'm in my example with the Kovach crisis what products are we gonna help bring to market quickly I need to be able to find out what the critical data elements are and can I trust it have I even done a quality scan and have teams commented on its trustworthiness to be used in this case if you haven't done anything like that in your organization that might be the first place to start pick the critical data elements for this initiative assess its quality and then start to implement the workflows to remediate and then when you get to putting it to use there's several methods for making data available you know one is simply making a data Mart available to a small set of users that's what most people do well first they make a spreadsheet of the data available but then if they need to have multiple people access it that's when like a data Mart might make sense technology like data virtualization eliminates the need for you to move data as you're in this prototyping phase and that's a great way to get started it doesn't cost a lot of money to get a virtual query set up to see if this is the right join or the right combination of fields that are required for this use case eventually you'll get to the need to use a high performance ETL tool for data integration but Nirvana is when you really get to that self-service data prep where users can query a catalog and say these are the data sets I need it presents you a list of data assets that are available I can point and click at these columns I want as part of my you know data pipeline and I hit go and it automatically generates that output for data science use cases for a Cognos dashboard right that's the most mature model and being able to iterate on that so quickly that as soon as you get feedback that that data elements are wrong or you need to add something you can do it push button and that's where data observation to bring organizations to well Julie I think there's no question that this kovat crisis is accentuated the importance of digital you know we talk about digital transformation a lot and it's it's certainly real although I would say a lot of people that we talk to will say well you know not on my watch or I'll be retired before that all happens will this crisis is accelerating that transformation and data is at the heart of it you know digital means data and if you don't have your data you know story together and your act together then you're gonna you're not going to be able to compete and data ops really is a key aspect of that so you know give us a parting word all right I think this is a great opportunity for us to really assess how well we're leveraging data to make strategic decisions and if there hasn't been a more pressing time to do it it's when our entire engagement becomes virtual like this interview is virtual write everything now creates a digital footprint that we can leverage to understand where our customers are having problems where they're having successes you know let's use the data that's available and use data ops to make sure that we can iterate access that data know it trust it put it to use so that we can respond to those in need when they need it Julie Locker your incredible practitioner really hands-on really appreciate you coming on the Kuban and sharing your knowledge with us thank you okay thank you very much it was a pleasure to be here all right and thank you for watching everybody this is Dave Volante for the cube and we will see you next time [Music]