>> Live from San Francisco, California, it's theCUBE, covering the IBM Chief Data Officer Summit. Brought to you by IBM. >> We're back at San Francisco. We're here at Fisherman's Wharf covering the IBM Chief Data Officer event #IBMCDO. This is the tenth year of this event. They tend to bookend them both in San Francisco and in Boston, and you're watching theCUBE, the leader in live tech coverage. My name is Dave Valante. John Thomas is here, Cube alum and distinguished engineer, Director of Analytics at IBM, and somebody who provides technical direction to the data science elite team. John, good to see you again. Steve Aliouk is back. He is the Vice President of Deep Learning in the Global Chief Data Office, thanks for comin' on again. >> No problem. >> Let's get into it. So John, you and I have talked over the years at this event. What's new these days, what are you working on? >> So Dave, still working with clients on implementing data science and AI data use cases, mostly enterprise clients, and seeing a variety of different things developing in that space. Things have moved into broader discussions around AI and how to actually get value out of that. >> Okay, so I know one of the things that you've talked about is operationalizing machine intelligence and AI and cognitive and that's always a challenge, right. Sounds good, we see this potential but unless you change the operating model, you're not going to get the type of business value, so how do you operationalize AI? >> Yeah, this is a good question Dave. So, enterprises, many of them, are beginning to realize that it is not enough to focus on just the coding and development of the models, right. So they can hire super-talented Python TensorFlow programmers and get the model building done, but there's no value in it until these models actually are operationalized in the context of the business. So one aspect of this is, actually we know, we are thinking of this in a very systematic way and talking about this in a prescriptive way. So, you've got to scope your use cases out. You got to understand what is involved in implementing the use case. Then the steps are build, run, manage, and each of these have technical aspects and business aspects around, right. So most people jump right into the build aspect, which is writing the code. Yeah, that's great, but once you build the code, build the models by writing code, how do you actually deploy these models? Whether that is for online invocation or back storing or whatever, how do you manage the performance of these models over time, how do you retrain these models, and most importantly, when these models are in production, how do I actually understand the business metrics around them? 'Cause this goes back to that first step of scoping. What are the business KPI's that the line of business cares about? The data scientist talks about data science metrics, position and recall and Area Under the ROC Curve and accuracy and so on. But how do these relate to business KPI's. >> All right, so we're going to get into each of those steps in a moment, but Steve I want to ask you, so part of your charter, Inderpal, Global Chief Data Officer, you guys have to do this for IBM, right, drink your own champagne, dog footing, whatever you call it. But there's real business reasons for you to do that. So how is IBM operationalizing AI? What kind of learnings can you share? >> Well, the beauty is I got a wide portfolio of products that I can pull from, so that's nice. Like things like AI open to Watson, some of the hardware components, all that stuffs kind of being baked in. But part of the reason that John and I want to do this interview together, is because what he's producing, what his thoughts are kind of resonates very well for our own practices internally. We've got so many enterprise use cases, how are we deciding, you know, which ones to work on, which ones have the data, potentially which ones have the biggest business impact, all those KPI's etcetera, also, in addition to, for the practitioners, once we decide on a specific enterprise use case to work on, when have they reached the level where the enterprise is having a return on investment? They don't need to keep refining and refining and refining, or maybe they do, but they don't know these practitioners. So we have to clearly justify it, and scope it accordingly, or these practitioners are left in this kind of limbo, where they're producing things, but not able to iterate effectively for the business, right? So that process is a big problem I'm facing internally. We got hundreds of internal use cases, and we're trying to iterate through them. There's an immense amount of scoping, understanding, etcetera, but at the same time, we're building more and more technical debt, as the process evolves, being able to move from project to project, my team is ballooning, we can't do this, we can't keep growing, they're not going to give me another hundred head count, another hundred head count, so we're definitely need to manage it more appropriately. And that's where this mentality comes in there's-- >> All right, so I got a lot of questions. I want to start unpacking this stuff. So the scope piece, that's we're setting goals, identifying the metrics, success metrics, KPI's, and the like, okay, reasonable starting point. But then you go into this, I think you call it, the explore or understanding phase. What's that all about, is that where governance comes in? >> That's exactly where governance comes in. Right, so because it is, you know, we all know the expression, garbage in, garbage out, if you don't know what data you're working with for your machine learning and deep learning enterprise projects, you will not have the resource that you want. And you might think this is obvious, but in an enterprise setting, understanding where the data comes from, who owns the data, who work on the data, the lineage of that data, who is allowed access to the data, policies and rules around that, it's all important. Because without all of these things in place, the models will be questioned later on, and the value of the models will not realized, right? So that part of exploration or understanding, whatever you want to call it, is about understanding data that has to be used by the ML process, but then at a point in time, the models themselves need to be cataloged, need to be published, because the business as a whole needs to understand what models have been produced out of this data. So who built these models? Just as you have lineage of data, you need lineage of models. You need to understand what API's are associated with the models that are being produced. What are the business KPI's that are linked to model metrics? So all of that is part of this understand and explore path. >> Okay, and then you go to build. I think people understand that, everybody wants to start there, just start the dessert, and then you get into the sort of run and manage piece. Run, you want a time to value, and then when you get to the management phase, you really want to be efficient, cost-effective, and then iterative. Okay, so here's the hard question here is. What you just described, some of the folks, particularly the builders are going to say, "Aw, such a waterfall approach. Just start coding." Remember 15 years ago, it was like, "Okay, how do we "write better software, just start building! "Forget about the requirements, "Just start writing code." Okay, but then what happens, is you have to bolt on governance and security and everything else so, talk about how you are able to maintain agility in this model. >> Yeah, I was going to use the word agile, right? So even in each of these phases, it is an agile approach. So the mindset is about agile sprints and our two week long sprints, with very specific metrics at the end of each sprint that is validated against the line of business requirements. So although it might sound waterfall, you're actually taking an agile approach to each of these steps. And if you are going through this, you have also the option to course correct as it goes along, because think of this, the first step was scoping. The line of business gave you a bunch of business metrics or business KPI's they care about, but somewhere in the build phase, past sprint one or sprint 2, you realize, oh well, you know what, that business KPI is not directly achievable or it needs to be refined or tweaked. And there is that circle back with the line of business and a course correction as it was. So it's a very agile approach that you have to take. >> Are they, are they, That's I think right on, because again, if you go and bolt on compliance and governance and security after the fact, we know from years of experience, that it really doesn't work well. You build up technical debt faster. But are these quasi-parallel? I mean there's somethings that you can do in build as the scoping is going on. Is there collaboration so you can describe, can you describe that a little bit? >> Absolutely, so for example, if I know the domain of the problem, I can actually get started with templates that help me accelerate the build process. So I think in your group, for example, IBM internally, there are many, many templates these guys are using. Want to talk a little bit about that? >> Well, we can't just start building up every single time. You know, that's again, I'm going to use this word and really resonate it, you know it's not extensible. Each project, we have to get to the point of using templates, so we had to look at those initiatives and invest in those initiatives, 'cause initially it's harder. But at least once we have some of those cookie-cutter templates and some of them, they might have to have abstractions around certain parts of them, but that's the only way we're ever able to kind of tackle so many problems. So no, without a doubt, it's an important consideration, but at the same time, you have to appreciate there's a lot of projects that are fundamentally different. And that's when you have to have very senior people kind of looking at how to abstract those templates to make them reusable and consumable by others. >> But the team structure, it's not a single amoeba going through all these steps right? These are smaller teams that are, and then there's some threading between each step? >> This is important. >> Yeah, that's tough. We were just talking about that concept. >> Just talking about skills and >> The bind between those groups is something that we're trying to figure out how to break down. 'Cause that's something he recognizes, I recognize internally, but understanding that those peoples tasks, they're never going to be able to iterate through different enterprise problems, unless they break down those borders and really invest in the communication and building those tools. >> Exactly, you talk about full stack teams. So you, it is not enough to have coding skills obviously. >> Right. What is the skill needed to get this into a run environment, right? What is the skill needed to take metrics like not metrics, but explainability, fairness in the moderates, and map that to business metrics. That's a very different skill from Python coding skills. So full stack teams are important, and at the beginning of this process where someone, line of business throws 100 different ideas at you, and you have to go through the scoping exercise, that is a very specific skill that is needed, working together with your coders and runtime administrators. Because how do you define the business KPI's and how do you refine them later on in the life cycle? And how do you translate between line of business lingo and what the coders are going to call it? So it's a full stack team concept. It may not necessarily all be in one group, it may be, but they have to work together across these different side loads to make it successful. >> All right guys, we got to leave it there, the trains are backing up here at IBM CDO conference. Thanks so much for sharing the perspectives on this. All right, keep it right there everybody. You're watchin' "theCUBE" from San Francisco, we're here at Fisherman's Wharf. The IBM Chief Data Officer event. Right back. (bubbly electronic music)