>> Live from San Francisco, it's theCUBE covering IBM Think 2019, brought to you by IBM. >> Welcome back everyone, live here in Moscone North in San Francisco, it's theCUBE's exclusive coverage of IBM Think 2019. I'm John Furrier, Dave Vellante. We're bringing down all the action, four days of live coverage. We've got two great guests here, Elenita Elinon, Executive Director of Quantitative Research at JP Morgan Chase, and John Thomas, Distinguished Engineer and Director of the Data Science Elite Team... great team, elite data science team at IBM, and of course, JP Morgan Chase, great innovator. Welcome to theCUBE. >> Welcome. >> Thank you very much. >> Thank you, thank you, guys. >> So I like to dig in, great use case here real customer on the cutting edge, JP Morgan Chase, known for being on the bleeding edge sometimes, but financial, money, speed... time is money, insights is money. >> Absolutely. Yes. >> Tell us what you do at the Quantitative Group. >> Well, first of all, thank you very much for having me here, I'm quite honored. I hope you get something valuable out of what I say here. At the moment, I have two hats on, I am co-head of Quantitative Research Analytics. It's a very small SWAT, very well selected group of technologists who are also physicists and mathematicians, statisticians, high-performance compute experts, machine learning experts, and we help the larger organization of Quantitative Research which is about 700-plus strong, as well as some other technology organizations in the firm to use the latest, greatest technologies. And how we do this is we actually go in there, we're very hands-on, we're working with the systems, we're working with the tools, and we're applying it to real use cases and real business problems that we see in Quantitative Research, and we prove out the technology. We make sure that we're going to save millions of dollars using this thing, or we're going to be able to execute a lot on this particular business that was difficult to execute on before because we didn't have the right compute behind it. So we go in there, we try out these various technologies, we have lots of partnerships with the different vendors, and IBM's been obviously one of few, very major vendors that we work with, and we find the ones that work. We have an influencing role as well in the organization, so we go out and tell people, "Hey, look, "this particular tool, perfect for this type of problem. "You should try it out." We help them set it up. They can't figure out the technology? We help them out. We're kind of like what I said, we're a SWAT team, very small compared to the rest of the organization, but we add a lot of value. >> You guys are the brain trust too. You've got the math skills, you've got the quantitative modeling going on, and it's a competitive advantage for your business. This is like a key thing, a lot of new things are emerging. One of things we're seeing here in the industry, certainly at this show, it's not your yesterday's machine learning. There's certainly math involved, you've got cognition and math kind of coming together, deterministic, non-deterministic elements, you guys are seeing these front edge, the problems, opportunities, for you guys. How do you see that world evolving because you got the classic math, school of math machine learning, and then the school of learning machines coming together? What kind of problems do you see these things, this kind of new model attacking? >> So we're making a very, very large investment in machine learning and data science as a whole in the organization. You probably heard in the press that we've brought in the Head of Machine Learning from CMU, Manuela Veloso. She's now heading up the AI Research Organization, JP Morgan, and she's making herself very available to the rest of the firm, setting strategies, trying different things out, partnering with the businesses, and making sure that she understands the use case of where machine learning will be a success. We've also put a lot of investments in tooling and hiring the right kinds of people from the right kinds of universities. My organization, we're changing the focus in our recruiting efforts to bring in more data science and machine learning. But, I think the most important thing, in addition to all that investment is that we, first and foremost, understand our own problems, we work with researchers, we work with IBM, we work with the vendors, and say, "Okay, this is the types of problems, "what is the best thing to throw at it?" And then we PoC, we prove it out, we look for the small wins, we try to strategize, and then we come up with the recommendations for a full-out, scalable architecture. >> John, talk about the IBM Elite Program. You guys roll your sleeves up. It's a service that you guys provide with your top clients. You bring in the best and you just jump in, co-create opportunities together, solving problems. >> That is exactly right. >> How does this work? What's your relationship with JP Morgan Chase? What specific use case are you going after? What are the opportunities? >> Yeah, so the Data Science Elite Team was setup to really help our top clients in their AI journey, in terms of bringing skills, tools, expertise to work collaboratively with clients like JP Morgan Chase. It's been a great partnership working with Elenita and her team. We've had some very interesting use cases related to her model risk management platform, and some interesting challenges in that space about how do you apply machine learning and deep learning to solve those problems. >> So what exactly is model risk management? How does that all work? >> Good question. (laughing) That's why we're building a very large platform around it. So model risk is one of several types of risk that we worry about and keep us awake at night. There's a long history of risk management in the banks. Of course, there's credit risk, there's market risk, these are all very well-known, very quantified risks. Model risk isn't a number, right? You can't say, "this model, which is some stochastic model "it's going to cost us X million dollars today," right? We currently... it's so somewhat new, and at the moment, it's more prescriptive and things like, you can't do that, or you can use that model in this context, or you can't use it for this type of trade. It's very difficult to automate that type of model risk in the banks, so I'm attempting to put together a platform that captures all of the prescriptive, and the conditions, and the restrictions around what to do, and what to use models for in the bank. Making sure that we actually know this in real time, or at least when the trade is being booked, We have an awareness of where these models are getting somewhat abused, right? We look out for those types of situations, and we make sure that we alert the correct stakeholders, and they do something about it. >> So in essence, you're governing the application of the model, and then learning as you go on, in terms of-- >> That's the second phase. So we do want to learn at the moment, what's in production today. Morpheus running in production, it's running against all of the trading systems in the firm, inside the investment bank. We want to make sure that as these trades are getting booked from day to day, we understand which ones are risky, and we flag those. There's no learning yet in that, but what we've worked with John on are the potential uses of machine learning to help us manage all those risks because it's difficult. There's a lot of data out there. I was just saying, "I don't want our Quants to do stupid things," 'cause there's too much stupidity happening right now. We're looking at emails, we're looking at data that doesn't make sense, so Morpheus is an attempt to make all of that understandable, and make the whole workflow efficient. >> So it's financial programming in a way, that's come with a whole scale of computing, a model gone astray could be very dangerous? >> Absolutely. >> This is what you're getting at right? >> It will cost real money to the firm. This is all the use-- >> So a model to watch the model? So policing the models, kind of watching-- >> Yes, another model. >> When you have to isolate the contribution of the model not like you saying before, "Are there market risks "or other types of risks--" >> Correct. >> You isolate it to the narrow component. >> And there's a lot of work. We work with the Model Governance Organization, another several hundred person organization, and that's all they do. They figure out, they review the models, they understand what the risk of the models are. Now, it's the job of my team to take what they say, which could be very easy to interpret or very hard, and there's a little bit of NLP that I think is potentially useful there, to convert what they say about a model, and what controls around the model are to something that we can systematize and run everyday, and possibly even in real time. >> This is really about getting it right and not letting it get out of control, but also this is where the scale comes in so when you get the model right, you can deploy it, manage it in a way that helps the business, versus if someone throws the wrong number in there, or the classic "we've got a model for that." >> Right, exactly. (laughing) There's two things here, right? There's the ability to monitor a model such that we don't pay fines, and we don't go out of compliance, and there's the ability to use the model exactly to the extreme where we're still within compliance, and make money, right? 'Cause we want to use these models and make our business stronger. >> There's consequences too, I mean, if it's an opportunity, there's upside, it's a problem, there's downside. You guys look at the quantification of those kinds of consequences where the risk management comes in? >> Yeah, absolutely. And there's real money that's at stake here, right? If the regulators decide that a model's too risky, you have to set aside a certain amount of capital so that you're basically protecting your investors and your business, and the stakeholders. If that's done incorrectly, we end up putting a lot more capital in reserve than we should be, and that's a bad thing. So quantifying the risks correctly and accurately is a very important part of what we do. >> So a lot of skillsets obviously, and I always say, "In the money business, you want the best nerds." Don't hate me for saying that... the smartest people. What are some of the challenges that are unique to model risk management that you might not see in sort of other risk management approaches? >> There are some technical challenges, right? The volume of data that you're dealing with is very large. If you are building... so at the very simplistic level, you have classification problems that you're addressing with data that might not actually be all there, so that is one. When you get into time series analysis for exposure prediction and so on, these are complex problems to handle. The training time for these models, especially deep learning models, if you are doing time series analysis, can be pretty challenging. Data volume, training time for models, how do you turn this around quickly? We use a combination of technologies for some of these use cases. Watson Studio running on power hardware with GPUs. So the idea here is you can cut down your model training time dramatically and we saw that as part of the-- >> Talk about how that works because this is something that we're seeing people move from manual to automated machine learning and deep learning, it give you augmented assistance to get this to the market. How does it actually work? >> So there is a training part of this, and then there is the operationalizing part of this, right? At the training part itself, you have a challenge, which is you're dealing with very large data volumes, you're dealing with training times that need to be shrunk down. And having a platform that allows you to do that, so you build models quickly, your data science folks can iterate through model creation very quickly is essential. But then, once the models have been built, how do you operationalize those models? How do you actually invoke the models at scale? How do you do workflow management of those models? How do you make sure that a certain exposure model is not thrashing some other models that are also essential to the business? How do you do policies and workflow management? >> And on top of that, we need to be very transparent, right? If the model is used to make certain decisions that have obvious impact financially on the bottom line, and an auditor comes back and says, "Okay, you made this trade so and so, why? What was happening at that time?" So we need to be able to capture and snapshot and understand what the model was doing at that particular instant in time, and go back and understand the inputs that went into that model and made it operate the way it did. >> It can't be a black box. >> It cannot be, yeah. >> Holistically, you got to look at the time series in real time, when things were happening and happened, happening, and then holistically tie that together. Is that kind of the impact analysis? >> We have to make our regulars happy. (laughing) That's number one, and we have to make our traders happy. We, as quantitative researchers, we're the ones that give them the hard math and the models, and then they use it. They use their own skillsets too to apply them, but-- >> What's the biggest needs that your stakeholders on the trading side want, and what's the needs on the compliance side, the traders want more, they want to move quickly? >> They're coming from different sides of it. Traders want to make more money, right? And they want to make decisions quickly. They want all the tools to tell them what to do, and for them to exercise whatever they normally exercise-- >> They want a competitive advantage. >> They want that competitive advantage, and they're also... we've got algo-trades as well, we want to have the best algo behind our trading. >> And the regulator side, we just want to make sure laws aren't broken, that there's auditing-- >> We use the phrase, "model explainability," right? Can you explain how the model came to a conclusion, right? Can you make sure that there is no bias in the model? How can you ensure the models are fair? And if you can detect there is a drift, what do you do to correct that? So that is very important. >> Do you have means of detecting sort of misuse of the model? Is that part of the governance process? >> That is exactly what Morpheus is doing. The unique thing about Morpheus is that we're tied into the risk management systems in the investment bank. We're actually running the same exact code that's pricing these trades, and what that brings is the ability to really understand pretty much the full stack trace of what's going into the price of a trade. We also have captured the restrictions and the conditions. It's in the Python script, it's essentially Python. And we can marry the two, and we can do all the checks that the governance person indicated we should be doing, and so we know, okay, if this trade is operating beyond maturity or a certain maturity, or beyond a certain expiry, we'll know that, and then we'll tag that information. >> And just for clarification, Morpheus is the name of the platform that does the-- >> Morpheus is the name of the model risk platform that I'm building out, yes. >> A final question for you, what's the biggest challenge that you guys have seen from a complexity standpoint that you're solving? What's the big complex... You don't want to just be rubber-stamping models. You want to solve big problems. What are the big problems that you guys are going after? >> I have many big problems. (laughing) >> Opportunities. >> The one that is right now facing me, is the problem of metadata, data ingestion, getting disparate sources, getting different disparate data from different sources. One source calls it a delta, this other source calls it something else. We've got a strategic data warehouse, that's supposed to take all of these exposures and make sense out of it. I'm in the middle because they're there, probably at the ten-year roadmap, who knows? And I have a one-month roadmap, I have something that was due last week and I need to come up with these regulatory reports today. So what I end up doing is a mix of a tactical strategic data ingestion, and I have to make sense of the data that I'm getting. So I need tools out there that will help support that type of data ingestion problem that will also lead the way towards the more strategic one, where we're better integrated with this-- >> John, talk about how you solve the problems? What are some of the things that you guys do? Give the plug for IBM real quick, 'cause I know you guys got the Studio. Explain how you guys are helping and working with JP Morgan Chase. >> Yeah, I touched upon this briefly earlier, which is from the model training perspective, Watson Studio running on Power hardware is very powerful, in terms of cutting down training time, right? But you've got to go beyond model building to how do you operationalize these models? How do I deploy these models at scale? How do I define workload management policies for these models, and connecting to their backbone. So that is part of this, and model explainability, we touched upon that, to eliminate this problem of how do I ingest data from different sources without having to manually oversee all of that. We need to manually apply auto-classification at the time of ingestion. Can I capture metadata around the model and reconcile data from different data sources as the data is being brought in? And can I apply ML to solve that problem, right? There is multiple applications of ML along this workflow. >> Talk about real quick, comment before we break, I want to get this in, machine learning has been around for a while now with compute and scale. It really is a renaissance in AI, it's great things are happening. But what feeds machine learning is data, the cleaner the data, the better the AI, the better the machine learning, so data cleanliness now has to be more real-time, it's less of a cleaning group, right? It used to be clean the data, bring it in, wrangle it, now you got to be much more agile, use speed of compute to make sure that you're qualifying data before it comes in, these machine learning. How do you guys see that rolling out, is that impacting you now? Are you thinking about it? How should people think about data quality as an input in machine learning? >> Well, I think the whole problem of setting up an application properly for data science and machine learning is really making sure that from the beginning, you're designing, and you're thinking about all of these problems of data quality, if it's the speed of ingestion, the speed of publication, all of that stuff. You need to think about the beginning, set yourself up to have the right elements, and it may not all be built out, and that's been a big strategy I've had with Morpheus. I've had a very small team working on it, but we think ahead and we put elements of the right components in place so data quality is just one of those things, and we're always trying to find the right tool sets that will enable use to do that better, faster, quicker. One of the things I'd like to do is to upscale and uplift the skillsets on my team, so that we are building the right things in the system from the beginning. >> A lot of that's math too, right? I mean, you talk about classification, getting that right upfront. Mathematics is-- >> And we'll continue to partner with Elenita and her team on this, and this helps us shape the direction in which our data science offerings go because we need to address complex enterprise challenges. >> I think you guys are really onto something big. I love the elite program, but I think having the small team, thinking about the model, thinking about the business model, the team model before you build the technology build-out, is super important, that seems to be the new model versus the old days, build some great technology and then, we'll put a team around it. So you see the world kind of being a little bit more... it's easier to build out and acquire technology, than to get it right, that seems to be the trend here. Congratulations. >> Thank you. >> Thanks for coming on. I appreciate it. theCUBE here, CUBE Conversations here. We're live in San Francisco, IBM Think. I'm John Furrier, Dave Vellante, stay with us for more day two coverage. Four days we'll be here in the hallway and lobby of Moscone North, stay with us.