>> Welcome to this IBM Watson Health Client Conversation. Here, we are probing the dynamics of the relationship between IBM and its key clients. We're looking back and we're also exploring the present situation. And we're going to talk about the future state of healthcare as well. My name is Dave Vellante from theCUBE and with me are Ryad Rondo who's the Associate director of Data Management at Veristat and Tim Watson, IBM Watson Health. Welcome gents. Tim, any relation? >> (chuckles) If I had a nickel for every time I was asked that question I'd be a wealthy man. >> Well, relationships and trust. I mean, they're pretty fundamental to any partnership and the pandemic certainly has tested us, and we've had to rely on those personal and professional relationships to get us through COVID. So let me start Ryad by asking you, how did the partnership with IBM support you last year? >> Last year as you know was particular year for our industry. So the relationship with our provider was key of the success of any studies we had last year with the new world we had. So we were working very close with IBM Clinical, and I think collaboration was key for successful (indistinct). >> So Tim, I wonder if you could talk about some of the things that you've done with Ryad and his team, maybe some of the things that you accomplished in 2020 anything that stands out. And then maybe take it from there and query Ryad on some of the more important topics that are top of mind for you. >> Yeah, absolutely. I think in 2020, we all know it was a challenging year but IBM actually put together a really good program to support our clients as as it relates to COVID-19 trials. And Veristat did a great job of taking advantage of that with a number of their clients that offered a free promotion for 18 months of a subscription to support individual sponsors in their efforts and trying to find a vaccine that supports the whole world out there. So I think we put together a program that Veristat and their clients were able to take advantage of as it relates to COVID. But in addition to that the platform supported their efforts to maintain the clinical trials that were ongoing. And that was actually probably even bigger challenge for the Veristat team. >> Yeah, so maybe do a little mock session here. Tim you're used to role-playing so let's do a little role-playing. So we're in 2021 you guys are sitting down, unfortunately you're not face-to-face, but imagine you were and Ryad, talk about the objectives that you have in 2021, as you think about your relationship with Watson Health and Tim I'd love for you to respond in real time as to how you're going to help Ryad. Ryad kick it off, what are you trying to get done in 2021? What's the priority? >> Let me take a step back from 2020 and I go to 2021. I think one of the biggest challenge we have in 2021 with studies we had, is the extreme rapid startup of several projects we had. So we needed to start, design and push studies live in the record time. We were able to design a study in a week, another study in two weeks from the protocol to the goal of the study. And all that was with the collaboration of IBM, of course. And 2020 actually brought a change of the approach the client have to the study. So now they are more willing to use more electronic solutions than before. 2020 forced our client and the industry in general to look at the solutions offered electronically by the ADC provider by IBM Clinical. So right now in 2021, we will be leveraging those solutions, I'm thinking about monitoring module, I'm thinking about ePRO, I'm thinking about eConsent which is coming soon and I'm thinking about visualization as well. So these solutions provided by the system are now more acceptable than before and we will be used in 2021. Visualization is in the top list of these solutions eConsent comes with it as well. >> Alright so Tim, how are you going to help? How's Watson Health going to be a great partner in 2021 and beyond? >> Well, I think IBM is continuing to focus on what do these solutions mean going forward and how can we extend the functionality of our platform out there? So with the release of eConsent that's something that I believe Veristat can take advantage of. And the near term is just a matter of getting the Veristat team educated on our eConsent functionality to be able to offer that out to their clients. And visualization is another area that we've had a number of discussions with Veristat on over the past 12 months and leveraging the tool set that we are able to bring to the table with smart reports and how that can provide additional value and then finding that balance where we can get them off the ground quickly maybe with some pre-packaged reports but also educating their team so that they're able to take that tool set and be able to extend that functionality to their clients. >> So Ryad what's the situation like? I wonder if you could think pre pandemic, post pandemic. A lot of clients that I talked to, they would talk the digital game, but in reality it's not that simple. You know things are done a certain way and then I've often called it the forced march to become digital. And that's kind of what happened to us. And so I'm curious as to your sense as to what the climate was like pre and post? How much if at all, I have to believe that everybody's digital strategies were compressed, but was it months? Was it years? And it was sort of overnight we had to make the changes. So it was like a Petri dish. You really didn't have time to plan, you just did. So by how much was that digital transformation compressed and what were the learnings and how do you see taking that forward? >> We historically would go to clients with solutions and you know human nature resists to change. So when we go offering electronic solutions before the pandemic, we always had to define, to use a lot of argument actually to explain to the client that this is the way to go. This is the time to do use more electronic solutions. With the pandemic, the fear forced the client to use these solutions. And they realized that it's working. They realize that we can do it. We can do it very well. Even for complex study, solutions are available and can be used. We also was forced somehow to shorten our timeline to find best way to push studies live in as short as possible timelines without jeopardizing the quality, finding solutions. Splitter is one of the solutions IBM can offer so this is one of the solution we used. The release of the ECRS done later in the edit. IBM offered the capability to do that without jeopardizing the quality. >> So Tim, maybe you could chime in here. I mean, that's really important point. We had sort of no choice but to rush into digital and electronic last year how do you help clients maintain that quality? Maybe you guys could talk amongst yourselves as to the kinds of things you did to maintain that both, when things were going crazy and they somewhat still are. And then how do you preserve that going forward maybe turn the dial maybe a little bit based on your learnings. >> I think one of the advantages that our platform brings to the table is the flexibility. And that flexibility is what Veristat was able to take advantage of in different situations, in different parts of the platform. So whether that was the ability to design trials very quickly and be very flexible with the rollout of that trial to address specific timelines or just the different areas of the platform like ePRO to be able to extend things out to their clients as well so that patients are entering data into the clinical trial so that they're not having to go visit sites necessarily out there. So there's a lot of things that we have within the platform that our clients are able to take advantage of that really came into full focus in the year of 2020. >> Does that resonate with you Ryad? Do you trust what Tim just said? Does it give you a good feeling based on your experience? How confident are you that IBM can deliver on that objective? >> Yeah, actually the pressure we received from the industry in general, in the last year and still this year is to always shorten with high quality deliverables. So we were able to use the flexibility IBM system offers to achieve that goal. Splitting release performing a complex MSU successfully. So all these features and the flexibility we have with the status and flexibility we have with the user roles all these features and flexibilities was key performing that high quality MSU complex updates and in the record time. >> I wonder if you could each talk about sort of personally and bring it professional if you like, but how have you changed as a result of the pandemic and how has it helped you position for what's coming ahead? Ryad maybe you you could start. >> One of the things I'd like to say about what happened last year is that, it has never been easier for me to explain what I do. Historically, when I asked what do they exactly? I had to spend hours explaining what I do. Now I tell them, do you know what's the phase three phase two we are all waiting for the vaccine? That's what we do. So that's I think the number one success of the year for me. Honestly, it's just proud to work in an industry like that. Proud to work in a company like Veristat who cares about the quality who cares about providing the safety of the patient using the best system in the market working with IBM Clinical in this case. We're working to achieve that goal. I think 2020 gave me that just another level of proud maybe, if I may say. Partnership with client, partnership with IBM offered free for COVID studies for an 18-month program. So all of these just confirm that we are, I am personally in the right place, right company, having the right partnership to help humanity actually get better. >> Thank you for that, that's great. And Tim, you too you're obviously part of that, but I wonder if you could comment. >> Well, I'd like to kind of echo just what Ryad had said that professionally I feel we play a very small part. The Veristat team, the Sponsors team they do all the hard work, trying to find new medications, new vaccines to bring to life. But it means a lot to play a small part in that process to offer a technology that helps them do that quicker. And if we can get those drugs and vaccines to market quicker then that's going to have a very positive impact on the world as a whole. So it's a very exciting time to be in this industry. >> Undoubtedly. Ryad what can IBM do to help you near-term, mid-term and long-term? What are some of the most important things that Tim and his company can help with? >> Yeah, as Tim mentioned, we will have eConsent coming soon. I'm sure IBM will have other electronic solution coming soon so partnership, support and training. So education between our two entities between Veristat and IBM to use this new feature now became key for the success of any study. So I expect partnership support on an education from them. We've been successfully doing that and hoping that we'll continue in this case. >> Tim, any comments on that? >> Well, we have a great relationship with Veristat and we try to have that same kind of relationship with all of our clients. We meet regularly with governance meetings. That's a great time to share new information, to revisit old questions that may still be out there. And so, we're going to continue to offer the additional training options to our clients to allow them to leverage that platform so that they can then cater to their clients, the sponsors that are contracting with them for their clinical trials. >> So Ryad, Tim did say lower the price. That's a good sign. What about that though? What about value for investment? How Ryad would you grade IBM's track record in that regard? If you had to put a grade on it, you know, A, B, C, D, E, F. >> I would put a good grade, actually. I think it's the right balance. You know, a client expect always to pay less, but we are the experts, we are doing the job. And we have to guide them, make the right decision. We tell them why they should pick that or that system. And what are they getting for the price they're paying for. Right now we didn't have much trouble selling IBM and there is something common I think so, is to revise the price to be more specific for study, I think will help the client actually. Will help selling the products. >> So if you had to put a letter grade on it what would you give them? >> A grade? (chuckles) >> A grade, come on A, B? >> An A. >> An A, you'd give them an A? >> Yes. Solid A, 4.0 that's great, Tim you got a-- >> A minus to just leave some space for improvement. (Tim chuckles) >> Okay A minus just because, hold the carrot out there right? That's good, it's okay. Tim, how do you feel about that? You don't mind having a little extra incentive, right? >> No, absolutely not. It's always great to work with the Veristat team and we have I think a great relationship and there are certainly opportunities that we can hopefully work together. And If the price needs to be addressed then we can address it and win the business. >> Tim, anything I missed? Anything that you feel like there's a gap there that you want to cover that I didn't touch on? >> No, I think we're good. >> Great, awesome. Well, great conversation guys. I really appreciate it and thanks for the good work that you guys are doing on behalf of everybody who's living through this. It's a critical time and it's amazing how your industry has responded so thank you for that. And thank you for spending some time with us. You're watching Client Conversations with IBM Watson Health.

>>Welcome to this IBM Watson health client conversation here. We're probing the dynamics of the relationship between IBM and its clients. And we're looking back, we're exploring the present situation and discussing the future state of healthcare. My name is Dave Volante from the cube and with me are Alex Dillard. Who's a senior director data analysis at blue choice, blue choice health plan, and Darryl decode, who is IBM with IBM Watson health. Of course. Welcome gentlemen. Good to see you. Thanks for coming on. >>Hey, >>So, you know, you think about lasting relationships. They're the foundation to any partnership and this past year, and it's tested all of us. We've had to rely on both personal and professional relationships to get us through the pandemic. So Alex, let me start with you. How has the partnership with IBM supported you in 2020? >>Well, uh, I've just a piece of a larger puzzle. Uh, the relationship that Darrell and I have had is confined to IBM Watson health, but blue cross blue shield, South Carolina, which food choice is a wholly owned subsidiary of has had a standing relationship with IBM on the it side. Uh, we are a mainframe shop, uh, about 70% of our it infrastructure is on a mainframe. And, uh, that puts us as a segment one client for IBM, we're in the top 300 of all of their clients in the Americans. And more specifically we're the fourth largest, um, uh, Linux on Z shop in the world. So, uh, we've got a lot of diversification at blue cross blue shield of South Carolina and the mainframe and the vastness of that. It infrastructure reflects that, uh, diversification. We are more than just a crossing the shield. Uh, that's typically what people think of is insurance when they think of crossing shield, but we also have a division that does a lot of subcontract work for government programs, uh, track air, which is the military healthcare, uh, claims processing and Medicare claims processing. >>We were a subcontractor of other folks that use our back office, it infrastructure to, to run their claims through. So that's, that's the larger, um, aspect of our relationship that, that blue cross blue shield of South Carolina house with IBM, uh, as it relates to Watson health, we have been a client since 1994 and obviously that predates the IBM proper. Uh, we were a client of med stat and then Truven, who then, uh, was bought by IBM. So we have used the products from Watson health throughout our system to support provider profiling, uh, count group reporting, um, and ad hoc analysis and to some extent to, uh, support our value-based products with, uh, ACO and PCMH, >>Uh, products. >>Awesome. Thank you for that. So Daryl is very long-term relationship. Obviously, if people forget sometimes that, uh, how IBM has modernized the Z Alex talked about, uh, Linux on the mainframe. That's pretty cool. I wonder if you could talk about specifically the things that, that you've done with Alex in his, in his, in his team, you know, thinking back last year, what were your accomplishments that you really stand out? >>Yeah, so, so one thing that jumps to mind is, uh, given the long standing relationship, I relied heavily on Alex to help us work through a multi-year renewal. And it was, it was a, um, a good adventure for us. We, we were able to laugh along the way. We certainly had some, some phone calls that, that were a little bit challenging, but the great thing about it was that the relationship that Alex and I had, he really views it as a partnership. And that was just so encouraging and uplifting. So to me, from my perspective, that was absolutely, uh, one of the highlights of my year and working through even through the pandemic and all that, we figured it out. >>So you guys, when you get together, go ahead, please. >>That's what I had as well. Um, you know, the, the unique thing about the Watson health contract is because it involves data. Uh, we take the stance that it's an it contract, so I'm on the business side. So I've got to just, as Daryl has to navigate it with me, we've got to navigate a large of your it bureaucracy. Um, and, uh, it, it was challenging. Um, you know, the business people kind of smooth the tracks and then you get the lawyers involved in, it just goes haywire. So, um, we were able to navigate that. Um, uh, so yeah, so it was a big accomplishment. So Alex, it's not real sexy to talk about, but we got it done >>Well. So Alex you're, you're in sales, so you're, you're used to role playing. So imagine you're, you're, you're sitting down, uh, sorry, Darryl. You're used to, role-playing out. Imagine you're sitting down with Alex and you're thinking about 20, 21 planning, so, you know, take it away. W what do you, what would you ask, what would you talk about or share with us? >>Yeah, yeah, absolutely. So, so I, I know that, you know, one of the key objectives is, uh, continued to ingest, engage with your members and you have key business strategies. I know you recently migrated over to a new PBM, and so there, there's some complexities that come with that. Um, but just, you know, Alex, if you don't mind, why don't you share a little bit about kind of your, your perspective on what 2020 would hold for you in your organization? Well, I think that due to the pandemic, we are, I personally kick the, can down the road on a couple of things, particularly >>Having a strategic roadmap discussion, um, you know, uh, I was going to get into this later, but I enjoy doing things face-to-face rather than, uh, over the phone or, or virtually. And so, uh, I guess I was a little too optimistic about maybe being able to get together late 2020 to have that strategic roadmap discussion. Um, I think, uh, given what has developed with, um, the pandemic and vaccines and stuff, I may, I may be able to get everybody on the same page later this year, hopefully. Uh, but certainly we want to have a strategic roadmap discussion. Um, we license, uh, Watson Hills, uh, cat group insights, uh, tool, which we use for employer group reporting. And we are currently in the beginning stages of rolling that out to our external clients, whether it's agents, brokers, um, those types of folks. And then it vanished we as our core product that we use for analysis, and that product is transitioning to what is called health insights. And so from an analytical standpoint, my staff and the staff of our cluster areas will need to sort of move to health insights since that's where it's going, uh, from an analytic standpoint. So we're going to work on that as well. Um, and then some more detailed things around database rebuilds and stuff like that. Those are all sort of on the roadmap for 2021. >>Yeah. So, you know, you talk about strategic planning and you think about the way planning used to be. I mean, sometimes you take a longer term horizon, maybe that's five years, you know, technology cycles, you know, even though they go very fast, but you see major technology shifts, they're like go through these seven year cycles, you see that in financial world. And then with the, with the pandemic, we're talking about seven day cycles, you know, how do I support people work from home? Do I open the store or not? You know, it's a day-to-day type of thing. So I wonder if you could each talk about personally and professionally w how, how is 2020, you know, changed you and maybe position you for, for what's ahead, maybe Alex, you could start, >>Well, you know, I'm an analyst, so I always fall back to the numbers. What are the numbers show us, um, you know, people can have four perceptions, but, uh, the numbers give us a reality. So the reality is that a year ago, pre pandemic, uh, just 13% of blue cross blue shield employees were working from home a hundred percent, uh fast-forward to today. And that number is now 87%. So think about, uh, just the lift from a it infrastructure to support that we almost, all of those people are using Citrix to get in to our network. Uh, we're using a remote desktop. So you've got this pipeline that probably had to go from, you know, this small, to huge, to get all this bandwidth, all this data and everything. So you've got that huge lift. Um, and then it affects different areas, um, differently. Uh, I don't have any first-line staff, any staff that are member facing, so I didn't really have to navigate, you know, how do these people talk to our member? >>How does staff talk to our members on the phone when they're at home, as opposed to in the office, and, you know, is there background noise, things like that. So I've got analysts, uh, they're just crunching numbers. Um, but my, my, my personal, uh, feeling was I like doing managing by walking around, you know, stopping and talking to other, working on. So that went away and I like face-to-face meetings, as I've mentioned, and that went away. So it was really a culture change for me personally, it was a culture change for our organization. Uh, and, and now we're having conversations with executive management that, you know, if you've got staff who have been doing a good job and they remain productive, you know, give me a reason they got to come back in, which is just, as you told me that I'm going to be the case a year ago, I would have been, you know, flabbergasted, but that's where we are right now. >>And so on a personal standpoint, you know, I went home for a little while and then came back. And so my wife also works for blue cross blue shield of South Carolina. Um, so, you know, she set up in the dining room working, uh, I have my own book in our living room working, and then we've got a great side, you know, the school is not in session, you know, in person. So he's doing virtual learning. So combine all those things, and you've got all kinds of crazy things that could happen. Uh, and then you've got staff who are in the same situation. Um, so it was a lot to handle. And the longer it goes on the novelty of working from home wears off, and you kind of realize, you know, I can't go do this. I can't go out to eat. I can't do all types of things that I used to do. And so that affects your mental health. So as, as a leader, um, of my small area, and then our executives really had to become more, uh, uh, in, in people's faces. So we've got, we've done a lot more video, uh, messages, a lot more emails. Um, I have been tasked with being very deliberate about checking on how everything is going at their house. Are they getting what they need? Um, you know, how are they feeling? Are they getting up and exercising, all those things that you took for granted, uh, beforehand. >>Yeah. So Daryl, anything you'd add to that in terms of specifically in terms of how you might, how you might change the way in which you interact with your clients generally, uh, an Alex specifically, Alex likes, face to face, you know, we can't wait. All right. >>Yeah, yeah. It's funny. We never quite got to do it Alex, but we were talking about doing a virtual happy hour at one point too, to just celebrate the success. Um, but for me, you know, typically I would travel and visit Alex face-to-face on maybe a monthly basis. And so it it's been really hard for me. I didn't realize how, how much I enjoyed that in-person interaction. And so that, that was something that I I've been, you know, working through and finding ways to, to still interact with people. And I'm certainly making, making the best of, of the video phone calls and, you know, that sort of thing. So, uh, just work working to maintain those relationships. >>I wonder if I could ask you when, when, when this thing, when we're through the pandemic, what do you expect the work from home percentage? I think I heard 13% prior to the pandemic, 87% today. What do you think is going to be post pandemic? >>That is a good question. Um, it, it may go back to maybe 60% at home. I think, I think there will be a simple majority, uh, working from home. Um, that's, that's from our planning, uh, space planning standpoint. That's, that's what we are, uh, what we're expecting, um, if, if production stays, um, at acceptable levels, um, >>Do you feel like productivity was negatively impacted positive? It will be impacted or it's kind of weird. >>Yeah. All the metrics that we track show that it was, it was sustained and in some areas even better. Uh, and if you really think about, um, sort of your typical day when you work from home, I found, uh, that I was logged on an hour earlier. That's probably what's happening with other staff as well is they're, they're motivated to get up and, and get online, uh, earlier. >>Yeah. Mostly tech leaders that I talk to share that sentiment, that the productivity is actually improved. So Darryl, I presume you see the same thing in your observation space. Yeah. >>Yeah. I, I do. And, and I have other clients too, and, and, and they are definitely looking at ways to continue to work remotely. I know that for a lot of people who are in the office all the time, uh, having a little bit more flexibility when you work from home can be a good thing. And, and like you said, you, you have to make sure that the productivity is still there and the productivity is up. Um, but I, I could see that the trend continuing absolutely >>I'd love for you to, to look at Darryl and say, and tell him what the kinds of things that IBM can do to help you both today, immediately 20, 21, and in the future and a Darryl, how, how your, how you'll respond. >>Well, I'll tell you that. Um, so in 2020, what, what changed most dramatically for us as a health plan? Uh, and, and I, it echoes what we see across the country is the gigantic shift in telehealth. Um, you know, if, if, again, if you look at the numbers, uh, our telehealth visits per thousand, so that's the number of visits per thousand members in a given month, went up 1472%. And so, you know, the common response to that is, well, you know, your visits overall probably, you know, were flat because, uh, you know, they just weren't happening in that. And that's not necessarily true for us. So if you look at visits overall, they written down four and a half percent. Um, so there was a shift, but it, it was not a big enough shift to account for, uh, visits overall sustaining the level that they were pre pandemic. >>Um, so as we look into 2021, uh, we will be investigating how we can maintain, uh, the, uh, the accessibility of our healthcare providers via telehealth. Um, you know, one of the projects that we started in 2020, uh, was based upon the choosing wisely campaign. So if you're not familiar with choosing wisely, it's a very well thought out process. It involves many, many provider specialties and its sole target is to reduce low value care. Uh, so we took it upon ourselves to Institute sort of a mirror of that plan or that program at, at blue cross here in South Carolina. And so as we moved to 2021, obviously those low value services just because of the pandemic were reduced, uh, and some of the high-value care was reduced as well. And so what we are going to try to do is bring back habit, bring back that high value care, but not bring back that low value care and so low value care or things like vitamin D testing. Uh, it can be other things like, um, uh, CT for head headaches, um, imaging for low abdominal pain, things like that. So, uh, we want to focus on low, uh, eliminating what value care, bringing back high value care, >>Okay, Dale, you're up? How are you going to help Alex achieve that? So, so good news is, is that we've got the analytic warehouse and the database where all of the data is captured. And so we we've got the treasure trove of information and data. And so what we'll do is we'll come alongside Alex and his team will do the analytics, we'll provide the analytic methods measures, and we'll also help him uncover where perhaps those individuals may be, who had postponed care, um, because of the pandemic. And so we can put together strategies to help make sure that they get the care that they need. Uh, I also a hundred percent agree that tele-health hopefully is something that will continue because I do think that that is a good way and efficient way to get care for people. Um, and, you know, as a, as, as a way to, to address some of their needs and, and in, in a safe way too. >>So, um, I, I look forward to working with Alec and his team over this coming year. I think there is, uh, knowing Alex and, and the partnership and his readiness to be a client reference for us. You know, those are all great, um, recognition of how he partners with us. And we really value and appreciate, uh, the relationship that we have with blue cross blue shield, South Carolina and, and blue choice. Excellent. Daryl's right. The, the, the database we use already has some of that low value care measures baked into it. And so throughout 2020, I've worked with our analytic consulting team. Uh, it's under Daryl too, to talk about what's on the product product roadmap for adding to the cadre of live low value care measures inside advantage suite. Uh, so that's something that we'll actively be, um, uh, discussing because certainly, you know, we're, we're obviously not the only client only health plan clients. So there may be other plans that have priorities that very different made very differently than ours. Uh, so we want to give them what we're studying, what we're interested in, so they can just add it in to all their other client feedback, uh, for advantage suites, roadmap. Excellent. >>Look, my last question, Alex is how would you grade IBM, if you had to take a bundle of sort of attributes, you know, uh, delivery, uh, value for service relationship, uh, et cetera, how would you grade the job that IBM is doing? >>I, the thing that I enjoy most about working with IBM and Darryl specifically, is that they're always challenging us to look at different things. Um, things that sometimes we hadn't considered, because obviously it may be an issue for another health plan client or an employer client that they've got. Uh, they tell us, this is what we're seeing. You know, you should look at it. Uh, a lot of times they do some of the foundational work in producing a report to show us what they're seeing in our data that is similar to what is in some of their other clients data. So that's refreshing to be, uh, challenged by IBM to look at things that we may not be, uh, looking at, uh, or maybe missing, because we've got our eye on the ball on something else you >>Care to put a letter grade on that. >>Oh, definitely. Definitely. Thank you. >>Well, Darryl, congratulations, that says a lot and, uh, we have to leave it there and one at a time, but, but Daryl, anything that I didn't ask Alex, that you, you wanted me to, >>So, um, Alex re able to keep your tennis game up during the pandemic? Uh, I, yes, I tried as, as often as my wife would let me good. I would play every time I was asked, but, uh, yeah, so I, I did have to temper it a little bit, although when you spend all day with her and, and my son, you know, she may be a little more, uh, lenient on letting me leave the house. Well, maybe she's >>Yeah. The tribute to the late great comedian Mitch Hedberg, who says, uh, you know, when I, I played tennis, I played against the wall walls. Really good, hard to beat if it's pandemic appropriate. >>Oh, that's good. That's a true statement. And there was a lot of that going on, a lot of that play and playing against the wall. >>Hey, thanks so much, stay safe and really appreciate the time. Thank you. >>Thank you. Thank you. You're >>Really welcome. It was a great conversation and thank you for watching and spending some time with client conversations with IBM Watson health.

>> Welcome to this IBM Watson Health client conversation. And we're probing the dynamics of the relationship between IBM and it's clients. We're going to look back at some of the challenges of 2020 and look forward to, you know, present year's priorities. We'll also touch on the future state of healthcare. My name is Dave Vellante. I'll be your host and I'm from theCUBE. And with me are Doctor Alex Towbin, who's Associate Chief Clinical Operations and Informatics at Cincinnati ChilDoctoren's Hospital and John Chrisman of course from IBM Watson health. Welcome gentlemen, Good to see you. Thanks for coming on. >> Thanks for having us. >> Yeah, thanks for having me. >> Yeah I know from talking to many clients around the world, of course virtually this past year, 11 months or so that relationships with technology partners they've been critical over during the pandemic to really help folks get through that. Not that we're through it yet but, we're still through the year now, there's I'm talking professionally and personally and Doctor Towbin, I wonder if you could please talk about 2020 and what role the IBM partnership played in helping Cincinnati children's, you know press on in the face of incredible challenges? >> Yeah, I think our story of 2020 really starts before the pandemic and we were fortunate to be able to plan a disaster and do disaster drill scenarios. And so, as we were going through those disaster drill scenarios, we were trying to build a solution that would enable us to be able to work if all of our systems were down and we worked with IBM Watson Health to design that solution to implement it, it involves using other solutions from our primary one. And we performed that disaster drill in the late January, early February timeframe of 2020. And while that drill had nothing to do with COVID it got us thinking about how to deal with a disaster, how to prepare for a disaster. And so we've just completed that and COVID was coming on the horizon. I'm starting to hear about it coming into the U.S for the first time. And we took that very seriously on our department. And so, because we had prepared for this this disaster drill had gone through the entire exercise and we built out different scenarios for what could happen with COVID what would be our worst case scenarios and how we would deal with them. And so we were able to then bring that to quickly down to two options on how our department and our hospital would handle COVID and deal with that within the radiology department and like many other sites that becomes options of working from home or working in a isolated way and an and an office scenario like where I'm sitting now and we planned out both scenarios and eventually made the decision. Our decision at that point was to work in our offices. We're fortunate to have private offices where we can retreat to and something like that. And so then our relationship with IBM was helpful and that we needed to secure more pieces of hardware. And so even though IBM is our PACS vendor and our enterprise imaging vendor, they also help us to secure the high resolution monitors that are needed. And we needed a large influx of those during the pandemic and IBM was able to help us to get those. >> Wow! So yeah you were able to sort of test your organization resilience before the pandemic. I mean, John, that's quite an accomplishment for last year. I'm sure there are many others. I wonder if one of you could pick it up from here and bring your perspectives into it and, you know maybe ask any questions that you would like to ask them. >> Yeah, sure, Doctor Towbin, that's great that we were able to help you with the hardware and procure things. So I'm just curious before the pandemic how many of the radiologists ever got to read from home, was that a luxury back then? And then post pandemic, are you guys going to shift to how many are on-site versus remote? >> Yeah, so we have a couple of scenarios. We've had talk about it both from our PACS perspective as well as from our VNA enterprise imaging perspective from PACS perspective we always designed our solution to be able to work from a home machine. Our machines, people would access that through a hospital-based VPN. So they would log in directly to VPN and then access the PACS that way. And that worked well. And many of our radiologists do that particularly when they're on call works best for our neuroradiologist who are on call a little bit more frequently. And so they do read from home in that scenario. With enterprise imaging and are used to the enterprise viewer and iConnect access. We always wanted that solution to work over the internet. And so it's set up securely through the internet but not through the VPN. And we have radiologists use that as a way to view studies from home, even not from home, so it can be over one of their mobile devices, such as an iPad and could be at least reviewing studies then. We, for the most part for our radiologist in the hospital that's why we made the decision to stay in the hospital. At COVID time, we have such a strong teaching mission in our department in such a commitment to the education of our trainees. We think that hospital being in the hospital is our best way to do that, it's so hard. We find to do it over something like zoom or other sharing screen-sharing technology. So we've stayed in and I think we'll continue to stay in. There will be some of those needs from a call perspective for example, reading from home, and that will continue. >> And then what's your success been with this with the technology and the efficiency of reading from home? Do you feel like you're just as efficient when you're at home versus onsite? >> The technology is okay. The, our challenges when we're reading from the PACS which is the preferred way to do it rather than the enterprise archive, the challenge is we have to use the PACS So we have to be connected through VPN which limits our bandwidth and that makes it a little bit slower to read. And also the dictation software is a little bit slower when we're doing it. So moving study to study that rapid turnover doesn't happen but we have other ways to make, to accelerate the workflow. We cashed studies through the worklist. So they're on the machine, they load a little bit more rapidly and that works pretty well. So not quite as fast, but not terrible. >> We appreciate your partnership. I know it's been going on 10 years. I think you guys have a policy that you have to look at the market again every 10 years. So what do you think of how the market's changed and how we've evolved with the VNA and with the zero footprint viewers? A lot of that wasn't available when you initially signed up with Amicas years ago, so. >> Yeah, we signed up so we've been on this platform and then, you know now the IBM family starting in 2010, so it's now now 11 years that we're, we've been on as this version of the PACS and about eight, seven or eight years from the iConnect platform. And through that, we've seen quite an evolution. We were one of the first Amicas clients to be on version six and one of the largest enterprises. And that went from, we had trouble at the launch of that product. We've worked very closely with Amicas then to merge. And now IBM from the development side, as well as the support side to have really what we think is a great product that works very well for us and drives our entire workflow all the operations of our department. And so we've really relished that relationship with now IBM. And it's been a very good one, and it's allowed us to do the things like having disaster drill planning that we talked about earlier as far as where I see the market I think PACS in particular is on the verge of the 3.0 version as a marketplace. So PACSS 1 one was about building the packs, I think, and and having electronic imaging digital imaging, PACS 2.0 is more of web-based technology, getting it out of those private networks within a radiology department. And so giving a little bit more to the masses and 3.0 is going to be more about incorporating machine learning. I really see that as the way the market's going to go and to where I think we're at the infancy of that part of the market now about how do you bring books in for machine learning algorithms to help to drive workflow or to drive some image interpretation or analysis, as far as enterprise imaging, we're on the cusp of a lot there as well. So we've been really driving deep with enterprise imaging leading nationally enterprise imaging and I have a role in the MSAM Enterprise Imaging Community. And through all of that work we've been trying to tackle works well from enterprise imaging point of view the challenges are outside of radiology, outside of cardiology and the places where we're trying to deal with medical photos, the photographs taken with a smart device or a digital camera of another type, and trying to have workflow that makes sense for providers not in those specialty to that don't have tools like a DICOM modality workloads store these giant million-dollar MRI scanners that do all the work for you, but dealing with off the shelf, consumer electronics. So making sure the workflow works for them, trying to tie reports in trying to standardize the language around it, so how do we tag photos correctly so that we can identify relevancy all of those things we're working through and are not yet standard within our, within the industry. And so we're doing a lot there and trying and seeing the products in the marketplace continuing to evolve around that on the viewer side, there's really been a big emergence as you mentioned about the zero footprint viewers or the enterprise viewer, allowing easy access easy viewing of images throughout the enterprise of all types of imaging through obtained in the enterprise and will eventually incorporate video pathology. The market is also trying to figure out if there can be one type of viewer that does them all that and so that type of universal viewer, a viewer that cardiologists can use the same as a radiologist the same as a dermatologist, same as a pathologist we're all I think a long way away from that. But that's the Marcus trying to figure those two things out. >> Yeah, I agree with you. I agree with your assessment. You talked about the non DICOM areas, and I know you've you've partnered with us, with ImageMover and you've got some mobile device capture taking place. And you're looking to expand that more to the enterprise. Are you also starting to use the XDS registry? That's part of the iConnect enterprise archive, or as well as wrapping things in DICOM, or are you going to stick with just wrapping things in DICOM? >> Yeah, so far we've been very bunched pro DICOM and using that throughout the enterprise. And we've always thought, or maybe we've evolved to think that there is going to be a role for XDS are I think our early concerns with XDS are the lack of other institutions using it. And so, even though it's designed for portability if no one else reads it, it's not portable. If no one else is using that. But as we move more and more into other specialties things like dermatology, ophthalmology, some of the labeling that's needed in those images and the uses, the secondary uses of those images for education, for publication, for dermatology workflow or ophthalmology workflow, needs to get back to that native file and the DICOM wrap may not make sense for them. And so we've been actively talking about switching towards XDS for some of the non DICOM, such as dermatology. We've not yet done that though. >> Given the era children's hospital has the impact on your patient load, then similar to what regular adult hospitals are, or have you guys had a pretty steady number of studies over the last year? >> In relay through the pandemic, we've had, it has been decreased, but children fortunately have not been as severely affected as adults. There is definitely disease in children and we see a fair amount of that. There are some unique things that happen in kids but that fortunately rare. So there's this severe inflammatory response that kids can get and can cause them to get very sick but it is quite rare. Our volumes are, I think I'm not I think our volumes are stable and our advanced imaging things like CT, MRI, nuclear medicine, they're really most decreased in radiography. And we see some weird patterns, inpatient volumes are relatively stable. So our single view chest x-rays, for example, have been stable. ER, visits are way down because people are either wearing masks, isolating or not wanting to come to the ER. So they're not getting sick with things like the flu or or even common colds or pneumonias. And so they're not coming into the ER as much. So our two view x-rays have dropped by like 30%. And so we were looking at this just yesterday. If you follow the graphs for the two we saw a dip of both around March, but essentially the one view chest were a straight line and the two view chest were a straight line and in March dropped 30 to 50% and then stayed at that lower level. Other x-rays are on the, stay at that low level side. >> Thanks, I know in 2021 we've got a big upgrade coming with you guys soon and you're going to stay in our standalone mode. I understand what the PACSS and not integrate deeply to the VNA. And so you'll have a couple more layers of storage there but can you talk about your excitement about going to 8.1 and what you're looking forward to based on your testimony. >> Yeah we're actually in, we're upgrading as we're talking which is interesting, but it's a good time for talking. I'm not doing that part of the work. And so our testing has worked well. I think we're, we are excited. We, you know, we've been on the product as I mentioned for over 10 years now. And for many of those years we were among the first, at each version. Now we're way behind. And we want to get back up to the latest and greatest and we want to stay cutting edge. There've been a lot of reasons why we haven't moved up to that level, but we do. We're very careful in our testing and we needed a version that would work for us. And there were things about previous versions that just didn't and as you mentioned, we're staying in that standalone mode. We very much want to be on the integrated mode in our future because enterprise imaging is so important and understanding how the comparisons fit in with the comparison in dermatology or chest wall deformity clinic, or other areas how those fit into the radiology story is important and it helped me as a radiologist be a better radiologist to see all those other pictures. So I want them there but we have to have the workflow, right. And so that's the part that we're still working towards and making sure that that fits so we will get there. It'll probably be in the next year or two to get to that immigrating mode. >> As you, look at the number of vendors you have I think you guys prefer to have less vendor partners than than more I know in the cardiology area you guys do some cardiology work. What has been the history or any, any look to the future of that related to enterprise imaging? Do you look to incorporate more of that into a singular solution? >> Cardiology is entirely part of our enterprise imaging solution. We all the cardiology amendments go to our vendor neutral archive on the iConnect platform. All of them are viewed across the enterprise using our enterprise viewer. They have their unique specialty viewer which is, you know, fine. I'm a believer that specialty, different specialties, deserve to have their specialty viewers to do theirs specialty reads. And at this point I don't think the universal viewer works or makes sense until we have that. And so all the cardiology images are there. They're all of our historical cardiology images are migrated and part of our enterprise solution. So they're part of the entire reference the challenge is they're just not all in PACSS. And so that's where, you know, an example, great example, why we need to get to this to the integrated mode to be able to see those. And the reason we didn't do that is the cardiology archive is so large to add a storage to the PACS archive. Didn't make sense if we knew we were going to be in an integrated mode eventually, and we didn't want to double our PACS storage and then get rid of it a couple of years later. >> So once you're on a new version of merge PACS and you're beyond this, what are your other goals in 2021? Are you looking to bring AI in? Are you using anybody else's AI currently? >> Yeah, we do have AI clinical it's phone age, so it's not not a ton of things but we've been using it clinically, fully integrated, it launches. When I open a study, when I opened a bone age study impacts it launches we have a bone age calculator as well that we've been using for almost two decades now. And so that we have to use that still but launching that automatically includes the patient's sex and birth date, which are keys for determining bone age, and all that information is there automatically. But at the same time, the images are sent to the machine learning algorithm. And in the background the machine determines a bone age that in the background it sends it straight to our dictation system and it's there when we opened the study. And so if I agree with that I signed the report and we're done. If I disagree, I copy it from my calculator and put it in until it takes just a couple of clicks. We are working on expanding. We've done a lot of research in artificial intelligence and the department. And so we've been things are sort of in the middle of translation of moving it from the research pure research realm to the clinical realm, something we're actively working on trying to get them in. Others are a little bit more difficult. >> That's the question on that John, Doctor, when you talk about injecting, you know machine intelligence into the equation. >> Yeah. >> What, how do you sort of value that? Does that give you automation? Does it improve your quality? Does it speed the outcome and maybe it's all of those but how do you sort of evaluate the impact to your organisation? >> I there's a lot of ways you can do it. And you touched on one of my favorite one of my favorite talking points, in a lot of what we've been doing and early machine learning is around image interpretation helping me as a radiologist to see a finding. Unfortunately, most of the things are fairly simple tasks that it's asking us to do. Like, is there a broken bone? Yes or no, I'm not trying to sound self-congratulatory or anything, but I'm really good at finding broken bones. I get, I've been doing it for a long time and, and radio, you know so machines doing that, they're going to perform as well as I can perform, you know, and that's the goal. Maybe they'll perform a little bit better maybe a little bit worse but we're talking tiny increments there they're really to me, not much value of that it's not something I would want. I don't value that at a time where I think machine learning can have real value around more on some of the things that you mentioned. So can it make me more efficient? Can it do the things that are so annoying that and they'd take, they're so tedious that they make me unhappy. A lot of little measurements for example are like that an example. So in a patient with cancer, we measure a little tumors everywhere and that's really important for their care, but it's tedious and so if a machine could do that in an automated way and I checked it that, you know, patient when because he or she can get that good quality care and I have a, you know, a workflow efficiency game. So that one's important. Another one that would be important is if the machine can see things I can't see. So I'm really good at finding fractures. I'm not really good at understanding what all the pixels mean and, you know in that same patient with cancer, oh what do all the pixels mean in that tumor? I know it's a tumor. I can see the tumor, I can say it's a tumor but sometimes those pixels have a lot of information in them and may give us prognosis, you know, say that this patient may, maybe this patient will do well with this specific type of chemotherapy or a specific or has a better prognosis with one with one drug compared to another. Those are things that we can't usually pick out. You know, it's beyond the level of that are I can perceive that one is really the cutting edge of machine learning. We're not there yet and then the other thing are things that, you know just the behind the scenes stuff that I don't necessarily need to be doing, or, you know so it's the non interpretive artificial intelligence. >> Dave: Right. >> And that's what I've been also trying to push. So an example of when the algorithms that we've been developing here we check airways. And this is a little bit historical in our department, but we want to make sure we're not missing a severe airway infection. That can be deadly, it's incredibly rare. Vaccines have made it go away completely but we still check airways. And so what happens is the technologist takes the x-ray. They come in to ask us if it's okay, we are interrupted from what we're doing. We open up the study, say yes or no. Okay, not okay, if it's not okay they go back, take another study. Then come back to us again and say, is it okay or not? And we repeat this a couple of times it takes them time that they don't need to spend and takes us time. And so we have, we've built an algorithm where the machine can check that and their machine is as good or a little bit worse than us, but give can give that feedback. >> Dave: Got it. >> The challenge is getting that feedback to the technologist quickly. And so that's, that's I think part for us to work on stuff. >> Thank you for that. So, John, we've probably got three or four minutes left. I'll let you bring it home and appreciate that Doctor Towbin >> I think one of the biggest impacts probably I knew this last year with the pandemic, Doctor Towbin is this, I know you're a big foodie. So having been to some good restaurants and dinners with the hot nurse in a house how's the pandemic affected you personally. And some of the things you like to do outside of work. >> Everything is shut down. And everything has changed. I have not left the house since March besides come to work and my family hasn't either. And so we're hardcore quarantining and staying you know, staying out and keeping it home. So we've not gone out to dinner or done much else. >> So its DoorDash and Uber Eats or just learned to cook at home. >> It's all cooking at home. We're fortunate, my wife loves to cook. My kids love to cook. I enjoy cooking, but I don't have the time as often. So we've done a lot of different are on our own experimenting. Maybe when the silver lining one of the things I've really relished about all this is all this time I get to spend with my family. And that closeness that we've been able to achieve because of being confined in our house the whole time. And so I've played get to play video games with my kids every night. We'd been on a big Fortnite Keck lately since it's been down making. So we've been playing that every night since we've watched movies a lot. And so as a family, we've, I it's something I'll look back fondly even though it's been a very difficult time but it's been an enjoyable time. >> I agree, I've enjoyed more family time this year as well, but final question is in 2021, beyond the PACS upgrade what are the top other two projects that you want to accomplish with us this year? And how can we help you? >> I think our big one is are the big projects are unexpanded enterprise imaging. And so we want to continue rolling out to other areas that will include eventually incorporating scopes, all the images from the operating room. We need to be able to get into pathology. I think the pathology is really going to be a long game. Unfortunately, I've been saying that already for 10 years and it's still probably another 10 years ago but we need to go. We can start with the gross pathology images all the pictures that we take for tumor boards and get those in before we start talking about whole slide scanning and getting in more of the more of the photographs in the institution. So we have a route ambulatory but we need inpatient and ER. >> All right one last question. What can IBM do to be a better partner for you guys? >> I think it's keep listening keep listening and keep innovating. And don't be afraid to be that innovative partner sort of thinking as the small company that startup, rather than the giant bohemoth that can sometimes happen with large companies, it's harder. It is fear to turn quickly, but being a nimble company and making quick decisions, quick innovations. >> Great, quick question. How would you grade IBM, your a tough grader? >> It depends on what I am a tough grader but it depends on what, you know as the overall corporate partnership? >> Yeah the relationship. >> I'd say it's A minus. >> Its pretty good. >> I think, I mean, I, we get a lot of love from IBM. I'm talking specifically in the imaging space. I not, maybe not, I don't know as much on the hardware side but we, yeah, we have a really good relationship. We feel like we're listened to and we're valued. >> All right, well guys, thanks so much. >> So even if it's not an A plus- >> Go ahead. >> I think there's some more to, you know, from the to keep innovating side there's little things that we just let you know we've been asking for that we don't always get but understand the company has to make business decisions not decisions on what's best for me. >> Of course got to hold that carrot out too. Well thanks guys, really appreciate your time. Great conversation. >> Yeah, thank you. >> All right and thank you for spending some time with us. You're watching client conversations with IBM Watson Health.

>> Welcome to this IBM Watson Health "Client Conversation." We're probing the dynamics of the relationships between IBM and its clients. And we're going to look back, we're going to explore the present situation and we're going to discuss the future state of healthcare. My name is Dave Vellante from theCUBE and with me are Dr. David Huelsman, who is a radiologist at TriHealth, which is a provider of healthcare in hospitals and John Kritzman who is with of course IBM Watson Health. Gentlemen welcome. Thanks so much for coming on. >> Thank you. >> Yeah, thanks for having us. >> Doctor let me say you're welcome. Let me start with you. As an analyst and a TV host in the tech industry, we often focus so much on the shiny new toy, the new widget, the new software. But when I talk to practitioners, almost to a person, they tell me that the relationship and trust are probably the most important elements of their success, in terms of a vendor relationship. And over the last year, we've relied on both personal and professional relationships to get us through some of the most challenging times any of us have ever seen. So, Dr. Huelsman, let me ask you, and thinking about the challenges you faced in 2020, what does partnership mean to you and how would you describe the relationship with IBM? >> Well, it is exactly the reason why when we started our journey on this enterprise imaging project at TriHealth, that we very early on made the decision We only wanted one vendor. We didn't want to do it piecemeal, like say get a vendor neutral archive from one organization, and the radiology viewer from another. We wanted to partner with the chosen vendor and develop that long-term relationship, where we learn from each other and we mutually benefit each other, in sort of not just have a transactional relationship, but that we share the same values. We share the same vision. And that's what stood out to us is Watson Health imagings vision, mirrored TriHealth's in what we were trying to achieve with our enterprise imaging project. >> You know, let me follow up with that if I could. A lot of times you hear the phrase, "Single throat to choke" and it's kind of a pejorative, right? It's a really negative term. And the way you just described that Dr. Huelsman is you were looking for a partnership. Yeah, sure. Maybe it was more manageable and maybe it was a sort of Singletree, but it was really about the partnership, going forward in a shared vision and really shared ownership of the outcome. Is that a fair characterization? >> Yeah, how about more positive is "One hand to shake." >> Wow, yeah, I love it. (chuckles) One hand to shake. I'm going to steal that line. That's good. I like it. Keep it positive. Okay, John, when you think about the past 12 months and I know you have history with TriHealth, and more recently have rejoined the account, but how would you kind of characterize that relationship and particularly anything you can add about the challenges of the 2020? What stands out to you? >> Yeah, I think going back to your one hand to shake or one vendor to hug all that's not allowed during COVID, but we're excited to be back working with you, I am in particular. And at the beginning of this sales process and RFP when you guys were looking for that vendor partner, we did talk to you about the journey, the journey with AI that we already had mature products on the vendor neutral archive side and all the product pieces that you were looking for. And I know you've recently went live over the last year and you've been working through, crawling through and learning to walk and starting to run, hopefully. And at some point we'll get to the end of the marathon, where you'll have all the AI pieces that you're looking for. But this journey has been eyeopening for all of us, from using consultants in the beginning, to developing different team members to help make you successful. So I think I've been tracking this from the outside looking in, and I'm happy to be back, more working direct with you this year to help ensure your longterm success. >> Yeah, that's great John. You have some history there. I'm going to probe that a little bit. So doctor, you talked about this enterprise imaging project. I presume that's part of, that's one of the vectors of this journey that you're on. What are you trying to accomplish in the sort of near term and midterm in 2021? John mentioned AI, is there a data element to this? Are there other, maybe more important pressing things? What are your main goals for 2021? >> Sure. Well, where we are, where we've started, the first step was getting all of our imaging stored consistently in the same place and in the same way. We had like many health system, as you grow, you acquire facilities, you acquire physician practices and they all have their own small packs system, different ways of storing the data. And so it becomes very unwieldy to be a large organization and try to provide a consistent manner of your physicians interacting with the data, with the imaging in the same way. And so it was a very large dissatisfier in our EMR to, oh if you wanted to see cardiovascular imaging, it's this tab. If you wanted to see radiology, it was this tab. If you wanted to see that, oh you got to go to the media tab. And so our big goal is, okay, let's get the enterprise archive. And so the Watson enterprise archive is to get all of our imaging stored in the same place, in the same way. And so that then our referring physicians and now with our patients as well, that you can view all the imaging, access it the same way and have the same tools. And so that's the initial step. And we're not even complete with that first step, that's where COVID and sort of diverting resources, but it's there, it's that foundation, it's there. And so currently we have the radiology, cardiology, orthopedics and just recently OB-GYN, all of those departments have their images stored on our Watson Enterprise Archive. So the ultimate goal was then any imaging, including not just what you typically think of radiology, but endoscopy and arthroscopy and those sort of images, or wound care images, in that any image, any picture in our organization will be stored on the archive. So that then when we have everything on that archive, it's easier to access consistently with the same tools. But it's also one of the large pieces of partnering with with Watson Health Imaging, is the whole cognitive solutions and AI piece. Is that, well now we're storing all the data in a consistent manner, you can access it in a consistent manner, well then we hope to analyze it in a consistent manner and to use machine learning, and the various protocols and algorithms that Watson Health Imaging develops, to employ those and to provide better care. >> Excellent, thank you for that. John, I wonder if you could add to that? I mean, you've probably heard this story before from other clients, as well as TriHealth, I call it EMR chaos. What can you add to this conversation? I'm particularly interested in what IBM Watson Health brings to the table. >> Sure, we've continued to work with TriHealth. And like we said earlier, you do have to walk before you can run. So a lot of this solution being put in place, was getting that archive stood up and getting all the images transferred out of the legacy systems. And I think that we're nearly done with that process. Doing some find audits, able to turn off some of the legacy systems. So the data is there for the easier to do modalities first, the radiology, the cardiology, the OB, as Dr. Huelsman mentioned and the ortho. And now it's really getting to the exciting point of really optimizing everything and then starting to bring in other ologies from the health system, trying to get everything in that single EMR view. So there was a lot of activity going on last year with optimizing the system, trying to fine tune hanging protocols, make the workflow for everybody, so that the systems are efficient. And I think we will continue on that road this year. We'll continue down further with other pieces of the solution that were not implemented yet. So there's some deeper image sharing pieces that are available. There are some pieces with mobile device image capture and video capture that can be deployed. So we look forward to working in 2021 on some of those areas, as well as the increased AI solutions. >> So Dr. Huelsman I wonder if you could double click on that. I mean if you're talking to IBM, what are the priorities that you have? What do you, what do you really need from Watson Health to get there? >> So I spoke with Daniel early last week, and sort of described it as now we have the foundation, we sort of have the skeleton and now it's time to put meat on the bones. And so what we're excited about is the upcoming patient synopsis would be the first piece of AI cognitive solutions that Watson Health Imaging provides. And it's sort of that partnership of we're not expecting it to be perfect, but is it better than we have today? There is no perfect solution, but does it improve our current workflow? And so we'll be very interested of when we go live with patients synopsis of does this help? Is this better than what we have today? And the focus then becomes partnering with Watson Health Imaging is how do we make it better for ourselves? How do we make it better for you? I think we're a large health organization and typically we're not an academic or heavy research institution, but we take care of a lot of patients. And if we can work together, I think we'll find solutions. It's really that triple aim of how to provide better care, at cheaper costs, with a better experience. And that's what we're all after. And what's your version of patient, the current version of patients synopsis, and okay does it work for us? Well, even if it does, how do we make it better? Or if it doesn't, how do we make it work? And I think if we work together, make it work for TriHealth, you can make it work at all your community-based health organizations. >> Yeah. So, John that brings me to, Dr. Huelsman mentioned a couple of things in terms of the outcomes. Lower costs, better patient experience, et cetera. I mean, generally for clients, how do you measure success? And then specifically with regard to TriHealth, what's that like? What's that part of the partnership? >> Yes, specifically with TriHealth, the measure of success will be when Dr. Huelsman is able to call and be a super reference for us, and have these tools working to his satisfaction. And when he's been able to give us great input from the customer side, to help improve the science side of it. So today he's able to launch his epic EMR in context and he has to dig through the data, looking for those valuable nuggets and with using natural language processing, when he has patients synopsis, that will all be done for him. He'll be able to pull up the study, a CT of the head for instance and he'll be able to get those nuggets of information using natural language processing that Watson services and get the valuable insights without spending five or 10 minutes interrogating the EMR. So we look forward to those benefits for him, from the data analytics side, but then we also look forward to in the future, delivering other AI for the imaging side, to help him find the slices of interest and the defects that are in that particular study. So whether that's with our partner AI solutions or as we bring care advisers to market. So we look forward to his input on those also. >> Can you comment on that Dr. Huelsman? I would imagine that you would be really looking forward to that vision that John just laid out, as well as other practitioners in your organization. Maybe you could talk about that, is that sort of within your reach? What can you tell us? >> Well, absolutely. That was sort of the shared vision and relationship that we hope for and sort of have that shared outlook is we have all this data, how do we analyze it to improve, provide better care cheaper? And there's no way to do that without you harnessing technology. And IBM has been on the cutting edge of technology for my lifetime. And so it's very exciting to have a partnership with WHI and IBM. There's a history, there's a depth. And so how do we work together to advance, because we want the same things. What impressed me was sure, radiology and AI has been in the news and been hyped and some think over-hyped, and what have you. Everyone's after that Holy grail. But it's that sense of you have the engineers that you talk to, but there is an understanding that don't design the system for the engineers, design it for the end user. Design it for the radiologist. Talk to the end user, because it can be the greatest tool in the world, but I can tell you as a radiologist, if it interrupts my workflow, if it interrupts my search pattern for looking at images, it doesn't help me and radiologists won't use it. And so just having a great algorithm won't help. It is how do you present it to the end user? How do I access it? How can I easily toggle on and off, or do I have to minimize and maximize, and log into a different system. We talked earlier is one throat to choke, or one vendor to hug, we only want one interface. Radiologists and users just want to look at their... They have the radiology viewer, they have their PACS, we look at it all day and you don't want to minimize that and bring up something else, you want to keep interacting with what you're used to. And the mouse buttons do the same thing, it's a mouse click away. And that's what the people at Watson Health Imaging that we've interacted with, they get it. They understand that's what a radiologist would want. They want to continue interacting with their PACS, not with a third vendor or another program or something else. >> I love that. That ton of outside in thinking, starting with the radiologist, back to the engineer, not the reverse. I think that's something that IBM, and I've been watching IBM for a long time, it's something that IBM has brought to the table with its deep industry expertise. I maybe have some other questions, but John I wanted to give you an opportunity. Is there anything that you would like to ask Dr. Huelsman that maybe I haven't touched on yet? >> Yeah. Being back on your account this year, what do you see as a success? What would you count as a success at the end of 2021, if we can deliver this year for you? >> The success would be say, at the end of the year, we've got the heavy hitters, all stored on the archive. Do we pick up all the little, we've got the low hanging fruit, now can we go after the line placement imaging and the arthroscopy and dioscomy, and all those smaller volume in pickups, that we truly get all of our imaging stored on that archive. And then the even larger piece is then do we start using the data on the archive with some cognitive solution? I would love to successfully implement, whether it's patient synopsis or one of the care advisors, that we start using sort of the analytics, the machine learning, some AI component that we successfully implement and maybe share good ideas with you. And sure we intend to go live with patient synopsis next month. I would love it by the end of the year, if the version that we're using patients synopsis and we find it helpful. And the version we use is better than what we went live with next month, because of feedback that we're able to give you. >> Great we looked forward to working with you on that. I guess, personally, with the pandemic in 2020, what have you become, I guess in 2020 that maybe you weren't a year ago before the pandemic, just out of curiosity? >> I'm not sure if we're anything different. A mantra that we've used in the department of radiology at TriHealth for a decade, "Improved service become more adaptable." And we're a service industry, so of course we want to improve service, but be adaptable, become more adaptable. And COVID certainly emphasize that need to be adaptable, to be flexible and the better tools we have. It was great early in the COVID when we had the shutdowns, we found ourselves, we have way more radiologists than we had studies that needed interpreted. So we were flexible all often and be home more. Well, the referring physicians don't know like, well is Dr. Huelsman working today? We don't expect them to look up our schedules. If I get a page that, Hey, can you take a look at this? It was great that at that time I didn't have a home workstation, but I had iConnect access. Before there was no way for me to access the images without getting on a VPN and logging on, it takes 10, 15 minutes before I'm able. Instead I could answer the phone, and I'm not going to say, "Oh, I'm sorry, I'm not at the hospital day, call this number someone else will help you." I have my iPad, go to ica.trihealth.com logged on, I'm looking at the images two minutes later. And so the ease of use, the flexibility, it helped us become adaptable. And I anticipate with we're upgrading the radiology viewer and the iConnect access next month as well, to try to educate our referring physicians, of sort of the image sharing capabilities within that next version of our viewer. Because telehealth has become like everywhere else. It's become much more important at TriHealth during this pandemic. And I think it will be a very big satisfier for both referring physicians and patients, that those image sharing capabilities, to be able to look at the same image, see the annotation that either the radiologist or the referring physician, oncologist, whoever is wanting to share images with the patient and the patient's family, to have multiple parties on at the same time. It will be very good. >> With the new tools that you have for working from home with your full workstation, are you as efficient reading at home? >> Yes. >> And having full access to the PACS as in-house? >> Absolutely. >> That's great to hear. Have you been able to take advantage of using any of the collaboration tools within iConnect, to collaborate with a referring physician, where he can see your pointer and you can see his, or is that something we need to get working? >> Hopefully if you ask me that a year from now, the answer will be yes. >> So does that exit a radiologist? Does that help a radiologist communicate with a referring physician? Or do you feel that that's going to be a- >> Absolutely. We still have our old school physicians that we love who come to the reading room, who come to the department of radiology and go over studies together. But it's dwindling, it's becoming fewer and fewer as certain individuals retire. And it's just different. But the more direct interaction we can have with referring physicians, the better information they can give us. And the more we're interacting directly, the better we are. And so I get it, they're busy, they don't want to, they may not be at the hospital. They're seeing patients at an outpatient clinic and a radiologist isn't even there, that's where that technology piece. This is how we live. We're an instantaneous society. We live through our phone and so great it's like a FaceTime capability. If you want to maintain those personal relationships, we're learning we can't rely on the orthopedist or whomever, whatever referring physician to stop by our reading room, our department. We need to make ourselves available to them and make it convenient. >> That market that you working in Cincinnati, we have a luxury of having quite a few customers with our iConnect solutions. There's been some talk between the multiple parties, of potentially being able to look across the other sites and using that common tool, but being able to query the other archives. Is that something that you'd in favor of supporting and think would add value so that the clinicians can see the longitudinal record? >> Yes And we already have that ability of we can view care everywhere in our EMR. So we don't have the images right away, but we can see other reports. Again, it's not convenient. It's not a click away, but it's two, three, four clicks away. But if I see, if it's one of my search patterns of I just worked the overnight shift last week and then you get something through the ER and there's no comparisons, and it's an abnormal chest CT. Well, I look in Care Everywhere. Oh, they had a chest CT at a different place in the city a year ago, and I can see the report. And so then at that time I can request, and it can take an hour or so, but look back and the images will be accessible to me. But so how do we improve on that? Is to make the images, that I don't have to wait an hour for the images. If we have image sharing among your organizations that can be much quicker, would be a big win. >> As you read in your new environment, do you swivel your chair and still read out of any other specialty systems, for any types of studies today? >> No, and that was a huge win. We used to have a separate viewing system for mammography and we were caught like there were dedicated viewing stations. And so even though we're a system, the radiologist working at this hospital, had to read the mammograms taken at that hospital. And one at the other hospital could only read the ones taken at that hospital. And you couldn't share the workload if it was heavy at one site and light at the other. Well, now it's all viewed through the radiology viewer if you merge PACS, in not just general radiology, but impressed. It has been so much better world that the workflow is so much better, that we can share the work list and be much more efficient. >> Do you feel that in your, your new world, that you're able to have less cherry picking between the group, I guess? Do you feel like there's less infighting or that the exams are being split up evenly through the work list? Or are you guys using some sort of assignment? >> No. And I'm curious with our next version of PACS, the next version of merge packs of 008. I forget which particular >> John: 008. >> It's 008, yeah. I know there's the feature of a smart work list to distribute the exams. Currently, we just have one. It's better than what we have before. It's one large list. We've subdivided, teased out some things that not all of the radiologist read of like MSK and cardiac and it makes it more convenient. But currently it is the radiologist choose what study they're going to open next. To me how I personally attack the list is I don't look at the list. Some radiologists can spend more time choosing what they're going to read next than they do reading. (chuckles) And so if you don't even look, and so the feature I love is just I don't want to take my eyes off my main viewer. And I don't want to swivel my chair. I don't want to turn my head to look at the list, I want everything right in front of me. And so currently the way you can use it is I never look at the list. I just use the keyboard shortcuts of, okay, well I'm done with that study. I mark it, there's one button I click on my mouse that marks it dictated, closes it and brings up the next study on the list. >> Hey guys, I got to jump in. We're running up against the clock, but John if you've got any final thoughts or Dr. Huelsman, please. >> Sure. Dr. Huelsman, I guess any homework for me? What are the top two or three things I can help you with in 2021 to be successful? >> Keep us informed of what you're working on, of what's available now. What's coming next, and how soon is it available? And you let us see those things? And we'll give you a feedback of hey, this is great. And we'll try to identify things, if you haven't thought of them, hey, this would be very helpful. >> Gents, great conversation. Gosh we could go on for another 45 minutes. And John you really have a great knowledge of the industry. And Dr. Huelsman, thanks so much for coming on. Appreciate it. >> Thank you. >> You're welcome >> And thanks for spending some time with us. You're watching "Client Conversations" with IBM Watson Health.

>>Welcome to this IBM Watson health client conversation here. We're probing the dynamics of the relationship between IBM and its clients. And we're looking back, we're going to explore the present. We're going to discuss the future state of healthcare. My name is Dave Volante from the Cuban with me are Robert Stell horn. Who's associate director, H E O R at sukha, otherwise known as pharmaceuticals, America and Rena Felton. Who's with of course, IBM Watson health. Welcome folks. Great to have you. Hi, so like strong relationships, as we know, they're the foundation of any partnership. And of course over the past year, we've had to rely on both personal and professional relationships to get us through some of the most challenging times, if not the most challenging times of our lives. So let me start with you, Robert, how has the partnership with IBM helped you in 2020? >>I think it was just a continuation of the excellent relationship we have with Rena and IBM. Um, starting in March, we had really a shift to an all remote, uh, workplace environment. And I think that constant communication with Rina and IBM helped that situation because she kept us up to date with, uh, additional products and offerings. And basically we came up with some additional solutions towards the end of the year. So we're gonna watch >>Pick it up from here. Let's go, let's go a little bit deeper and maybe you can talk about some of the things that you've done with Robert and his team and, and maybe some of the accomplishments that you're most proud of in 2020. >>No, absolutely. And I have to kind of echo what you first said about the foundation and our partnerships being the foundation, um, of our past present and future. So I do want to take the opportunity to thank Rob again for joining us today. It is, um, I know, you know, with his kids home and remote learning, um, it's a lot, uh, to, to ask in addition to, you know, your day to day work. So, so thank you, Rob. Um, I guess the question that I have for you is what would be the greatest accomplishment, um, that Watseka and IBM Watson had in 2020? >>I would say it was the addition of the linked claims EMR data, the LDCD product that we were able to license in-house, uh, thanks to your attention and to show the advantages and the strengths of that data. We are able to license that in to our, uh, set up assets we have internally. And what that's gonna allow us to do is really find out more information about the patients. Uh, we're existing users of the Mark IBM, uh, market scan data. Um, this is going to allow us to tie into those same patients and find out more about them. Um, in particular, uh, a lot of our products are in the mental health space and a lot about standing questions we have are why are the patients getting different products? And with the notes are available in that link data. We're going to now be able to tap into more information about what is happening with the patient. >>Okay. Can I ask a question on that? Um, if you guys don't mind, I mean, you know, when you, when you hear about, you know, uh, EMR, uh, in the early days, it was a lot about meaningful use and getting paid. It sounds like you guys are taking it much deeper and as a, as a, you know, as an individual, right, you're, you're really happy to hear that this information is now going to be used to really improve, uh, healthcare is, do I have that right? Is that, you know, kind of the nature of where you guys are headed? >>Well, I think ultimately it's the, the, the, the main goal is to help the patients and provide the products that can really, um, help them in their daily lives. So, um, really with this data, now, we're going to be able to tap into more of the why, um, exist in claims data. We cannot really get that information, why VC information, about what diagnoses they incurred during their treatment history. And we also can see, uh, different prescriptions that are given to them, but now we're going to be able to tie that together and get more understanding to really see more focused treatment pattern for them. >>So, Reno, w w you sit down with Rob, do you have like a, sort of a planning session for 2021? Why don't you sort of bring us up to, uh, to what your thinking is there and how you guys are working together this year? >>Yeah, no, absolutely. Um, actually, before we get to that, I wanted to kind of add onto what Rob was saying as well. It's interesting given, you know, the pandemic in 2020 and what the LCD data is going to do, um, to really be able to look back. And as Rob mentioned, looking specifically at mental health, the ability to look back and start looking at the patients and what it's really done to our community and what it's really done to our country, um, and looking at patients, you know, looking back at, at sort of their, their patient journey and where we are today. Um, but Rob and I talk all the time, we talk all the time, we probably talk three or four times a day sometimes. So I would say, um, we, we text, uh, we do talk and have a lot of our strategic, um, sessions, uh, our outlook for 2021 and what the data strategy is for Otsuka. Um, in addition, additional data assets to acquire from IBM, as well as how can we sort of leverage brander IBM, um, assets like our red hat, our OpenShift, our cloud-based solutions. So, you know, Rob and I are constantly talking and we are, um, looking for new ways to bring in new solutions into Otsuka. Um, and you know, yeah, we, we, we talk a lot. What do you think, Rob? >>I think we have an excellent partnership. Uh, basically, um, I think their relationship there is excellent. Um, we have excellent communication and, you know, I find when there's situations where I may be a bind Reno's is able to help out instantly. Um, so it's, it's really a two way street and it's an excellent partnership. >>I wonder if I could double click on that. I mean, relative to maybe some of your, I mean, I'm sure you have lots of relationships with lots of different companies, but, but what makes it excellent specifically with regard to IBM? Is there, is there anything unique Rob, that stands out to you? >>It would be the follow-up, um, really, it's not just about, uh, delivering the data and say, okay, here you have your, your product work with it in basically the, the, the vendor disappears, it's the constant followup to make sure that it's being used in any way they can help and provide more information to really extract the full value out of it. >>So I'm gonna forget to ask you guys, maybe each of you, you know, both personally and professionally, I feel like, you know, 20, 20 never ended it just sort of blended in, uh, and, and, but some things have changed. We all talk about, geez, what's going to be permanent. How have you each been affected? Um, how has it helped you position for, for what's coming in in the years ahead, maybe Reena, you could start and then pick it up with, with Rob. >>Oh man. Um, you know, 2020 was definitely challenging and I think it was really challenging given the circumstances and in my position where I'm very much used to meeting with our customers and having lunch and really just kind of walking down the hallways and bumping into familiar faces and really seeing, you know, how we can provide value with our solutions. And so, you know, that was all stripped in 2020. Uh, so it's been, it's been quite challenging. I will say, working with Rob, working with some of my other customers, um, I've had, uh, I've had to learn the resilience and to be a little bit more relentless with phone calls and follow ups and, and being more agile in my communications with the customers and what their needs are, and be flexible with calendars because there's again, remote learning and, and, um, and the like, so I think, you know, positioned for 2021 really well. Um, I am excited to hopefully get back out there and start visiting our customers. But if not, I certainly learned a lot and just, um, the follow-up and again, the relentless phone calls and calling and checking up on our customers, even if it's just to say, hi, see how everyone's doing a mental check sometimes. So I think that's, that's become, um, you know, what 2020 was, and, and hopefully, you know, what, 2021 will be better and, uh, kind of continue on that, that relentless path. >>What do you think, Rob? Hi, how are you doing? >>I would echo a lot of Rina's thoughts and the fact of, yeah, definitely miss the in-person interaction. In fact, I will say that I remember the last time I was physically in the office that Scott, it was to meet with Rina. So I distinctively remember that they remember the date was March, I believe, March 9th. So it just shows how this year as has been sort of a blur, but at the same time, you remember certain milestones. And I think it's because of that relationship, um, we've developed with IBM that I can remember those distinctive milestones and events that took place. >>So Rob, I probably should have asked you upfront, maybe tell us a little bit about Alaska, uh, maybe, maybe give us the sort of quick soundbite on where you guys are mostly focused. Sure. >>Oh, it's guys, uh, a Japanese pharmaceutical company. The focus is in mental health and nephrology, really the two main business areas. Um, my role at guys to do the internal research and data analytics within the health economics and outcomes research group. Um, currently we are transitioned to a, uh, name, which is global value and real world evidence. Um, fact that transition is already happened. Um, so we're going to have more of a global presence going forward. Um, but my role is really to, uh, do the internal research across all the brands within the company. >>So, so Rena, I wonder this, thank you for that, Robert. I wonder if you could think, thinking about what you know about Scott and your relationship with Robin, your knowledge of, of the industry. Uh, there's so much that IBM can bring to the table. Rob was talking about data earlier, talking about EMR, you were talking about, you know, red hat and cloud and this big portfolio you have. So I wonder if you could sort of start a conversation for our audience just around how you guys see all those assets that you have and all the knowledge, all that data. How do you see the partnership evolving in the future to affect, uh, the industry and the, in the future of healthcare? >>Well, I would love to see, um, the entire, uh, uh, platform, um, shift to, to the IBM cloud, um, and certainly, you know, leverage the cloud pack and analytics that, that we have to offer, um, baby steps most definitely. Um, but I do think that there is, uh, the opportunity to really move, um, and transform the business into something a lot more than, than what it is. >>Rob has the pandemic effected sort of how you think about, um, you know, remote services and cloud services and the, like, were you already on the path headed there? Did accelerate things, have you, you know, have you not had time because things have been so busy or maybe you could comment? >>Yeah, I think it's really a combination. And so I think you hit on a, a fair point there, just the time, uh, aspect. Um, it's definitely been a challenge and your, um, I have two children and remote learning has definitely been a challenge from that perspective. So time has definitely been, uh, on the short side. Um, I do see that there are going to in the future be more and more users of the data. So I think that shift to a potential cloud environment is where things are headed. >>So we, I have a bunch more questions, but I want to step back for a second and see if there's anything that you'd like to ask Rob before I go onto my next section. Okay. So I wonder if you could think about, um, maybe both of you, the, the, when you think back on, on 2020 and all the, you know, what's transpired, what, what transitions did you guys have to make? Uh, maybe as a team together IBM and Alaska. Um, and, and, and what do you see as sort of permanent or semi-permanent is work from home? We're gonna going to continue at a higher rate, uh, are there new practice? I mean, I know just today I made an online appointment it's for a remote visit with my doctor, which never could have happened before the pandemic. Right. But are there things specific to your business and your relationship that you see as a transition that could be permanent or semi-permanent? >>Well, I, I think it's there, there's definitely a shift that's happened that will is here to stay, but I don't know if it's full, it's going to be a combination in the future. I think that in-person interactions, especially what Rena mentioned about having that face-to-face interaction is still going to be one things are in the right place and safe they're going to happen again. But I think the ability to show that work can happen in a virtual or a full remote workplace, that's going to just allow that to continue and really give the flex of people. The flexibility I know for myself, flexibility is key. Like I mentioned, with two small children, um, that, that, that becomes such a valuable addition to your work, your life and your work life in general, that I think that's here to stay. >>Okay. Um, so let me ask you this, uh, w one of the themes of this event is relentless re-invention. So what I'm hearing from you Rob, is that it kind of a hybrid model going forward, if you will, uh, maybe the option to work from home, but that face to face interaction, especially when you're creating things like you are in the pharmaceutical business and the deep R and D that collaborative aspect, you know, you, it's harder when you're, when, when, when you're remote. Um, but maybe you could talk about, you know, some of those key areas that you're, you're going to be focused on in 2021 and, and really where you would look for IBM to help. >>I think in 2021, the team I'm part of it, part of is, is growing. So I think there's going to be additional demand for internal research, uh, uh, capabilities for analysis done within the company. So I think I'm going to be looking to Rena to, uh, see what new data offerings are available and all what new products are going to be available. But beyond that, um, I think it's the potential that, you know, there's so much, uh, projects, um, that are going to be coming to the table. We may need to outsource some of that projects and IBM could be potentially be a partner there to do some of the analysis on to help out there. >>Anything you'd add. >>Uh, no, I think that, that sounds good. >>How would you grade IBM and your relationship with IBM Rob? >>Well, I have to be nice to Rina cause she's been very nice to me. I would say an a, an a plus >>My kids, I got kids in college. Several, they get A's, I'm happy. Oh, that's good. You know, you should be proud. So, congratulations. Um, anything else Reno, you give you, I'll give you a last word here before we wrap, >>You know, 2020 was, was a challenging. And, you know, we talked a little bit about, you know, what time in 2020, you know, Rob and I have always had a really good relationship. I think 2020, we got closer, um, with just both professionally and really diving in to key business challenges that they have, and really working with him to understand what the customer needs are and how we can help, not only from, you know, an HR perspective, but also how can we help Otsuka, um, as a company in, in totality. So, you know, we've been able to do that, but personally, I would say that I really appreciated the relationship. I mean, we can go from talking about work to talking about children, to talking about family, um, all in the same five minute conversation or 10 minute conversation, sometimes our conversation. So, you know, thank you, Rob 2020 was definitely super challenging. >>I know for you on so many levels. Um, but I have to say you've been really great at just showing up every time picked up the phone, asked questions. If I needed something I can call you, I knew you were going to pick up, I had an offering and be like, do you have 10 minutes? Can I share this with you? And you would pick up the phone, no problem, and entertain a call or set up a call with all your internal colleagues. And I, I appreciate that so much. And, you know, I appreciate our relationship. I appreciate the business and I, I do hope that we can continue on in 2021, we will continue on in 2021. Uh, but, um, but yeah, I thank you so much. >>Rain has been extremely helpful. I don't want to thank you for all the help. Um, just to add to that one point there, you know, we have, uh, also another product, which I forgot to mention that we licensed in from IBM, it's the treatment pathways, um, tool, which is an online tool. Um, and we have users throughout the globe. So there's been times where I've needed a new user added very quickly for someone in the home office in Japan. And Rena has been extremely helpful in getting things done quickly and very proactively. >>Well, guys, it's really clear that the depth of your relationship I'm interested that you actually got closer in 2020. Uh, the fact that you communicate, you know, several times a day is I think Testament to that relationship. Uh, I'm really pleased to hear what you're doing and the potential with the EMR data for patient outcomes. Uh, as I say in the early days, I used to hear all about how well you have to do that to get paid. And it's really great to see a partnership that's, that's really focused on, on, on patient health and, and changing our lives. So, and mental health is such an important area that for so many years was so misunderstood and the, and the data that we now have, and of course, IBM's heritage and data is key. Uh, the relationship and the follow-up and also the flexibility is, is something I think we all learned in 2020, we have to, we've kind of redefined, you know, resilience in our organizations and, uh, glad to see you guys are growing. Congratulations on the relationship. And thanks so much for spending some time with me. >>Thank you. Thank you, Dave. Thank you, Raina >>For watching this client conversation with IBM Watson health.

>> Narrator: Live from Miami, Florida. It's theCUBE. Covering IBM's data and AI forum. Brought to you by IBM. >> Welcome back to the port of Miami everybody. This is theCube, the leader in live tech coverage. We're here covering the IBM AI and data forum. Of course, the centerpiece of IBM's AI platform is Watson. Beth Smith is here, she's the GM of IBM Watson. Beth, good to see you again. >> You too. Always good to be with theCUBE. >> So, awesome. Love it. So give us the update on Watson. You know, it's beyond Jeopardy. >> Yeah, yeah. >> Oh, wow. >> That was a long time ago now. (laughs) >> Right, but that's what a lot of people think of, when they think of Watson. What, how should we think about Watson today? >> So first of all, focus Watson on being ready for business. And then, a lot of people ask me, "So what is it?" And I often describe it as a set of tools, to help you do your own AI and ML. A set of applications that are AI applications. Where we have prebuilt it for you, around a use case. And there is examples where it gets embedded in a different application or system that may have existed already. In all of those cases, Watson is here, tuned to business enterprise, how to help people operational-wise, AI. So they can get the full benefit, because at the end of the day it's about those business outcomes. >> Okay, so the tools are for the super geeks, (Beth laughs) who actually want to go in and build the real AI. >> (laughs) That's right, that's right. >> The APPS are, okay. It's prebuilt, right? Go ahead and apply it. >> That's right. >> And the embedded is, we don't even know we're using it, right? >> That's right, or you may. Like, QRadar with Watson has an example of using Watson inside of it. Or, OpenPages with Watson. So sometimes you know you're using it. Sometimes you don't. >> So, how's the mix? I mean, in terms of the adoption of Watson? Are there enough like, super techies out there, who are absorbing this stuff? Or is it mostly packaged APPS? Is it a mix? >> So it is a mix, but we know that data science skills are limited. I mean, they're coveted, right? And so those are the geeks, as you say, that are using the tool chain as a part of it. And we see that in a lot of customers and a lot of industries around the world. And then from a packaged APP standpoint, the biggest use case of adoption is really around customer care, customer service, customer engagement. That kind of thing. And we see that as well. All around the world, all different industries. Lots of great adoption. Watson Assistant is our flagship in that. >> So, in terms of, if you think about these digital initiatives, we talked about digital transformation, >> Yup. >> Last few years, we kind of started in 2016 in earnest, it's real when you talk to customers. And there was a ton of experimentation going on. It was almost like spaghetti. Throw against the wall and see what sticks. Are you seeing people starting to place their bets on AI, Narrowing their scope, and really driving you know, specific business value now? >> Beth: Yeah. >> Or is it still kind of all over the place? >> Well, there's a lot of studies that says about 51% or so still stuck in experimentation. But I would tell you in most of those cases even, they have a nice pilot that's in production, that's doing a part of the business. So, 'cause people understand while they may be interested in the sexiness of the technology, they really want to be able to get the business outcomes. So yes, I would tell 'ya that things have kind of been guided, focused towards the use cases and patterns that are the most common. You know, and we see that. Like I mentioned, customer care. We see it in, how do you help knowledge workers? So you think of all those business documents, and papers and everything that exists. How do you assist those knowledge workers? Whether or not it's an attorney or an engineer, or a mortgage loan advisor. So you see that kind of use case, and then you see customers that are building their own. Focused in on, you know, how do they optimize or automate, or predict something in a particular line of business? >> So you mentioned Watson Assistant. So tell us more about Watson Assistant, and how has that affected adoption? >> So Watson Assistant as I said, it is our flagship around customer care. And just to give you a little bit of a data point, Watson Assistant now, through our public cloud, SaaS version, converses with 82 million end users a month. So it's great adoption. And this is, this is enabling customers. Customers of our customers, to be able to get self-service help in what they're doing. And Watson Assistant, you know, a lot of people want to talk about it being a chat bot. And you can do simple chat bots with it. But it's to sophisticated assistance as well. 'Cause it shows up to do work. It's there to do a task. It's to help you deal with your bank account, or whatever it is you're trying to do, and whatever company you're interacting with. >> So chat bots is kind of a, (laughs) bit of a pejorative. But you're talking about digital systems, it's like a super chat bot, right? >> Beth: Yeah. I saw a stat the other day that there's going to be, by I don't know, 2025, whatever. There's going to be more money spent on chat bot development, or digital assistance, than there is on mobile development. And I don't know if that's true or not, >> Beth: Mhm, wow. But it's kind of an interesting thing. So what are you seeing there? I mean, again I think chat bots, people think, oh, I got to talk into a bot. But a lot of times you don't know you're, >> Beth: That's right. >> so they're getting, they're getting better. I liken it to fraud detection. You know, 10 years ago fraud detection was like, six months later you'll, >> Right. >> you'll get a call. >> Exactly. >> And so chat bots are just going to get better and better and better, and now there's this super category that maybe we can define here. >> That's right. >> What is that all about? >> That's right. And actually I would tell you, they kind of, they can become the brain behind something that's happening. So just earlier today I was, I was with a customer and talking about their email CRM system, and Watson Assistant is behind that. So chat bots aren't just about what you may see in a little window. They're really about understanding user intent, guiding the user through what they're trying to either find out or do, and taking the action as a part of it. And that's why we talk about it being more than chat bots. 'Cause it's more than a FAQ interchange. >> Yes, okay. So it's software, >> Beth: Yes. >> that actually does, performs tasks. >> Beth: Yes. >> Probably could call other software, >> Beth: Absolutely. >> to actually take action. >> That's right. >> I mean, I see. We think of this as systems of agency, actually. Making, sort of, >> That's right. >> decisions and then I guess, the third piece of that is, having some kind of human interaction, where appropriate, right? >> That's right. >> What do you see in terms of, you know, infusing humans into the equation? >> So, well a couple of things. So one of the things that Watson Assistant will do, is if it realizes that it's not the expert on whatever it is, then it will pass over to an expert. And think of that expert as a human agent. And while it's doing that, so you may be in the queue, because that human person is tied up, you can continue to do other things with it, while you're waiting to actually talk to the person. So that's a way that the human is in the loop. I would tell you there's also examples of how the agents are being assisted in the background. So they have the interaction directly with the user, but Watson Assistant is helping them, be able to get to more information quicker, and narrow in on what the topic is. >> So you guys talk about the AI ladder, >> Beth: Mhm. >> Sort of, Rob talked about that this morning. My first version of the AI ladder was building blocks. It was like data and AI analytics, ML, and then AI on top of that. >> Beth: Yup. >> I said AI. Data and IA. >> Beth: Yup. >> Information Architecture. Now you use verbs. Sort of, to describe it. >> Beth: Yup. Which is actually more powerful. Collect, organize, analyze and infuse. Now infuse is like the Holy Grail, right? 'Cause that's operationalizing and being able to scale AI. >> Beth: That's right. >> What can you tell us about how successful companies are infusing AI, and what is IBM doing to help them? >> So, I'm glad you picked up first of all, that these are verbs and it's about action. And action leads to outcome, which is, I think, critical. And I would also tell you yes, infuse is, you know, the Holy Grail of the whole thing. Because that's about injecting it into business processes, into workflows, into how things are done. So you can then see examples of how attorneys may be able to get through their legal prep process in just a few minutes, versus 10, 15 hours on certain things. You can see conversion rates of, from a sales standpoint, improve significantly. A number of different things. We've also got it as a part of supply chain optimization, understanding a little bit more about both inventory, but also where the goods are along the way. And particularly when you think about a very complicated thing, there could be a lot of different goods in various points of transit. >> You know, I was sort of joking. Not joking, but mentioning Jeopardy at first. 'Cause a lot of people associate Watson with Jeopardy. >> Beth: Right. >> I can't remember the first time I saw that. It had to be the mid part of the last decade. What was it? >> Beth: February of 2011. >> 2011, okay I thought I even saw demos before that. I'm actually sure I did. Like in, back in some lab in IBM. And of course, the potential like, blew your mind. >> Right. >> I suspect you guys didn't even know what you had at the time. You were like, "Okay, we're going to go change the world." And you know, when you drive up and down 101 in Silicone Valley, it's like, "Oh, Watson this, Watson that." You know, you get the consumer guys, doing facial recognition, ad serving. You know, serving up fake news, you know. All kinds of applications. But IBM started to do something different. You're trying to really change business. Did you have any clue as to what you had at the time? And then how much of a challenge you were taking on, and then bring us to where we are now, and what do you see as a potential for the next 10 years? >> So, of course we had a clue. So let me start there. (Dave laughs) But with that, I think the possibilities of it weren't completely understood. There's no question in my mind about that. And what the early days were, were understanding, okay, what is that business application? What's the pattern that's going to come about as a part of it? And I think we made tremendous progress on that along the way. I would tell you now, you mentioned operationalizing stuff, and you know, now it's about, how do we help companies have it more throughout their company? Through different lines of business, how does it tie to various things that are important to us? And so that brings in things like trust, explainablity, the ethics of what it's doing. Bias detection and mitigation. And I actually believe a lot of that, and the operationalizing it within the processes, is where we're going to head, going forward. Of course there'll continue to be advancements on the features and the capabilities, but it's going to be about that. >> Alright, I'm going to ask you the it's depends question. (Beth laughs) So I know that's your answer, but at the macro, can machines make better diagnosis than doctors today, and if not, when will they be able to, in your view? >> So I would actually tell you that today they cannot, but what they can do is help the doctor make a better diagnosis than she would have done by herself. And because it comes back to this point of, you know, how the machine can process so much information, and help the expert, in this case the doctor's the expert, it could be an attorney, it could be an engineer, whatever. Help that expert be able to augment the knowledge that he or she has as a part of it. So, and that's where I think it is. And I think that's where it will be for my lifetime. >> So, there's no question in your mind that machines today, AI today, is helping make better diagnosis, it's just within augmented or attended type of approach. >> Absolutely. >> And I want to talk about Watson Anywhere. >> Beth: Okay, great. >> So we saw some discussion in the key notes and some demos. My understanding is, you could bring Watson Anywhere, to the data. >> That's right. >> You don't have to move the data around. Why is that important? Give us the update on Watson Anywhere. >> So first of all, this is the biggest requirement I had since I joined the Watson team, three and a half years ago. Was please can I have Watson on-prem, can I have Watson in my company data center, etcetera. And you know, we needed to instead, really focus in on what these patterns and use cases were, and we needed some help in the platform. And so thanks to Cloud Pak for data, and the underlying Red Hat OpenShift and container platform, we now are enabled to truly take Watson anywhere. So you can have it on premise, you can have it on the other public clouds, and this is important, because like you said, it's important because of where your data is. But it's also important because the workloads of today and tomorrow are very complex. And what's on cloud today, may be on premise tomorrow, may be in a different cloud. And as that moves around, you also want to protect the investment of what you're doing, as you have Watson customize for what your business needs are. >> Do you think you timed it right? I mean, you kind of did. All this talk about multicloud now. You really didn't hear much about it four or five years ago. For awhile I thought you were trying to juice your cloud business. Saying, "You want, if you want Watson, you got to go to the IBM cloud." Was there some of that, or was it really just, "Hey, now the timing's right." Where clients are demanding it, and hybrid and multicloud and on-prem situations? >> Well look, we know that cloud and AI go hand in hand. So there was a lot of positive with that. But it really was this technology point, because had I taken it anywhere three and a half years ago, what would've happened is, every deployment would've been a unique environment, a unique stack. We needed to get to a point that was a modern day, you know, infrastructure, if you will. And that's what we get now, with a container based platform. >> So you're able to scale it, such that every instance isn't a snowflake, >> That's right. >> that requires customization. >> That's right. So then I can invest in the enhancements to the actual capabilities it is there to do, not supporting multiple platform instantiations, under the covers. >> Well, okay. So you guys are making that transparent to the customer. How much of an engineering challenge is that? Can you share that with us? You got to run on this cloud, on that cloud, or on forever? >> Well, now because of Cloud Pak for data, and then what we have with OpenShift and Kubernetes and containers, it becomes, well, you know, there's still some technical work, my engineering team would tell you it was a lie. But it's simple now, it's straightforward. It's a lot of portability and flexibility. In the past, it would've been every combination of whatever people were trying to do, and we would not have had the benefit of what that now gives you. >> And what's the technical enable there? Is it sort of open API's? Architecture that allows for the interconnectivity? >> So, but inside of Watson? Or the overall platform? >> The overall platform. >> So I would say, it's been, at it's, at it's core it's what containers bring. >> Okay, really. So it's that, it's that. It's the marriage of your tech, >> Yeah. >> with the container wave. >> That's right. That's right. Which is why the timing was critical now, right? So you go back, yes they existed, but it really hadn't matured to a point of broad adoption. And that's where we are now. >> Yeah, the adoption of containers, Kubernetes, you know, micro services. >> Right, exactly. Now it's on a very steep curve. >> Exactly. >> Alright, give your last word on, big take away, from this event. What do you hearing, you know, what are you, some of the things you're most excited about? >> So first of all, that we have all of these clients and partners here, and all the buzz that you see. And that we've gotten. And then the other thing that I would tell you is, the great client examples. And what they're bragging on, because they are getting business outcomes. And they're getting better outcomes than they thought they would achieve. >> IBM knows how to throw an event. (Beth laughs) Beth, thanks so much for coming to theCUBE. >> Thank you, good to >> Appreciate it. >> see you again. >> Alright, great to see you. Keep it right there everybody, we'll be back. This is theCUBE live, from the IBM Data Forum in Miami, we'll be right back. (upbeat instrumental music)

(upbeat pop music) >> From our studios in the heart of Silicon Valley, Palo Alto, California, this is a CubeConversation. >> Hello, everyone welcome to this CUBE Conversation here in Palo Alto, California, I'm John Furrier, co-host of theCUBE. We are here forth Inhi Cho Suh General Manager of IBM Watson, Customer Engagement, Former Cube alumni, I think she's been on dozens of times. Great to see you again. Welcome to our Palo Alto Studios. >> Yeah, great being here, John. >> So, we haven't chatted in awhile. IBM thing just happened, a little bit of a rainy event, here in February. Interesting change over since we last talked, but first give an update on what you're up to these days, what group are you leading, what's new? >> Okay, well first of all, I'm here based in California, which I'm excited about, and I lead our Watson West office, which is our Watson headquarters, here on the west coast, in downtown San Francisco, and we hosted our Think Conference, and at Think I lead with, in IBM, what we call our Watson Customer Engagement Business Unit, which is really the business applications, of how we apply Watson and other disruptive tech to a line of business audiences, both SAS and on premise software, so really excited about the areas of applying AI and machine learning as well as Blockchain to things like supply chain, and logistics, to order management, to next generation of retail. A lot of new, exciting areas. >> Yeah, we've had many conversations over the years from big data to as your career spanned across IBM, and you have a much more horizontal view of things, now. You're horizontally scalable, as we say in the cloud world. What's your observation of the trends these days? Because there's a lot waves. Actually, the waves that you guys announced, was the IBM, Watson NE ware and the cloud private ware. Marvin and I had an amazing conversation that video went viral. This is now getting a big tailwind for IBM. What's your thoughts in general about the overall ecosystem, because you're here in Silicon Valley, you've seen the big waves, you've got another big data world, cloud is here, multi cloud. What's your thoughts on the big mega-trends? >> Yeah, that's a good question. I think the first chapter of cloud, everyone ran to public cloud. When you look at it through the lens of enterprise, though, the hot topic right now in the second chapter is really about not just public cloud, but multi-cloud, hybrid cloud. Meaning, whether it's a private, public, it's about thinking about the applications and the nature of the applications and regardless of where the data sits, what are the implications of actually getting work done? Through, kind of, new container services, new ways of microservices in the development, of how APIs are integrated, and so, the hot topic right now is definitely hybrid cloud, multi cloud. And the work we've done to certify, what we call, IBM cloud private really enables us to not just take any business application to any cloud in our cloud, as well, but actually to enable Watson and Watson based applications also across multi cloud environments. >> So, chapter two, Jenny mentioned that in her key notes, I want to dig into that because we've been talking a lot about multi cloud architecture, and one of the big debates has been, in the industry, oh, don't pick a soul cloud. I've been writing a bunch of content about that at this DOD jedi deal with Amazon and Oracle, fighting for it out there, but that's also happening at the enterprise, but the reality is, everyone has multiple clouds. If you've got a sales force or if you've got this and that and the other thing, you probably have multiple clouds, so it's not so much soul cloud vs. as it is, workloads having a cloud for the right job and that seems to be validated at IBM Think, in talking to the top technical people and in the industry. They all say, pick the right cloud for the job. And we've heard that before in Big Data. Pick the right tool for the job. So, given that, workloads seem to be driving the demand for cloud. Since you're on the app side, how are you seeing that? Because the world's flipped. It used to be infrastructure and software enable the app's capabilities. Now the workloads have infrastructure as code, made with cloud, they're driving the requirements. This is a change over. >> It is a big change and part of, I would say, when people first ran to the cloud, and a lot of the public cloud services were digital SaaS services, where people were wanting to stitch multiple applications across clouds, and that became a challenge, so in this next iteration, that I'm seeing is, really, a couple things. One is, data gravity. So, where does the data actually reside, for the workload that's actually happening? Whether it's the transactions, whether it's customer information, whether it's product information, that's one piece. The second piece is a lot more analytics, right? And the spectrum of analytics running from traditional warehouse capabilities, to more, let's say, larger scale big data projects to full blown advanced algorithms and AI applications, is, people are saying, look, not only do I want to stitch these applications across multiple clouds; I also want to make sure I can actually tap into the data to apply new types of analytics and derive new services and new values out of relationships, understanding of how products are consumed, and so forth. So, for us, when we think about it is, we want to be able to enable that fluid understanding of data across the clouds, as well as protect and be thoughtful about the data privacy rights around it, compliance around GDPR, as well as how we think about the security aspects as well, for the enterprise. >> That is a great point. I think I want to drill down on the data piece, your background on data obviously is going to be key in your job now obviously, it's pretty obvious with Watson, but David Floyd, a wiki bonds research analyst, just posted a taxonomy of hybrid cloud research report that laid out the different kinds of cloud you could have. There's edge clouds, there's all kinds of things from public to edge, so when you look at that, you're thinking, okay, the data plain is the critical nature of the cloud. Now, depending on which cloud architecture for the use case, the workload, whatever, the data plain seems to be this magical opportunity. AI is going to have a big part of that. Can you just talk about how you guys see that evolving? Because, obviously, AI is a killer part of your strategy. This data piece is inter-operating across the clouds. >> Yes. >> Data management governs you're smiling, cause there's a killer answer coming. >> Totally. This is such a great set up. Actually, Ginni even said it in her keynote at Think, which was, you can't have an AI strategy without an information architecture strategy, which is an IA strategy, and information architecture is all about what you said: it's data preparation; understanding the foundation of it, making sure you've got the right governance structure, the integration of it, and then actually how you apply the more advanced analytics on top. So, information architecture and thinking about the data aspects in all kinds of data. Majority of the data actually sits behind, what I would say, the traditional public firewall. So, it sits behind the firewalls of our enterprise clients, like 80 plus percent of it, and then, many of the clients, we actually recently did a study, with about 5,000 senior executives, across many, many thousands of organizations, and 85% of them want to apply AI to improve their customer service, to improve the way they engage their clients and their products and services, so this is a huge opportunity right now for pretty much every organization to think through; kind of their data strategy. Their information architecture strategy, as part of their overall AI strategy. >> So, a question a got on twitter comes up a lot, and, also on my notes here, I wanted to ask you is, how can companies increase transparency trust and mitigate bias in AI? Because this comes up a lot and that's the questions that come in from the community is, Hey, I got my site, my apps running in Germany. I've got users over there, I'm global. I have to manage compliance, I got all this governess now, I'm over my shoulders, kind of a pain in the butt, but also I don't want to have the software be skewed on bias and other things, and then, I also get this whole Facebook dynamic going on, where it's like, I don't trust people holding my data. This is a big, huge issue. >> It is enormous. >> You guys are in the middle of it, what's your thoughts, what's the update, what's the dynamic and what's the solution? >> So, this is a big topic. I think we could do a whole episode just on this topic alone. So, trust and developing trust and transparency in AI should be a fundamental requirement across many, many different types of institutions. So, first of all, the responsibility doesn't sit only with the technology vendors; it's a shared responsibility across government institutions, the consumers, as well as the business leaders, in terms of how they're thinking about it. The more important piece, though, is when you think about the population that's available, that really understands AI, and they're actually coding and developing on it, is that we have to think about the diverse population that's participating in the governance of it, because you don't want just one tribe or one group that's coding and developing the algorithms, or deciding the decision models. >> Like the nerds or the geeks; they're a social aspect, society aspect as well, right? Social science. >> Exactly. I actually just did a recent conversational series with Northwestern Kellogg's business school, around the importance of developing trust and transparency, not only in the algorithms themselves, but the methodology of how you think about culture and value and ethics come into play through different lens, depending on the country you live in, as you kind of referenced, depending on your different values and religious backgrounds. It may because of different institutional and/or policy positions, depending on the nature, and so there has to be a general awareness of this that's thoughtful. Now, why I'm so excited about the work we're doing at IBM is we've actually launched a couple new initiatives. One is, what we call, AI OpenScale, which is really a platform and an opportunity to have the ability to begin to apply AI, see how AI operations and models function in production. We have methodologies in terms of engaging understanding fairness, so there's a 360 degree fairness kit, which is actually available in the open source world, there's a set of tools to understand and train people on recognizing bias, so even just definitions of, what do you mean by bias? It could be things like, group think, it could be, you're just self selecting on certain data sets to reinforce your hypotheses, it could be unconscious levels and it's not just traditionally socially oriented, types of bias. >> It could be data bias, too. It could be data bias, right? >> Totally. Machine generated biases in IOT world, also. >> So, contextual and behavioral biases kind of kick into play here. >> Yeah, but it starts with transparency trust. It also starts with thoughtful governance, it starts with understanding in your position on policy around data privacy, and those things are things that should be educational conversations across the entire industry. >> How far along are we on the progress bar there? I mean, it seems like it's early and we seem to be talking for awhile, but it seems even more early than most people think. Still a lot more work. Your thoughts on where the progress bar is on this whole mash up of tech and social issues around bias and data? Where are we? >> We're really at the early stages, and part of the reason we're at the early stages is I think people have, so far, really applied AI in very simple task oriented applications. The more, what we call, broad AI, meaning multi task work flow applications are starting, and we're also starting seeing in the enterprise. Now, in the enterprise world, you can still have bias, so, for example, when you talked about data bias, one of the simple examples I use is, think about loan approvals. If one of the criteria may be based on gender, you may have a sensitivity around the lack of women owned business leaders, and that could be a scoring algorithm that says, hey, maybe it's a higher risk when in fact, it's not necessarily a higher risk, it's just that the sampling is off, right. So, that would be a detection to say, hey maybe you have sensitivity around that data set, because you actually have an insufficient amount of data. So, part of data detection and understanding biases; where you have sampling of data that's incorrect, where your segmentation could be rethought, where it may just require an additional supervision or like decision making criteria as part of your governance process. >> This is actually a great area for young people to get involved, whether at their universities or curriculum, this kind of seems to be, whether it's political science and/or data science kind of coming together, you kind of have a mash. What's your advice to people watching that might be either in high school, college, or rethinking their career, because this seems to be hot area. >> It is a hot area. I would recommend it for every student at every age, quite frankly and we're at such an early stage that it's not too late to join and you're not too young nor are you too old to actually get in the industry, so that's point one. This is a great time for everyone to get involved. The second piece is, I would just start with online courses that are available, as well as participate in communities and companies like IBM, where we actually make available on a number of our web based applications, that you can actually do some online training and courses to understand the services that we have, to begin to understand the taxonomy and the language, so a very simple set, would be like, learn the language of AI first, and then, as you're learning coding, if you're more technically inclined, there's just a myriad of classes available. >> Final question, before I move on to the topic around inclusion and diversity, machine learning is impacting all verticals. I was just in an interview, talking with Don En-ju-bin-ski, she's got a company where it's neuroscience and machine learning coming together. Machine learning's being impacted all over. We mentioned basic data bias, and machine learning can help there. Machine learning meets blank every vertical, every market, is being impacted machine learning, which will trigger some of the things you're seeing on the app side. Your thoughts, looking at where you've come from in your career at IBM to now, just the evolution of what machine learning has enabled, your thoughts on the impact of machine learning. >> Oh, it's exciting and I'll give you a real simple example, so one of the great things my own team actually did was apply machine learning to, everyone loves the holiday shopping period, right? Between Thanksgiving to New Years, so we actually develop, what we call, Watson Order Optimizer and one of my favorite brands is REI, so the recreational equipment incorporated company, they actually applied our Watson Order Optimizer to optimize in real time. The best place, let's say you want to order a kayak or a T-shirt or a hiking boot, but the best way to create the algorithms to ship from different stores, and shipping from stores, for most retailers, is a high cost variable, because you don't know what the inventory positions are, you don't necessarily know the movement of traffic into that store, you may not even know what the price promotions are, so what was exciting about putting machine learning algorithms to this was, we could actually curate things like shipping and tax information, inventory positions of products in stores, pricing, a movement of goods as part of that calculation. So, this is like a set of business rules that are automatically developed, using Watson, in a way that would be almost impossible for any human to actually come up with all of the possible business roles, right? Because this is such a complex situation, and then you're trying to do it at the peak time, which is, like Black Friday, Cyber Monday Weekend, so we were able to actually apply Watson Machine Learning to create the business roles for when it should be shipped from a warehouse or a particular store. In order to meet the customer requirement, which is the fulfillment of that brand experienced, or the product experienced, so my view is, there are so many different places across the industry, that we could actually apply machine learning to, and my team is really excited about what we've been doing, especially in the next generation of supply chain. >> And it's also causing students to be really attracted to computer science, both men and women. My daughter, who is a senior at Berkeley, is interested in it, so you're starting to see the impact of machine learning is hitting all main stream, which is a good segue to my next question, we've been very passionate, I know it's one of your passions is inclusion and diversity or diversity and inclusion, there's always debates: D before I or I before D? Some say inclusion and diversity or diversity and inclusion. It's all the same thing, there's just a lot of effort going on to bring the tech industry up to par with the reality of the world, and so you have a study out. I've got a copy here. Talk about this study: Women in Leadership and the Priority Paradox. Talk about the study; what was behind it and what were some of the findings? >> Sure, and I'm excited that your daughter, that's a senior in college, is going to be another woman that's entering the workforce, and especially being in tech, so the priority paradox is that we actually looked at over 2,300 organizations, these are some of the top institutions around the world, that are curating and attracting the best talent and skills. Now, when you look at that population, we were surprised to find out that you would think by 2019-2018 that only 18% of those organizations actually had women in senior leadership positions, and what I categorize as senior leading positions, are in the see-swee, as vice presidents, maybe senior executives or senior managers; director level folks. So, that's one piece, which is, wow, given the size and the state where we are in the industry, only 18%: we could do better. Now, why do we believe that? The second piece is, you want the full population of the human capacity to think and creatively solve. Some of the world's biggest complex problems; you don't want a small population of the world trying to do this, so, the second piece of the paradox, which was the most surprising, is that 79% of these companies actually said that formalizing or prioritizing gender, fostering that kind of inclusive culture, was not a business priority, and that they had a harder time actually mapping that gap. Now, in the study, what we actually discovered though, was those companies, that did make it a priority, actually had first mover advantage, and making it a priority is quite simple. It's about understanding how to create that inclusive culture, to allow different perspectives and different experiences to be allowed in the co-creation and development. >> So, first mover advantage, in terms of what? >> Performance, actual business performance, so even though 80% of the organizations that we interviewed actually said that they've not made it a business priority, the 20% that did, we actually saw higher performance in their outcomes, in terms of business performance. >> So, this is actually a business benefit, too. I think your point is, the first mover advantage is saying, those companies that actually brought in the leadership to create that different perspective, had higher performance. >> Absolutely. >> We've talked about this before; one of the things I always say is that, tech is now mainstream, and it's 18% of the target audience of tech isn't the market, it's 50/50 or 51. Some say 51% women/men, so who's building the products for half the audience? So, again, this doesn't make any sense, so this is a good statistic. >> It is, and if you think about the students that are actually graduating out of graduate school, recently, there's actually more women graduating out of grad school than men. When you think about that population that's now entering the workforce, and what's actually happening through the pipeline, I think there's got to be thoughtful focus and programmatic improvements across the industry, around how to develop talent and make sure that different companies and organizations can move. Like you said, problem solve for creating new products that actually serve the world, not just serve certain populations, but also do it in a way that's thoughtful about, kind of, the makeup. >> And the mainstream and prep of tech obviously makes it more attractive, I mean, you're seeing a lot more women thinking about machines, like my daughter, the question is, how do they come in and not lose their footing, mentor-ship? So, what are the priorities that you see the industry needs to do? What are some of the imperatives to keep the pipeline and keep all the mentoring, obviously mentoring is hot, we see the networking built. >> Yeah, mentoring is huge. >> What's your thoughts on the best practices that you've been involved in? >> Some of the best practices we've actually done a number with an IBM, we've done a program called, Tech Re-Entry, so women that have decided to come back into the tech workforce, we actually have a 12 week internship program to do that. Another is a big initiative that we have around P-TECH, which is the next generation of workers aren't just going to have a formal college and or PHD masters type degrees. The next generation, which we're calling, is not necessarily a white collar, blue collar, what we're calling it is, new collar, meaning these are students that are able to combine their equivalent of a high school degree and early college education in one to be kind of, if you think about it, next generation of technical vocational schools, right? That quickly enter the workforce, are able to do jobs in terms of web development, in terms of cloud management, cloud services, it could be next generation of-- >> It's a huge skill gap opportunity, this is a big opportunity for people. >> It is, and we're seeing great adoption. We've seen it on a number of states across the US, this is an effort that we partner with, the states and the governors of each state, because public education has got to be done in a systematic way that you can actually sustain it for many, many years and this is something that we were excited about championing in the state of New York first. >> The ReEntry program and other things, I always tell myself, the technology is so new now you could level up a lot faster than, there's not that linear school kind of mentality, you don't need eight years to learn something. You could literally learn something pretty quickly these days because the gap between you and someone else is so short now, because it's all new skills. >> It's true, it's true. We talk about digital disruption through the lens of businesses, but there's a huge digital disruption through the lens of what you're talking about, which is our individual development and talent, and the ability to learn through so many different channels that's available now, and the focus around micro degrees, micro skills, micro certifications, there's so many ways for everyone to get involved, but I really do encourage everyone across every industry to have some knowledge and basis and understanding of tech, because tech will redefine how services and products are delivered across every category. >> And that's not male or female: that's just everyone. Again, back to technology for good, we can solve technology problems, You guys have been doing it at IBM, solve technology problems, but now the people problem is about getting people empowered, all gender, races, et cetera, the people getting the skills, getting employed, working for clouds, this is an opportunity. >> This is a huge opportunity. I think this is an exciting time. We feel like we're entering this next phase of, what I call, chapter two of cloud, this is chapter two of digital reinvention, of the enterprise, digital reinvention of the individual, actually, and it's an opportunity for every country, every population group to get involved, in so many new and creative ways, and we're at the early foundation stages in terms of both AI development, as well as new capabilities like Blockchain. So, it's an exciting time for everybody. >> Well, that's a whole nother topic. We'll have to bring you back, Inhi. Great to see you, in fact, welcome to Palo Alto. First time in our studio. Let's co-host something together, me and you. We'll do a series: John and Inhi. >> I would love that. That would be fun. I'm excited to be here. >> You can drop by our studio anytime now that you live in Palo Alto, we're neighbors. Inhi Cho Suh here, general manager IBM Watson, customer engagement, friend of theCUBE, here inside our studios, Palo Alto. I'm John Furrier, thanks for watching. (upbeat music)

I from Las Vegas Nevada it's the cube covering IBM world of Watson 2016 brought to you by IBM now here are your hosts John furrier and Dave vellante hey welcome back everyone we're here live in Las Vegas for IBM's world of Watson at the mandalay bay here this is the cube SiliconANGLE media's flagship program we go out to the events and extract the signal-to-noise I'm John Ford SiliconANGLE i'm here with dave vellante my co-host chief researcher red Wikibon calm and our next guest is inderpal bhandari who's the chief global chief data officer for IBM welcome to the cube welcome back thank you thank you meet you you have in common with Dave at the last event 10 years Papa John was just honest we just talked about the ten year anniversary of I OD information on demand and Dave's joke why thought was telling we'll set up the says that ten years ago different data conversation how do you get rid of it is I don't want the compliance and liability now it shifted to a much more organic innovative exciting yeah I need a value add what's the shift what's the big change in 10 years what besides the obvious of the Watson vision how did what it move so fast or too slow what's your take on this ya know so David used to be viewed as exhaust right the tribe is something to get rid of like you pointed out and now it's much more to an asset and in fact you know people are even talking about about quantifying it as an asset so that you can reflect it on the balance sheet and stuff like that so it certainly moved a long long way and I think part of it has to do with the fact that we are inundated with data and data does contain valuable information and to the extent that you're able to glean it and act on it efficiently and quickly and accurately it leads to a competitive advantage what's the landscape for architects out there because a lot of things that we hear is that ok i buy the day they I got a digital transformation ok but now I got to get put the data to work so I need to have it all categorized what's the setup is there a general architecture philosophy that you could share with companies that are trying to set themselves up for some baseline foundational sets of building blocks I mean I think they buy the Watson dream that's a little Headroom I just want to start in kindergarten or in little league or whatever metaphor we want to use any to baseline what's today what's the building blocks approach the building blocks approach I mean from a if you're talking about a pure technical architectural that kind of approach that's one thing if you're really going after a methodology that's going to allow you to create value from data I would back you up further I would say that you want to start with the business itself and gaining an understanding of how the business is going to go about monetizing itself not its data but you know what is the businesses monetization strategy how does the business plan to make money over the next few years not how it makes money today but over the next few years how it plans to make money that's the right starting point once you've understood that then it's basically reflecting on how data is best used in service of that and then that leads you down to the architecture the technologies the people you need the skills makes the process Tanner intuitive the way it used to be the ivory tower or we would convene and dictate policy and schemas on databases and say this is how you do it you're saying the opposite business you is going to go in and own the road map if you will the business it's a business roadmap and then figure it out yeah go back then go back well that's that's really the better way to address it than my way so the framework that we talked about in in Boston and now and just you're like the professor I'm the student so and I've been out speaking to other cheap date officers about it it's spot on this framework so let me briefly summarize it and we can I heard you not rebuilding it to me babe I'm saying this is Allah Falls framework I've stolen it but with no shame no kidding and so again we're doing a live TV it's you know he can source your head I will give him credit so but you have said they're there are two parallel and three sequential activities that have to take place for data opposite of chief data officer the two parallel our partnership with the line of business and get the skill sets right the three sequential are the thing you just mentioned how you going to monetize data access to data data sources and Trust trust the data okay so great framework and I'd say I've tested it some CEOs have said to me well I geeza that's actually better than the framework I had so they've sort of evolved as I said you're welcome and oh okay but now so let's drill into that a little bit maybe starting with the monetization piece in the early days Jonna when people are talking about Big Data it was the the mistake people made was I got to sell the data monetize the data itself not necessarily it's what you're saying yes yes I think that's the common pitfall with that when you start thinking about monetization and you're the chief data officer your brain naturally goes to well how do I monetize the data that's the wrong question the question really is how is the business planning to monetize itself what is the monetization strategy for the overall business and once you understand that then you kind of back into what data is needed to support it and that's really kind of the sets the staff the strategy in place and then the next two steps off well then how do you govern that data so it's fit for the purpose of that business lead that you just identified and finally what data is so critical that you want to centralize it and make sure that it's completely trusted so you back into those three those three steps so thinking about data sources you know people always say well should you start with internal should you start with external and the answer presumably is it depends it depends on the business so how do you how do you actually go through that decision tree what's that process like yeah I mean if you know you start with the monetization strategy of the company so for example I'll use IBM a banana and the case of IBM took me the first few months to understand that our monetization strategy was around cognitive business specifically making enterprises into cognitive businesses and so then the strategy that we have internally for IBM's data is to enable cognition within within IBM the enterprise and move forward with that and then that becomes a showcase for our customers because it is after all such a good example of a complex enterprise and so backing you know backing in from that strategy it becomes clear what are some of the critical data elements that you need to master that you need to trust that you need to centralize and you need to govern very very rigorously so that's basically how I approached it did I answer your question daivam do you get so so you touched on the on the second part I want to drill into the the third sequential activities which which is sources so i did so you did we just talk about this well the sources i mean if you had something add to that yes in terms of the i think you mentioned the internal versus external so one thing else i'll mention especially if you kind of take that 10-year outlook that we were talking about 10 years ago serials had very internal outlook in terms of the data was all internal business data today it's much more external as well there's a lot more exogenous data that we have to handle and validity and that's because we're making use of a lot more unstructured data so things like news feeds press releases articles that have just been written all our fair game to amplify the view that you have about some entity so for example if we're dealing with a new supplier you know previously we might gather some information by talking with them now we'd also be able to look at essentially everything that's out there about them and factor that in so it is a there's an element of the exogenous data that's brought to bear and then that obviously becomes part of the realm of the CDO as well to make sure that that data is available and you unusable by the business is John Kelly said something go ahead sorry well Jeff Jonas would say that's the observation space right that you want to have the news feeds it's extra metadata that could change the alchemy if you will of whatever the mix of the data is that kind of well yeah I would say you might even go further than just metadata i would say that in some some sense it's part of your intrinsic data set because you know it gives you additional information about the entities that you're collecting data on and that measuring the John Kelly in the keynote this morning he made two statements he said one is in three to five years every health care practitioners going to going to want to consult Watson and then he also said same thing for MA because watch is going to know every public piece of data about every single company right so it's would seem that within the three to five year time frame that the shift is going to be increasingly toward external data sources not necessarily the value in the lever points but in terms of the volume certainly of data is that fair I think it's a it's a fair statement I mean I think if you think of it in the healthcare context if you know a patient comes in and there's a doctor or a practitioner that's examining the patient right there they're generating some data based on their interaction but then if you think about the exogenous data that's relevant and pertinent to that case that could involve you know thousands of journals and articles and so you know your example of essentially saying that the external data could be far greater than the internal data out say we're already there okay and then the third sequential piece is trust are you gonna be able to trust the trust we talk a lot about we were down to Big Data NYC the same week you guys made your big announcement the data works everybody talks about data Lakes we joke gets the data swamp and can't really trust the data yeah we further away from a single version of the truth than we ever were so how are you dealing with that problem internally at IBM and what's the focus is it more on reporting is it more on supporting lines of business in product yeah the focus internal within IBM is in terms of driving cognition at the way I would describe it is at points where today we have significant human judgment being exercised to make decisions and that's you know thousands of points in our enterprise or complicated enterprise like IBM's and each of those decision points is actually an opportunity to inject cognitive technology and play and then bring to bear and augmented intelligence to those decisions that you know a factors in the exogenous data so leaving a much better informed decision but also them a much more accurate decision okay the two parallel activities let's start with the first one line of business you know relationships sounds like bromide why is it not just sort of a trite throwaway statement what where's the detail behind that so the detail behind that if you go back to the very first and the most important step and this whole thing with regard to the monetization strategy of the company understanding that if you don't have those deep relationships with the lines of business there's no way that you'll be able to understand the monetization strategy of the business so that's why that's a concurrent activity that has to start on day one otherwise you won't even get past the you know that that very first first base in terms of understanding what the monetization strategies are for the business and that can only really come by working directly with the business units meeting with their leadership understanding their business so you have to do that due diligence and that's where that partnership becomes critical then as you move on as you progress to that sequence you need them again so for instance once you understood the strategy and now you understood what data you need to follow that strategy and to govern it you need their help in governing the business because in many cases the businesses may be the ones collecting the data or at least controlling the source systems for that data so that partnership then just gets deeper and deeper and deeper as you move forward in that program I love the conscience of monetizing earlier and this some tweets going around you know what's holding it back cost of building it obviously and manageability but I want to bring that back and bring a developer perspective here because a lot of emphasis is on developing apps where the data is now part of the development process I wrote a blog post in 2008 saying that dated some new development kit radical at the time but reality it came out to be true and that they're looking at data as library of value to tap into so if stuffs annandale they could be sitting there for years but I could pull something out and be very relevant in context in real time and change the game on some insight and the insight economy is bob was saying so what is your strategy for IBM 21 on board more developer goodness and to how do you talk to customers were really trying to figure out a developer strategy so they can build apps and not to go back and rewrite it make it certainly mobile first etc but what's how does a date of first appt get built and I should developers be programming with you I'll give you a way to think about it right i mean and going back again to that ten-year paradigm shift right so ten years ago if somebody wanted to write an application and put it on the internet and it was based on data the hardest part was getting hold of the data because it was just very very difficult for them to get all of it to access the data and then those who did manage to get all of the data they were very successful in being able to utilize it so now with the the paradigm shift that's happened now is the approaches that you make the data available to developers and so they don't have to go through that work both in terms of accessing collecting finding that data then cleaning it it's also significant and so time consuming that it could put put back there their whole process of eventually getting to the app so to the extent that you have large stores of data that are ready to go and you can then make that available to a body of developers it just unleashes it's like having a library of code available is it all the hard work and I think that's a good way to look at it I mean that's think that's a very good way to look at it because you've also got technologies like the deep learning technologies where you can essentially train them with data so you don't need to write the code they get trained to later so I see a DevOps of data means like an agile meets I'm again you're right a lot of the cleaning and this is where you no more noise we all know that problem or data creates more noise better cleaning tools so however you can automate that yes seems to be the secret differentiator it's an accelerator it's amazing accelerator for development if you have good sets of data that are available for them to used so I want to round out my my little framework here your frame working with my my learnings for the fifth one being skills yes so this is complicated because it involves organization skills changes as pepper going through the lava here we try to get her on the cube Dave home to think the pamper okay babe yeah so should I take over pepper you want to go see pepper I want to see pepper on the cube hey sorry exact dress but so a lot of issues there there's reporting structures so what do you mean when you talk about sort of the skill sets and rescaling so and I'll describe to you a little bit about the organization that I have at IBM as an example some of that carries over and some of that doesn't the reason I say that is again I mean the skills piece there are some generic skill sets that you need for to be achieved data officer to be a successful chief data officer in an enterprise there is one pillar that I have in my organization is around data science data engineering DevOps deep learning and these are the folks who are adept at those technologies and approaches and methodologies and they can take those and apply them to the enterprise so in a sense these are the more technical people then another pillar that's again pretty generic and you have to have it is the information and data governance pillow so that anything that's flowing any data that's flowing through the data platform that I spoke off in the first pillar that those that that data is governed and fit for purpose so they have to worry about that as soon as any data is you even think of introducing that into the platform these folks have to be on that and they're essentially governing it making sure that people have the right access security the quality is good its improving there's a path to improving it and so forth I think those are some fairly generic you know skill sets that we have to get in the case of the first pillar what's difficult is that there aren't that many people with those skills and so it's hard to find that talent and so the sooner you get on it so that would that's the biggest barrier in the case of the second pillar what's the most difficult piece there is you need people who can walk the balance between monetization and governance too much governance and you essentially slow everything down and nothing moved a cuff and you're handcuffed and then you know if it's too much monetization you might run aground because you you ignored some major regulation so walking that loss of market value yeah that's what you have to really get ahead of your skis as they say and have a faceplant you'll try too hard to live boost mobile web startups like Twitter that's big cock rock concert with Twitter Facebook if you try to monetize too early yes you lose the flywheel effect of value absolutely so walking that balance is critical so that's that that's really finding the skill set to be able to do that that's that's what what's at play in that second or the third one is if you are applying it to an enterprise you have to integrate these you know this platform into the workflow off the enterprise itself otherwise you're not going to create any impact because that's where the impact gets created right that's basically where the data is that the tip of the spear to so to speak so you it's going to create value and in a large enterprise which has legacy systems which are silos which is acquiring companies and so on and so forth that's enough itself a significant job and that skill set is that's a handicapped because if you have that kind of siloed mentality you don't get the benefits of the data sharing right so what's that what's said how much how much effort would it take I'm just kind of painting that picture kind of like out there like well a lot of massively hard ya know that that's you know a lot of you know a lot of people think that data mining is all about my data you know this is my data I'm not going to give it to you the one of the functions of the chief data office is to change that mindset yeah and to stop making use of the data in a broader context than just a departmental siloed type of approach and now some data can legitimately be used only departmentally but the moment you need two or more department start using that data I mean it's essentially corporate data so are those roles a shared service everybody see that works it maybe varies but is it a shared service that reports into the chief data officer or is it embedded into the business those those skill sets that you talked about I think those skill sets are definitely part of the chief data officer you know organization now it's interesting you mentioned that about embedding them and the business units now in a in a large enterprise a complicated enterprise like IBM the different business units and that potentially have different business objectives and so forth you know you you do need a chief data officer role for each of these business units and that's something that I've been advocating that's my fault pillar and we are setting that up and then within the context of IBM so that they serve the business unit but they essentially reporting to me so that they can make use of the overall corporate structure you do their performance review the performance review is done by the business unit it is ok but the functional direction is given by me ok so I get back to still go either way oh yes that's a balance loon yeah absolutely under a lot of time for sure i'll get back to this data mining because you bring up a good point we can maybe continue on our next time we talk but data monies were all the cutting edge kind of best practices are were arsed work what we're relations are still there technically if you're here but that the dynamic of data mining is is that you're assuming no new data so with if you have a lot of data coming in most of the best data mining techniques are like a corpus you attack it and learned but if the pile of data is getting bigger faster that you could date a mine it what good is against or initial circular hole I'm going to again you know just take you back 10 years from now and now right and the differences between the two so it's very interesting points that you bring up I'll give you an example from 10 years ago this data mining example not ten years ago actually my first go-around at IBM so it's like 94 yeah one of the things I've done was we had a program a computer program that every team in the National Basketball Association started using and this was a classic data mining program it would look at the data and find insights and present them and one of the insights that it came up with and this was for a critical playoff game it told the coach you got to play your backup point guard and your backup forward now think about that which same coach would actually go with that so it's very hard for them to believe that they don't know if it's right or wrong in my own insurance and the way we got around that was we essentially pointed back to the snippets of video where those circumstances occurred and now the coach could see what is going on make a you know an informed decision flash forward to now the systems we have now can actually look at all that context all at once what's happening in the video what's happening in the audio also the data can piece together the context so data mining is very different today than what it was them now it's all about weaving the context and the story together and serving it up yeah what happened what's happening and what's going to happen kinda is the theaters of yes there are in sight writing what happened it's easy just yeah look at the data and spit out some insight what's happening now is a bit harder in memory I think that's the difference between cognition as it away versus data mining as you know we understood a few years ago great cartridge we can go for another hour but do we ever get enough love to follow up on some of the deep learning maybe come down to armonk next time we're in this certainly on the sports data we have a whole program on sports data so we love the sports with the ESPN of tech and bringing you all the action right here yes I did Doug before Moneyball you know my mistake was letting right yeah yeah right the next algorithm but that's okay you know we put a little foot mark on the cube notes for that thank you very much thank you appreciate okay live in Mandalay Bay we're right back with more live coverage I'm Sean for a table on thing great back today I am helping people

hi from Las Vegas Nevada it's the cube covering IBM world of Watson 2016 brought to you by IBM now here are your hosts John Fourier as Dave Volante hey welcome back everyone we are here live at the Mandalay Bay at the IBM world of Watson this is Silicon angles cube our flagship program we go out to the events and extract the signal from the noise I'm John Fourier with my co-host Dave allanté for the two days of wall-to-wall coverage our next guest is michelle fools so who's the chief marketing officer for IBM knew the company fairly new within the past year yes welcome to the queue last month I think you check all these new hires a lot of new blood coming inside me but this is a theme we heard from Staples to be agile to be fast you're new what's what's your impressions and what's your mandate for the branding the IBM strong brand but yes what's the future look well look I'm I'm thrilled to be here and I'm thrilled to be here because this is an extraordinary company that makes real difference in the world right and that I think you feel it here at the world of Watson in the sort of everyday ways that Watson and IBM touches consumers such as end-users makes their health better you know allows them to have greater experiences so so that's incredible to be part of my kind of company having said that and exactly to your point it's a time of acceleration and change for everyone in IBM is not immune to that and so my mandate here in my remit here and coming in and being a huge fan of what IBM has to say well how do we sharpen our messaging how do we always feel like a challenger brand you know how do we think about what Watson can do for people what the cloud can do what our services business can do and how is that distinctive and differentiated from everybody else out there and I think we have an incredible amount of assets to play with that's got to be through the line you know it's no longer the case that we can have a message on TV and that you know attracts the world the digital experiences are having every single day when they're clicking through on an ad when they're chatting with somebody when their car call center when they have a sales interaction is that differentiated message that brand resident all the way through second thing is marketing's become much more of a science you know and that to me is super exciting I've been a CEO most of my career and you know that the notion that marketing has to drive revenue that marketing has to drive retention and loyalty and expansion that we can come to the table with much more science in terms of what things are most effective in making sure that more clients love us more deeply for longer I'm gonna ask you the question because we had we've had many conversations with Kevin he was just here he was on last year Bob Lord the new chief digital officer we talked to your customers kind of the proof points in today's market is about transparency and if you're not a digital company how could you expect customers to to work with them so this has been a big theme for IBM you guys are hyper focused on being a digital company yes yes and how does it affect the brand a brand contract with the users what's your thoughts on that well first of all Bob Lord is awesome we've known each other for 10 years so it's so wonderful to be working with him again and Dave Kenny as well I think that the at the end of the day consumers have experiences and and you know think of every business you know out there as a consumer and they're having experiences all the time their expectations are being shaped by the fact that they go on Amazon and get prime delivery right their expectations are being shaped by they can go on Netflix and get you know personalized recommendations for them or Spotify and so our job of course and we have some of the greatest technical minds in the world it's to make sure that every experience lines up with the highest of their expectations and so much of that is digital and so my passion my background is entirely in the digital space I have a CEO of Travelocity and then CEO of gilt chief marketing a digital officer at Citigroup so the notion that you know the world's greatest digital experiences is something I'm very passionate about you mentioned Zelda so big TV ads and you think of the smarter planet which was so effective but it was a big TV campaign so you do what's the what's the sort of strategy that you're envisioning is in sort of digital breadcrumbs maybe you could talk about deadly yeah well think about Watson it's a perfect place to think about the Watson branding what does Watson really mean right Watson is and Ginni has said this so well of course it's cognitive and but at the end of the day it's about helping people make better decisions and so you can do some advertising with Watson and Bob Dylan and Watson and you know the young young girl with Serena and and you can get that messaging high but then you've got to bring it all the way through so that's why it's something like this is so powerful to see Woodside up their alley or all these companies talking about staples how they are using Watson embedded in their processes their tools to make their end-users experiences better and how nobody else could do this for them the way Watson's doing it that's taking a brand on high and advertising message on high and delivering value for businesses for patients for consumers all the way through that's what we have to do I got to ask you about that ad advertising trends I so we all see ad blocker in the news digital is a completely different new infrastructure expanded dynamic with social what not you can talk about Bob and I were talking last night about it too you Trevor you know banner ads are all out there impression base and then coded URLs to a landing page email marketing not gonna go away anytime soon but it's changing rapidly we have now new channels yeah what's your thoughts because this is now a new kind of ROI equation is there any thoughts on how you look at that and is it going to integrate into the top level campaigns how are you looking at the new digital that the cutting-edge digital stuff huge amounts of thoughts on this topic so I think you know if you think back 15 20 years ago there were always something called market mix modelling which helps advertisers and marketers to understand the effectiveness of their TV campaigns and frankly not too dissimilar from Nielsen you know there were so there was art and science at best in it and then all of a sudden the digital world evolved and you could get at a tactical level very very clear about attribution and whether you drove something and the challenge for us now is much more sophisticated models that are multi-touch attribution because the reality is an average consumer doesn't do one thing or have one interaction with a brand they're gonna see a TV show and watch a commercial while they're watching that commercial that business user or that end consumer is on their iPad or on their phone they're seeing a digital ad the next day at work they're being retargeted because they were aughts company they search for something they see a search campaign our job is to connect those dots and understand what really moves that consumer that business user to take an action and there are many sophisticated multi-touch attribution models where you model you know a standard set of behaviors and you test correlations against a bunch of different behaviors so you understand of what I did all the money I spent what really drove impact and by cohort I think that's the other credit there's no more the sense of sort of aggregated everything you really have to break it out yeah I didn't space my cohort to see what moves me and improve that experience right which has been you you get the example in the day of the Hilton retirees you already know that the retard the hotel was full so so obviously Watson plays a role in them Satyam plays a role in that so it's all about data it's all about you know that's where I think Watson can be extraordinarily helpful so if you think about the tool as a marketer has they're becoming more and more sophisticated and retargeting with something out of 10 years ago whenever was introduced that helped all of us a little bit and getting that message but it is only as good as the API is behind it and the the experience behind it when now when I was at gilt I was CEO of gilt we would put over a thousand products on sale every day that would be sold out by the next day sales down this 24-hour flash sale we had to get really really good at knowing how to how to retarget because last thing you want is to retarget something that sold out right or gone the next day and understand the user that was in and out and they're coming back and of course in that cohort that's where Watson to me is very exciting and you probably saw this in some of the demos of where Watson can help marketers you know where Watson can can really understand what are the drivers of behavior and what is likely to drive the highest purpose why were you so successful at guild and and how are the challenges different years because there's a sort of relatively more narrow community or city group to I was called the chief marketing and digital officer at Citigroup and and you know a tremendous budget and a lot of transactions you have to drive every day a lot of people you want to open credit cards and bank accounts so around the world I think that the the relentless focus on on marketing being art and science you know art and science and I think that's you know that passion for analytics passion for measurement having been CEO that passion for being able to say this is what we're doing and this is what we're driving so you've been kind of a data geek in your career you mentioned the financial services you can't to measure everything but back to the ad question you know the old saying used to be wasting half my advertise I just don't know which half yeah and my archives is wasted but now for the first time in the history of business in the modern era you measure everything online that's right so does that change your view and the prism of how you look at the business cuz you mentioned multi-touch yeah so now does that change the accountability for the suppliers I mean at agencies doing the big campaign I think it changes the game for all of us and there's no destination this is every day you can get better at optimizing your budget and and I would be the first to tell you as much of a sort of engineering and data geek because I've always been and deep-fried in the reality is there is art even in those attribution models what look back windows you choose etc that you know you're making decisions as a company but once you make those decisions you can start arraying all of your campaigns and saying what really moved the needle what was the most effective it's not an indictment that say what are we can do differently tomorrow you know the best marketers are always optimizing they're always figuring out at what point in the final can we get better tomorrow well in answer about talent because that's one of the things that we always talk about and also get your thoughts on Women in Technology scheme we were just at Grace Hopper last week and we started to fellowship called the tech truth and we're doing it's real passion area for us we have a site up QP 65 net / women in tech all women interviews we're really trying it the word out but this is now a big issue because now it's not stem anymore it's team arts is in there and we were also talking to the virtual reality augmented reality user experience is now potentially going to come into the immersion students and there's not enough artists yeah so you starting to see a combination of new discipline talents that are needed in the professions as well as the role of women in technology yeah your thoughts on that because this isn't you've been very successful what's your view on that at what's your thoughts about thank you for what you're doing right it takes a lot of people up there saying that this is important to make a difference so most of all thank you you know I think that this this is obviously a place I've been passion about forever I remember being a and being pregnant and that becoming this huge you know issue a news story and you're trying to juggle it right and how could a woman CEO be pregnant so it's so funny how people ridiculous took attention but but I think that the point is that the the advantage as a company has when there are great women in engineering and great women in data science and great women and user experience and design are just palpable they're probable in a variety of ways right when the team thinks differently the team is more creative the team is more open to new ideas the output for the customers are better right I mean they just saw a snapchat today just announced that in 2013 70% of their users were women so all the early adopters were women you know now it's balance but the early the early crowd were women and so we have got to figure out how to break some of the minds now I'm incredibly encouraged though while we still have a long way to go the numbers would suggest that we're having the conversation more and more and women are starting to see other women like them that they want to be it's a global narrative which is good why we're putting some journalists on there and funding it as and just as a fellowship because this it's a global story yeah okay and the power women I mean it's like there are real coders and this real talent coming in and the big theme that came out of that was is that 50% of the consumers of product are women's but therefore they should have some women features and related some vibe in there not just a male software driven concept well and should too when a powerful individual male individual like Satya steps in it and and you know understands what the mistaken and someone like refer to his speech two years ago where he said that you should just bad karma don't speak up and opening up transparency he got some heat yeah but that talk as you probably know but my opinion it's it's it's a positive step when an individual like that it was powerful and opening transparency within their company yeah that's it is that great networking I host a core I've been doing this for a year years with a good friend of mine Susan line from AOL we host a quarterly breakfast for women in tech every every quarter in New York City and we've been doing it for a long time it's amazing when those women come together the conversations we have the discussions we have how to help each other and support each other and so that's that's a real passion we were lost in a few weeks ago for the data science summit which Babu Chiana was hosting in and one of the folks was hosting the data divas breakfast we a couple there were a couple day two dudes who walked in and it was interesting yeah the perspectives 25 percent of the women or the chief data officer were women mm-hmm which was an interesting discussion as well so great 1,000 men at 15 you know as you see that techno but it's certainly changing when I get back to the mentoring thing because one of the things that we're all so passionate about is you've been a pioneer okay so now there's now an onboarding of new talent new personas new professions are being developed because we're seeing a new type of developer we're seeing new types of I would say artists becoming either CG so there's new tech careers that weren't around and a lot of the new jobs that are going to be coming online haven't even been invented yet right so you see cognition and what cognitive is enabling is a new application of skills yep can your thoughts on that because this is an onboarding opportunity so this could change the the number of percentage of women is diverse when you think about what I mean it's clear your notion of steam right your notion of stem that is a male and female phenomena and that is what this country needs it's what this world needs more of and so there's a policy and education obligation and all of us have to the next generation to say let's make sure we're doing right by them in terms of education and job opportunities when you think about onboarding I mean to me that the biggest thing about onboarding is the world is so much more interconnected than it used to be if you're a marketer it's not just art or science you have to do both it's a right brain left brain connectivity and I think 1020 years ago you could grow up in a discipline that was functional and maybe siloed and maybe you were great at left brain or great at right brain and the world demands so much more it's a faster pace it's an accelerated pace and the interconnection is critical and I've one of the things we're doing is we're putting together these diamond teams and I think it's going to really help lead the industry diamond teams are when you have on every small agile marketing team and analytics head a product marketing had a portfolio marketing had a design or a social expert these small pods that work on campaigns gone are the days that you could say designer designs it product comes up with the concept then it goes so it's design team then it goes to a production team then it goes to an analytics team we're forcing this issue by putting these teams together and saying you work together every day you'll get a good sense of where the specialty is and how you learn how to make your own discipline better because you've got the analytics person asked a question about media buying and media planning advertising as we're seeing this new real-time wet web yeah world mobile world go out the old days of planned media buyers placed the advertisement was a pacing item for execution yep now things you mentioned in the guild flash sales so now you're seeing new everyday flash opportunities to glob on to an opportunity to be engagement yeah and create a campaign on the fly yes and a vision of you guys I mean do you see that and does it change the cadence of how you guys do your execution of course of course that's one of the reasons we're moving to this diamond team and agile I think agile will ultimately be as impactful to marketing as it was to engineering and development and so I think the of course and that has to start with great modeling and great attribution because you have to know where things are performing so that you can iterate all the time I mean I believe in a world where you don't have marketing budgets and I know that sounds insane but I believe in a world where you set target and ranges on what you think you're gonna spend at the beginning of the year and every week like an accordion you're optimizing spend shipping code you've been marketing you should be doing like code so much of marketing is its episodic you boom and then it dies in a moment it's gone to the next one and you're talking about something that's I love that you know the personas to your point are much more fluid as well you got Millennials just creating their own vocations yes well this is where I think consumer companies have led the path and you know if you think about a lot of b2b companies we've had this aggregated CIO type buyer and now we've got to get much more sophisticated about what does the developer want you know what's important to the developer the messaging the tools the capabilities the user experience what about the marketer you know what the person in financial services and so both industry and professional discipline and you know schooling now with Watson you don't have to guess what they want you can actually just ask them yeah well you can actually the huge advantage you got you observe the observation space is now addressable right so you pull that in and say and that's super important even the stereotype of the persona is changing you've been saying all week that the developer is increasingly becoming business oriented maybe they don't they want they don't want to go back and get their MBA but they want to learn about capex versus op X and that's relevant to them and they to be a revolutionary you have to understand the impact right and and and they want to ship code they want to change the world I mean that is every engineering team I've ever worked at the time only worked with I mean I've been as close to engineering as from day one of the internet or early on in the internet great engineers are revolutionaries they want to change the world and they change the world they want to have a broader and broader understanding of what levers are at their disposal and I will say that I you know and I am one of the reasons I came to yam is I am passionate about this point technology cannot be in the hands of a few companies on the west coast who are trying to control and dominate the experience technology has to exist for all those amazing developers everywhere in the world who will make a difference to end user this is IBM strategy you actually have a big presence on the west coast also in Germany so you guys are going to where the action centers ours but not trying to just be so Malory point is what exactly because my point is IBM has always been there for making businesses stronger and better we don't monetize their data that's not our thing our thing is to use our cloud our cognitive capabilities and Watson to make actual businesses better so that ultimately consumers have better health care and better results I know you're new on the job silence this is not a trick question just kind of a more conversational as you talk to Bob lower Bob Chiana Jeanne yeah what's the promise of the brand and you used to be back in the days when you know Bob piano we talk about when we I worked at IBM in the 80s co-op student and it was you'll never get fired for buying IBM mainframe the kind of concept but it's evolved and I'll see we see a smarter plan what's the brand promise now you guys talk about what's the brainstorm on its head I think that I think the greatest innovators the world the most passionate business leaders of tomorrow come to IBM to make the world better and I I believe this is a brand for the forward the forward lookers the risk takers the you know the makers I think that you come to IBM because there's extraordinary assets and industry knowledge real humans real relationships that we exist to make your business better not our business will be a vibrato be exist to make your business better that has always been where IBM has been strong you know it's interesting that brings up a good point and just riffing on that Dave and I were just observing you know at the Grace Hopper with our tech truth mentorship which is promoting the intersection of Technology and social justice you're seeing that mission of Technology business value and social justice as an integral part of strategies because now the consumer access the consumerization of business yeah software based is now part of that feedback you're not doing good Millennials demand it I mean Millennials now when you look at the research in the next generation high Millennials are very very you know they want to know what are you doing for the world I mean who could do a 60 minute show besides IBM who could have who could be on 60 minutes changing cancer changing cancer outcomes for people beside IBM that that is an extraordinary testament to what the brand is and how it comes to life every day and that's important for Millennials we had Mary click-clack Clinton yesterday she is so impressive we're talking about how though these ozone layer is getting smaller these are us problems it can be solved they have to be so climate change can be solved so the whole getting the data and she's weather compass oh she's got a visit view on that is interesting her point is if we know what the problems are we as a community global society could actually solve them completely and it's an you know the more we make this a political and we say here is a problem and we have the data and we have the tools we have the people and capabilities to solve it that is where IBM Stan's tallest well I think with Watson use its focused on some big hairy problems to start with and now you're knocking off some some of the you know maybe more mundane but obviously significant to a marketer incredible that a company can start with the hardest most complicated problems the world has and actually make a difference my final question when I asked Mary this yesterday and she kind of talked about if she could have the magic Watson algorithm to just do something magical her and what would it be and she said I'll send Watson to the archives of all the weather data going back to World War two just compile it all and bring it back or addressability so the question is if you could have a Magic Watson algorithm for your chief marketing officer job what would you assign it to do like what would it be it's like first task well first of all reaction of course I'm a mom of six year olds an eight year old and so I want Watson to optimize my time no but a chief marketing officer I mean I think it really does go back to getting Watson's help in understanding how we use a dollar better how we use a dollar smarter how we affect more customers and and and connect connects with more customers in the way we you know we communicate the way we engage the way we've put our programs out that would be extraordinary and that's possible that's becoming more and more possible you know bringing science into the art of marketing I think will have great impact on what we're doing in also just the world I mean nobody wants to have you know maybe targeted ten times for something that's sold out well we asked one more time here so I got some more couple of questions because it's not getting the hook yet I gotta ask you see you mentioned Travelocity you know the web you've been through the web 1.2.0 yeah yeah so on so URLs and managing URLs was a great tracking mechanism from the old impressions weren't working and go to call to action get that look right there but now we different where that world is kind of like become critical infrastructure for managing technology since you're kind of geeking out with us here what's your view of the API economy because now apps don't use URLs they use tokens they use api's they use new push notification based stuff what sure how does api's change the marketing opportunities both right it's clearly changes the engineering environment and sort of opens up the world of possibilities in terms of who you partner with and how etc and I think it changes the marketing world too and entirely right you think about the API economy and the access you have to new ways of doing business new potential partnerships new ways of understanding data you know that that is absolutely you know at the fore of a lot of our thinking it might change the agency relationships to if they got to be more technical in changing as much as fast as companies are and they have to you know they are an extension they're your best you should be able to look in a room of agency and your team and not know who is who when you can tell who is who you have a problem and so agencies themselves have to become you know way more scientific harder-hitting faster pace and outcomes orient and somebody sees now are saying you know what pay me on outcomes I love that I love that mode to say we're in the boat with you pay me on outcome and the big s eyes are right there - absolutely yes Michele Palooza new chief marketing officer at IBM changing the game bring in some great mojo to IBM they're lucky to have you great conversations and thanks for coming on the cube live at Mandalay Bay this is silicon angles the cube I'm John four with Dave Volante be right back with more after this short break

(upbeat music beginning) >> From the Cube Studios in Palo Alto and Boston, bringing you data-driven insights from the Cube and ETR, this is "Breaking Analysis" with Dave Vellante. >> Making predictions about the future of enterprise tech is more challenging if you strive to lay down forecasts that are measurable. In other words, if you make a prediction, you should be able to look back a year later and say, with some degree of certainty, whether the prediction came true or not, with evidence to back that up. Hello and welcome to this week's Wikibon Cube Insights, powered by ETR. In this breaking analysis, we aim to do just that, with predictions about the macro IT spending environment, cost optimization, security, lots to talk about there, generative AI, cloud, and of course supercloud, blockchain adoption, data platforms, including commentary on Databricks, snowflake, and other key players, automation, events, and we may even have some bonus predictions around quantum computing, and perhaps some other areas. To make all this happen, we welcome back, for the third year in a row, my colleague and friend Eric Bradley from ETR. Eric, thanks for all you do for the community, and thanks for being part of this program. Again. >> I wouldn't miss it for the world. I always enjoy this one. Dave, good to see you. >> Yeah, so let me bring up this next slide and show you, actually come back to me if you would. I got to show the audience this. These are the inbounds that we got from PR firms starting in October around predictions. They know we do prediction posts. And so they'll send literally thousands and thousands of predictions from hundreds of experts in the industry, technologists, consultants, et cetera. And if you bring up the slide I can show you sort of the pattern that developed here. 40% of these thousands of predictions were from cyber. You had AI and data. If you combine those, it's still not close to cyber. Cost optimization was a big thing. Of course, cloud, some on DevOps, and software. Digital... Digital transformation got, you know, some lip service and SaaS. And then there was other, it's kind of around 2%. So quite remarkable, when you think about the focus on cyber, Eric. >> Yeah, there's two reasons why I think it makes sense, though. One, the cybersecurity companies have a lot of cash, so therefore the PR firms might be working a little bit harder for them than some of their other clients. (laughs) And then secondly, as you know, for multiple years now, when we do our macro survey, we ask, "What's your number one spending priority?" And again, it's security. It just isn't going anywhere. It just stays at the top. So I'm actually not that surprised by that little pie chart there, but I was shocked that SaaS was only 5%. You know, going back 10 years ago, that would've been the only thing anyone was talking about. >> Yeah. So true. All right, let's get into it. First prediction, we always start with kind of tech spending. Number one is tech spending increases between four and 5%. ETR has currently got it at 4.6% coming into 2023. This has been a consistently downward trend all year. We started, you know, much, much higher as we've been reporting. Bottom line is the fed is still in control. They're going to ease up on tightening, is the expectation, they're going to shoot for a soft landing. But you know, my feeling is this slingshot economy is going to continue, and it's going to continue to confound, whether it's supply chains or spending. The, the interesting thing about the ETR data, Eric, and I want you to comment on this, the largest companies are the most aggressive to cut. They're laying off, smaller firms are spending faster. They're actually growing at a much larger, faster rate as are companies in EMEA. And that's a surprise. That's outpacing the US and APAC. Chime in on this, Eric. >> Yeah, I was surprised on all of that. First on the higher level spending, we are definitely seeing it coming down, but the interesting thing here is headlines are making it worse. The huge research shop recently said 0% growth. We're coming in at 4.6%. And just so everyone knows, this is not us guessing, we asked 1,525 IT decision-makers what their budget growth will be, and they came in at 4.6%. Now there's a huge disparity, as you mentioned. The Fortune 500, global 2000, barely at 2% growth, but small, it's at 7%. So we're at a situation right now where the smaller companies are still playing a little bit of catch up on digital transformation, and they're spending money. The largest companies that have the most to lose from a recession are being more trepidatious, obviously. So they're playing a "Wait and see." And I hope we don't talk ourselves into a recession. Certainly the headlines and some of their research shops are helping it along. But another interesting comment here is, you know, energy and utilities used to be called an orphan and widow stock group, right? They are spending more than anyone, more than financials insurance, more than retail consumer. So right now it's being driven by mid, small, and energy and utilities. They're all spending like gangbusters, like nothing's happening. And it's the rest of everyone else that's being very cautious. >> Yeah, so very unpredictable right now. All right, let's go to number two. Cost optimization remains a major theme in 2023. We've been reporting on this. You've, we've shown a chart here. What's the primary method that your organization plans to use? You asked this question of those individuals that cited that they were going to reduce their spend and- >> Mhm. >> consolidating redundant vendors, you know, still leads the way, you know, far behind, cloud optimization is second, but it, but cloud continues to outpace legacy on-prem spending, no doubt. Somebody, it was, the guy's name was Alexander Feiglstorfer from Storyblok, sent in a prediction, said "All in one becomes extinct." Now, generally I would say I disagree with that because, you know, as we know over the years, suites tend to win out over, you know, individual, you know, point products. But I think what's going to happen is all in one is going to remain the norm for these larger companies that are cutting back. They want to consolidate redundant vendors, and the smaller companies are going to stick with that best of breed and be more aggressive and try to compete more effectively. What's your take on that? >> Yeah, I'm seeing much more consolidation in vendors, but also consolidation in functionality. We're seeing people building out new functionality, whether it's, we're going to talk about this later, so I don't want to steal too much of our thunder right now, but data and security also, we're seeing a functionality creep. So I think there's further consolidation happening here. I think niche solutions are going to be less likely, and platform solutions are going to be more likely in a spending environment where you want to reduce your vendors. You want to have one bill to pay, not 10. Another thing on this slide, real quick if I can before I move on, is we had a bunch of people write in and some of the answer options that aren't on this graph but did get cited a lot, unfortunately, is the obvious reduction in staff, hiring freezes, and delaying hardware, were three of the top write-ins. And another one was offshore outsourcing. So in addition to what we're seeing here, there were a lot of write-in options, and I just thought it would be important to state that, but essentially the cost optimization is by and far the highest one, and it's growing. So it's actually increased in our citations over the last year. >> And yeah, specifically consolidating redundant vendors. And so I actually thank you for bringing that other up, 'cause I had asked you, Eric, is there any evidence that repatriation is going on and we don't see it in the numbers, we don't see it even in the other, there was, I think very little or no mention of cloud repatriation, even though it might be happening in this in a smattering. >> Not a single mention, not one single mention. I went through it for you. Yep. Not one write-in. >> All right, let's move on. Number three, security leads M&A in 2023. Now you might say, "Oh, well that's a layup," but let me set this up Eric, because I didn't really do a great job with the slide. I hid the, what you've done, because you basically took, this is from the emerging technology survey with 1,181 responses from November. And what we did is we took Palo Alto and looked at the overlap in Palo Alto Networks accounts with these vendors that were showing on this chart. And Eric, I'm going to ask you to explain why we put a circle around OneTrust, but let me just set it up, and then have you comment on the slide and take, give us more detail. We're seeing private company valuations are off, you know, 10 to 40%. We saw a sneak, do a down round, but pretty good actually only down 12%. We've seen much higher down rounds. Palo Alto Networks we think is going to get busy. Again, they're an inquisitive company, they've been sort of quiet lately, and we think CrowdStrike, Cisco, Microsoft, Zscaler, we're predicting all of those will make some acquisitions and we're thinking that the targets are somewhere in this mess of security taxonomy. Other thing we're predicting AI meets cyber big time in 2023, we're going to probably going to see some acquisitions of those companies that are leaning into AI. We've seen some of that with Palo Alto. And then, you know, your comment to me, Eric, was "The RSA conference is going to be insane, hopping mad, "crazy this April," (Eric laughing) but give us your take on this data, and why the red circle around OneTrust? Take us back to that slide if you would, Alex. >> Sure. There's a few things here. First, let me explain what we're looking at. So because we separate the public companies and the private companies into two separate surveys, this allows us the ability to cross-reference that data. So what we're doing here is in our public survey, the tesis, everyone who cited some spending with Palo Alto, meaning they're a Palo Alto customer, we then cross-reference that with the private tech companies. Who also are they spending with? So what you're seeing here is an overlap. These companies that we have circled are doing the best in Palo Alto's accounts. Now, Palo Alto went and bought Twistlock a few years ago, which this data slide predicted, to be quite honest. And so I don't know if they necessarily are going to go after Snyk. Snyk, sorry. They already have something in that space. What they do need, however, is more on the authentication space. So I'm looking at OneTrust, with a 45% overlap in their overall net sentiment. That is a company that's already existing in their accounts and could be very synergistic to them. BeyondTrust as well, authentication identity. This is something that Palo needs to do to move more down that zero trust path. Now why did I pick Palo first? Because usually they're very inquisitive. They've been a little quiet lately. Secondly, if you look at the backdrop in the markets, the IPO freeze isn't going to last forever. Sooner or later, the IPO markets are going to open up, and some of these private companies are going to tap into public equity. In the meantime, however, cash funding on the private side is drying up. If they need another round, they're not going to get it, and they're certainly not going to get it at the valuations they were getting. So we're seeing valuations maybe come down where they're a touch more attractive, and Palo knows this isn't going to last forever. Cisco knows that, CrowdStrike, Zscaler, all these companies that are trying to make a push to become that vendor that you're consolidating in, around, they have a chance now, they have a window where they need to go make some acquisitions. And that's why I believe leading up to RSA, we're going to see some movement. I think it's going to pretty, a really exciting time in security right now. >> Awesome. Thank you. Great explanation. All right, let's go on the next one. Number four is, it relates to security. Let's stay there. Zero trust moves from hype to reality in 2023. Now again, you might say, "Oh yeah, that's a layup." A lot of these inbounds that we got are very, you know, kind of self-serving, but we always try to put some meat in the bone. So first thing we do is we pull out some commentary from, Eric, your roundtable, your insights roundtable. And we have a CISO from a global hospitality firm says, "For me that's the highest priority." He's talking about zero trust because it's the best ROI, it's the most forward-looking, and it enables a lot of the business transformation activities that we want to do. CISOs tell me that they actually can drive forward transformation projects that have zero trust, and because they can accelerate them, because they don't have to go through the hurdle of, you know, getting, making sure that it's secure. Second comment, zero trust closes that last mile where once you're authenticated, they open up the resource to you in a zero trust way. That's a CISO of a, and a managing director of a cyber risk services enterprise. Your thoughts on this? >> I can be here all day, so I'm going to try to be quick on this one. This is not a fluff piece on this one. There's a couple of other reasons this is happening. One, the board finally gets it. Zero trust at first was just a marketing hype term. Now the board understands it, and that's why CISOs are able to push through it. And what they finally did was redefine what it means. Zero trust simply means moving away from hardware security, moving towards software-defined security, with authentication as its base. The board finally gets that, and now they understand that this is necessary and it's being moved forward. The other reason it's happening now is hybrid work is here to stay. We weren't really sure at first, large companies were still trying to push people back to the office, and it's going to happen. The pendulum will swing back, but hybrid work's not going anywhere. By basically on our own data, we're seeing that 69% of companies expect remote and hybrid to be permanent, with only 30% permanent in office. Zero trust works for a hybrid environment. So all of that is the reason why this is happening right now. And going back to our previous prediction, this is why we're picking Palo, this is why we're picking Zscaler to make these acquisitions. Palo Alto needs to be better on the authentication side, and so does Zscaler. They're both fantastic on zero trust network access, but they need the authentication software defined aspect, and that's why we think this is going to happen. One last thing, in that CISO round table, I also had somebody say, "Listen, Zscaler is incredible. "They're doing incredibly well pervading the enterprise, "but their pricing's getting a little high," and they actually think Palo Alto is well-suited to start taking some of that share, if Palo can make one move. >> Yeah, Palo Alto's consolidation story is very strong. Here's my question and challenge. Do you and me, so I'm always hardcore about, okay, you've got to have evidence. I want to look back at these things a year from now and say, "Did we get it right? Yes or no?" If we got it wrong, we'll tell you we got it wrong. So how are we going to measure this? I'd say a couple things, and you can chime in. One is just the number of vendors talking about it. That's, but the marketing always leads the reality. So the second part of that is we got to get evidence from the buying community. Can you help us with that? >> (laughs) Luckily, that's what I do. I have a data company that asks thousands of IT decision-makers what they're adopting and what they're increasing spend on, as well as what they're decreasing spend on and what they're replacing. So I have snapshots in time over the last 11 years where I can go ahead and compare and contrast whether this adoption is happening or not. So come back to me in 12 months and I'll let you know. >> Now, you know, I will. Okay, let's bring up the next one. Number five, generative AI hits where the Metaverse missed. Of course everybody's talking about ChatGPT, we just wrote last week in a breaking analysis with John Furrier and Sarjeet Joha our take on that. We think 2023 does mark a pivot point as natural language processing really infiltrates enterprise tech just as Amazon turned the data center into an API. We think going forward, you're going to be interacting with technology through natural language, through English commands or other, you know, foreign language commands, and investors are lining up, all the VCs are getting excited about creating something competitive to ChatGPT, according to (indistinct) a hundred million dollars gets you a seat at the table, gets you into the game. (laughing) That's before you have to start doing promotion. But he thinks that's what it takes to actually create a clone or something equivalent. We've seen stuff from, you know, the head of Facebook's, you know, AI saying, "Oh, it's really not that sophisticated, ChatGPT, "it's kind of like IBM Watson, it's great engineering, "but you know, we've got more advanced technology." We know Google's working on some really interesting stuff. But here's the thing. ETR just launched this survey for the February survey. It's in the field now. We circle open AI in this category. They weren't even in the survey, Eric, last quarter. So 52% of the ETR survey respondents indicated a positive sentiment toward open AI. I added up all the sort of different bars, we could double click on that. And then I got this inbound from Scott Stevenson of Deep Graham. He said "AI is recession-proof." I don't know if that's the case, but it's a good quote. So bring this back up and take us through this. Explain this chart for us, if you would. >> First of all, I like Scott's quote better than the Facebook one. I think that's some sour grapes. Meta just spent an insane amount of money on the Metaverse and that's a dud. Microsoft just spent money on open AI and it is hot, undoubtedly hot. We've only been in the field with our current ETS survey for a week. So my caveat is it's preliminary data, but I don't care if it's preliminary data. (laughing) We're getting a sneak peek here at what is the number one net sentiment and mindshare leader in the entire machine-learning AI sector within a week. It's beating Data- >> 600. 600 in. >> It's beating Databricks. And we all know Databricks is a huge established enterprise company, not only in machine-learning AI, but it's in the top 10 in the entire survey. We have over 400 vendors in this survey. It's number eight overall, already. In a week. This is not hype. This is real. And I could go on the NLP stuff for a while. Not only here are we seeing it in open AI and machine-learning and AI, but we're seeing NLP in security. It's huge in email security. It's completely transforming that area. It's one of the reasons I thought Palo might take Abnormal out. They're doing such a great job with NLP in this email side, and also in the data prep tools. NLP is going to take out data prep tools. If we have time, I'll discuss that later. But yeah, this is, to me this is a no-brainer, and we're already seeing it in the data. >> Yeah, John Furrier called, you know, the ChatGPT introduction. He said it reminded him of the Netscape moment, when we all first saw Netscape Navigator and went, "Wow, it really could be transformative." All right, number six, the cloud expands to supercloud as edge computing accelerates and CloudFlare is a big winner in 2023. We've reported obviously on cloud, multi-cloud, supercloud and CloudFlare, basically saying what multi-cloud should have been. We pulled this quote from Atif Kahn, who is the founder and CTO of Alkira, thanks, one of the inbounds, thank you. "In 2023, highly distributed IT environments "will become more the norm "as organizations increasingly deploy hybrid cloud, "multi-cloud and edge settings..." Eric, from one of your round tables, "If my sources from edge computing are coming "from the cloud, that means I have my workloads "running in the cloud. "There is no one better than CloudFlare," That's a senior director of IT architecture at a huge financial firm. And then your analysis shows CloudFlare really growing in pervasion, that sort of market presence in the dataset, dramatically, to near 20%, leading, I think you had told me that they're even ahead of Google Cloud in terms of momentum right now. >> That was probably the biggest shock to me in our January 2023 tesis, which covers the public companies in the cloud computing sector. CloudFlare has now overtaken GCP in overall spending, and I was shocked by that. It's already extremely pervasive in networking, of course, for the edge networking side, and also in security. This is the number one leader in SaaSi, web access firewall, DDoS, bot protection, by your definition of supercloud, which we just did a couple of weeks ago, and I really enjoyed that by the way Dave, I think CloudFlare is the one that fits your definition best, because it's bringing all of these aspects together, and most importantly, it's cloud agnostic. It does not need to rely on Azure or AWS to do this. It has its own cloud. So I just think it's, when we look at your definition of supercloud, CloudFlare is the poster child. >> You know, what's interesting about that too, is a lot of people are poo-pooing CloudFlare, "Ah, it's, you know, really kind of not that sophisticated." "You don't have as many tools," but to your point, you're can have those tools in the cloud, Cloudflare's doing serverless on steroids, trying to keep things really simple, doing a phenomenal job at, you know, various locations around the world. And they're definitely one to watch. Somebody put them on my radar (laughing) a while ago and said, "Dave, you got to do a breaking analysis on CloudFlare." And so I want to thank that person. I can't really name them, 'cause they work inside of a giant hyperscaler. But- (Eric laughing) (Dave chuckling) >> Real quickly, if I can from a competitive perspective too, who else is there? They've already taken share from Akamai, and Fastly is their really only other direct comp, and they're not there. And these guys are in poll position and they're the only game in town right now. I just, I don't see it slowing down. >> I thought one of your comments from your roundtable I was reading, one of the folks said, you know, CloudFlare, if my workloads are in the cloud, they are, you know, dominant, they said not as strong with on-prem. And so Akamai is doing better there. I'm like, "Okay, where would you want to be?" (laughing) >> Yeah, which one of those two would you rather be? >> Right? Anyway, all right, let's move on. Number seven, blockchain continues to look for a home in the enterprise, but devs will slowly begin to adopt in 2023. You know, blockchains have got a lot of buzz, obviously crypto is, you know, the killer app for blockchain. Senior IT architect in financial services from your, one of your insight roundtables said quote, "For enterprises to adopt a new technology, "there have to be proven turnkey solutions. "My experience in talking with my peers are, "blockchain is still an open-source component "where you have to build around it." Now I want to thank Ravi Mayuram, who's the CTO of Couchbase sent in, you know, one of the predictions, he said, "DevOps will adopt blockchain, specifically Ethereum." And he referenced actually in his email to me, Solidity, which is the programming language for Ethereum, "will be in every DevOps pro's playbook, "mirroring the boom in machine-learning. "Newer programming languages like Solidity "will enter the toolkits of devs." His point there, you know, Solidity for those of you don't know, you know, Bitcoin is not programmable. Solidity, you know, came out and that was their whole shtick, and they've been improving that, and so forth. But it, Eric, it's true, it really hasn't found its home despite, you know, the potential for smart contracts. IBM's pushing it, VMware has had announcements, and others, really hasn't found its way in the enterprise yet. >> Yeah, and I got to be honest, I don't think it's going to, either. So when we did our top trends series, this was basically chosen as an anti-prediction, I would guess, that it just continues to not gain hold. And the reason why was that first comment, right? It's very much a niche solution that requires a ton of custom work around it. You can't just plug and play it. And at the end of the day, let's be very real what this technology is, it's a database ledger, and we already have database ledgers in the enterprise. So why is this a priority to move to a different database ledger? It's going to be very niche cases. I like the CTO comment from Couchbase about it being adopted by DevOps. I agree with that, but it has to be a DevOps in a very specific use case, and a very sophisticated use case in financial services, most likely. And that's not across the entire enterprise. So I just think it's still going to struggle to get its foothold for a little bit longer, if ever. >> Great, thanks. Okay, let's move on. Number eight, AWS Databricks, Google Snowflake lead the data charge with Microsoft. Keeping it simple. So let's unpack this a little bit. This is the shared accounts peer position for, I pulled data platforms in for analytics, machine-learning and AI and database. So I could grab all these accounts or these vendors and see how they compare in those three sectors. Analytics, machine-learning and database. Snowflake and Databricks, you know, they're on a crash course, as you and I have talked about. They're battling to be the single source of truth in analytics. They're, there's going to be a big focus. They're already started. It's going to be accelerated in 2023 on open formats. Iceberg, Python, you know, they're all the rage. We heard about Iceberg at Snowflake Summit, last summer or last June. Not a lot of people had heard of it, but of course the Databricks crowd, who knows it well. A lot of other open source tooling. There's a company called DBT Labs, which you're going to talk about in a minute. George Gilbert put them on our radar. We just had Tristan Handy, the CEO of DBT labs, on at supercloud last week. They are a new disruptor in data that's, they're essentially making, they're API-ifying, if you will, KPIs inside the data warehouse and dramatically simplifying that whole data pipeline. So really, you know, the ETL guys should be shaking in their boots with them. Coming back to the slide. Google really remains focused on BigQuery adoption. Customers have complained to me that they would like to use Snowflake with Google's AI tools, but they're being forced to go to BigQuery. I got to ask Google about that. AWS continues to stitch together its bespoke data stores, that's gone down that "Right tool for the right job" path. David Foyer two years ago said, "AWS absolutely is going to have to solve that problem." We saw them start to do it in, at Reinvent, bringing together NoETL between Aurora and Redshift, and really trying to simplify those worlds. There's going to be more of that. And then Microsoft, they're just making it cheap and easy to use their stuff, you know, despite some of the complaints that we hear in the community, you know, about things like Cosmos, but Eric, your take? >> Yeah, my concern here is that Snowflake and Databricks are fighting each other, and it's allowing AWS and Microsoft to kind of catch up against them, and I don't know if that's the right move for either of those two companies individually, Azure and AWS are building out functionality. Are they as good? No they're not. The other thing to remember too is that AWS and Azure get paid anyway, because both Databricks and Snowflake run on top of 'em. So (laughing) they're basically collecting their toll, while these two fight it out with each other, and they build out functionality. I think they need to stop focusing on each other, a little bit, and think about the overall strategy. Now for Databricks, we know they came out first as a machine-learning AI tool. They were known better for that spot, and now they're really trying to play catch-up on that data storage compute spot, and inversely for Snowflake, they were killing it with the compute separation from storage, and now they're trying to get into the MLAI spot. I actually wouldn't be surprised to see them make some sort of acquisition. Frank Slootman has been a little bit quiet, in my opinion there. The other thing to mention is your comment about DBT Labs. If we look at our emerging technology survey, last survey when this came out, DBT labs, number one leader in that data integration space, I'm going to just pull it up real quickly. It looks like they had a 33% overall net sentiment to lead data analytics integration. So they are clearly growing, it's fourth straight survey consecutively that they've grown. The other name we're seeing there a little bit is Cribl, but DBT labs is by far the number one player in this space. >> All right. Okay, cool. Moving on, let's go to number nine. With Automation mixer resurgence in 2023, we're showing again data. The x axis is overlap or presence in the dataset, and the vertical axis is shared net score. Net score is a measure of spending momentum. As always, you've seen UI path and Microsoft Power Automate up until the right, that red line, that 40% line is generally considered elevated. UI path is really separating, creating some distance from Automation Anywhere, they, you know, previous quarters they were much closer. Microsoft Power Automate came on the scene in a big way, they loom large with this "Good enough" approach. I will say this, I, somebody sent me a results of a (indistinct) survey, which showed UiPath actually had more mentions than Power Automate, which was surprising, but I think that's not been the case in the ETR data set. We're definitely seeing a shift from back office to front soft office kind of workloads. Having said that, software testing is emerging as a mainstream use case, we're seeing ML and AI become embedded in end-to-end automations, and low-code is serving the line of business. And so this, we think, is going to increasingly have appeal to organizations in the coming year, who want to automate as much as possible and not necessarily, we've seen a lot of layoffs in tech, and people... You're going to have to fill the gaps with automation. That's a trend that's going to continue. >> Yep, agreed. At first that comment about Microsoft Power Automate having less citations than UiPath, that's shocking to me. I'm looking at my chart right here where Microsoft Power Automate was cited by over 60% of our entire survey takers, and UiPath at around 38%. Now don't get me wrong, 38% pervasion's fantastic, but you know you're not going to beat an entrenched Microsoft. So I don't really know where that comment came from. So UiPath, looking at it alone, it's doing incredibly well. It had a huge rebound in its net score this last survey. It had dropped going through the back half of 2022, but we saw a big spike in the last one. So it's got a net score of over 55%. A lot of people citing adoption and increasing. So that's really what you want to see for a name like this. The problem is that just Microsoft is doing its playbook. At the end of the day, I'm going to do a POC, why am I going to pay more for UiPath, or even take on another separate bill, when we know everyone's consolidating vendors, if my license already includes Microsoft Power Automate? It might not be perfect, it might not be as good, but what I'm hearing all the time is it's good enough, and I really don't want another invoice. >> Right. So how does UiPath, you know, and Automation Anywhere, how do they compete with that? Well, the way they compete with it is they got to have a better product. They got a product that's 10 times better. You know, they- >> Right. >> they're not going to compete based on where the lowest cost, Microsoft's got that locked up, or where the easiest to, you know, Microsoft basically give it away for free, and that's their playbook. So that's, you know, up to UiPath. UiPath brought on Rob Ensslin, I've interviewed him. Very, very capable individual, is now Co-CEO. So he's kind of bringing that adult supervision in, and really tightening up the go to market. So, you know, we know this company has been a rocket ship, and so getting some control on that and really getting focused like a laser, you know, could be good things ahead there for that company. Okay. >> One of the problems, if I could real quick Dave, is what the use cases are. When we first came out with RPA, everyone was super excited about like, "No, UiPath is going to be great for super powerful "projects, use cases." That's not what RPA is being used for. As you mentioned, it's being used for mundane tasks, so it's not automating complex things, which I think UiPath was built for. So if you were going to get UiPath, and choose that over Microsoft, it's going to be 'cause you're doing it for more powerful use case, where it is better. But the problem is that's not where the enterprise is using it. The enterprise are using this for base rote tasks, and simply, Microsoft Power Automate can do that. >> Yeah, it's interesting. I've had people on theCube that are both Microsoft Power Automate customers and UiPath customers, and I've asked them, "Well you know, "how do you differentiate between the two?" And they've said to me, "Look, our users and personal productivity users, "they like Power Automate, "they can use it themselves, and you know, "it doesn't take a lot of, you know, support on our end." The flip side is you could do that with UiPath, but like you said, there's more of a focus now on end-to-end enterprise automation and building out those capabilities. So it's increasingly a value play, and that's going to be obviously the challenge going forward. Okay, my last one, and then I think you've got some bonus ones. Number 10, hybrid events are the new category. Look it, if I can get a thousand inbounds that are largely self-serving, I can do my own here, 'cause we're in the events business. (Eric chuckling) Here's the prediction though, and this is a trend we're seeing, the number of physical events is going to dramatically increase. That might surprise people, but most of the big giant events are going to get smaller. The exception is AWS with Reinvent, I think Snowflake's going to continue to grow. So there are examples of physical events that are growing, but generally, most of the big ones are getting smaller, and there's going to be many more smaller intimate regional events and road shows. These micro-events, they're going to be stitched together. Digital is becoming a first class citizen, so people really got to get their digital acts together, and brands are prioritizing earned media, and they're beginning to build their own news networks, going direct to their customers. And so that's a trend we see, and I, you know, we're right in the middle of it, Eric, so you know we're going to, you mentioned RSA, I think that's perhaps going to be one of those crazy ones that continues to grow. It's shrunk, and then it, you know, 'cause last year- >> Yeah, it did shrink. >> right, it was the last one before the pandemic, and then they sort of made another run at it last year. It was smaller but it was very vibrant, and I think this year's going to be huge. Global World Congress is another one, we're going to be there end of Feb. That's obviously a big big show, but in general, the brands and the technology vendors, even Oracle is going to scale down. I don't know about Salesforce. We'll see. You had a couple of bonus predictions. Quantum and maybe some others? Bring us home. >> Yeah, sure. I got a few more. I think we touched upon one, but I definitely think the data prep tools are facing extinction, unfortunately, you know, the Talons Informatica is some of those names. The problem there is that the BI tools are kind of including data prep into it already. You know, an example of that is Tableau Prep Builder, and then in addition, Advanced NLP is being worked in as well. ThoughtSpot, Intelius, both often say that as their selling point, Tableau has Ask Data, Click has Insight Bot, so you don't have to really be intelligent on data prep anymore. A regular business user can just self-query, using either the search bar, or even just speaking into what it needs, and these tools are kind of doing the data prep for it. I don't think that's a, you know, an out in left field type of prediction, but it's the time is nigh. The other one I would also state is that I think knowledge graphs are going to break through this year. Neo4j in our survey is growing in pervasion in Mindshare. So more and more people are citing it, AWS Neptune's getting its act together, and we're seeing that spending intentions are growing there. Tiger Graph is also growing in our survey sample. I just think that the time is now for knowledge graphs to break through, and if I had to do one more, I'd say real-time streaming analytics moves from the very, very rich big enterprises to downstream, to more people are actually going to be moving towards real-time streaming, again, because the data prep tools and the data pipelines have gotten easier to use, and I think the ROI on real-time streaming is obviously there. So those are three that didn't make the cut, but I thought deserved an honorable mention. >> Yeah, I'm glad you did. Several weeks ago, we did an analyst prediction roundtable, if you will, a cube session power panel with a number of data analysts and that, you know, streaming, real-time streaming was top of mind. So glad you brought that up. Eric, as always, thank you very much. I appreciate the time you put in beforehand. I know it's been crazy, because you guys are wrapping up, you know, the last quarter survey in- >> Been a nuts three weeks for us. (laughing) >> job. I love the fact that you're doing, you know, the ETS survey now, I think it's quarterly now, right? Is that right? >> Yep. >> Yep. So that's phenomenal. >> Four times a year. I'll be happy to jump on with you when we get that done. I know you were really impressed with that last time. >> It's unbelievable. This is so much data at ETR. Okay. Hey, that's a wrap. Thanks again. >> Take care Dave. Good seeing you. >> All right, many thanks to our team here, Alex Myerson as production, he manages the podcast force. Ken Schiffman as well is a critical component of our East Coast studio. Kristen Martin and Cheryl Knight help get the word out on social media and in our newsletters. And Rob Hoof is our editor-in-chief. He's at siliconangle.com. He's just a great editing for us. Thank you all. Remember all these episodes that are available as podcasts, wherever you listen, podcast is doing great. Just search "Breaking analysis podcast." Really appreciate you guys listening. I publish each week on wikibon.com and siliconangle.com, or you can email me directly if you want to get in touch, david.vellante@siliconangle.com. That's how I got all these. I really appreciate it. I went through every single one with a yellow highlighter. It took some time, (laughing) but I appreciate it. You could DM me at dvellante, or comment on our LinkedIn post and please check out etr.ai. Its data is amazing. Best survey data in the enterprise tech business. This is Dave Vellante for theCube Insights, powered by ETR. Thanks for watching, and we'll see you next time on "Breaking Analysis." (upbeat music beginning) (upbeat music ending)

>> From theCUBE Studios in Palo Alto in Boston, bringing you data-driven insights from theCUBE and ETR, this is "Breaking Analysis" with Dave Vellante. >> A decade of big data investments combined with cloud scale, the rise of much more cost effective processing power. And the introduction of advanced tooling has catapulted machine intelligence to the forefront of technology investments. No matter what job you have, your operation will be AI powered within five years and machines may actually even be doing your job. Artificial intelligence is being infused into applications, infrastructure, equipment, and virtually every aspect of our lives. AI is proving to be extremely helpful at things like controlling vehicles, speeding up medical diagnoses, processing language, advancing science, and generally raising the stakes on what it means to apply technology for business advantage. But business value realization has been a challenge for most organizations due to lack of skills, complexity of programming models, immature technology integration, sizable upfront investments, ethical concerns, and lack of business alignment. Mastering AI technology will not be a requirement for success in our view. However, figuring out how and where to apply AI to your business will be crucial. That means understanding the business case, picking the right technology partner, experimenting in bite-sized chunks, and quickly identifying winners to double down on from an investment standpoint. Hello and welcome to this week's Wiki-bond CUBE Insights powered by ETR. In this breaking analysis, we update you on the state of AI and what it means for the competition. And to do so, we invite into our studios Andy Thurai of Constellation Research. Andy covers AI deeply. He knows the players, he knows the pitfalls of AI investment, and he's a collaborator. Andy, great to have you on the program. Thanks for coming into our CUBE studios. >> Thanks for having me on. >> You're very welcome. Okay, let's set the table with a premise and a series of assertions we want to test with Andy. I'm going to lay 'em out. And then Andy, I'd love for you to comment. So, first of all, according to McKinsey, AI adoption has more than doubled since 2017, but only 10% of organizations report seeing significant ROI. That's a BCG and MIT study. And part of that challenge of AI is it requires data, is requires good data, data proficiency, which is not trivial, as you know. Firms that can master both data and AI, we believe are going to have a competitive advantage this decade. Hyperscalers, as we show you dominate AI and ML. We'll show you some data on that. And having said that, there's plenty of room for specialists. They need to partner with the cloud vendors for go to market productivity. And finally, organizations increasingly have to put data and AI at the center of their enterprises. And to do that, most are going to rely on vendor R&D to leverage AI and ML. In other words, Andy, they're going to buy it and apply it as opposed to build it. What are your thoughts on that setup and that premise? >> Yeah, I see that a lot happening in the field, right? So first of all, the only 10% of realizing a return on investment. That's so true because we talked about this earlier, the most companies are still in the innovation cycle. So they're trying to innovate and see what they can do to apply. A lot of these times when you look at the solutions, what they come up with or the models they create, the experimentation they do, most times they don't even have a good business case to solve, right? So they just experiment and then they figure it out, "Oh my God, this model is working. Can we do something to solve it?" So it's like you found a hammer and then you're trying to find the needle kind of thing, right? That never works. >> 'Cause it's cool or whatever it is. >> It is, right? So that's why, I always advise, when they come to me and ask me things like, "Hey, what's the right way to do it? What is the secret sauce?" And, we talked about this. The first thing I tell them is, "Find out what is the business case that's having the most amount of problems, that that can be solved using some of the AI use cases," right? Not all of them can be solved. Even after you experiment, do the whole nine yards, spend millions of dollars on that, right? And later on you make it efficient only by saving maybe $50,000 for the company or a $100,000 for the company, is it really even worth the experiment, right? So you got to start with the saying that, you know, where's the base for this happening? Where's the need? What's a business use case? It doesn't have to be about cost efficient and saving money in the existing processes. It could be a new thing. You want to bring in a new revenue stream, but figure out what is a business use case, how much money potentially I can make off of that. The same way that start-ups go after. Right? >> Yeah. Pretty straightforward. All right, let's take a look at where ML and AI fit relative to the other hot sectors of the ETR dataset. This XY graph shows net score spending velocity in the vertical axis and presence in the survey, they call it sector perversion for the October survey, the January survey's in the field. Then that squiggly line on ML/AI represents the progression. Since the January 21 survey, you can see the downward trajectory. And we position ML and AI relative to the other big four hot sectors or big three, including, ML/AI is four. Containers, cloud and RPA. These have consistently performed above that magic 40% red dotted line for most of the past two years. Anything above 40%, we think is highly elevated. And we've just included analytics and big data for context and relevant adjacentness, if you will. Now note that green arrow moving toward, you know, the 40% mark on ML/AI. I got a glimpse of the January survey, which is in the field. It's got more than a thousand responses already, and it's trending up for the current survey. So Andy, what do you make of this downward trajectory over the past seven quarters and the presumed uptick in the coming months? >> So one of the things you have to keep in mind is when the pandemic happened, it's about survival mode, right? So when somebody's in a survival mode, what happens, the luxury and the innovations get cut. That's what happens. And this is exactly what happened in the situation. So as you can see in the last seven quarters, which is almost dating back close to pandemic, everybody was trying to keep their operations alive, especially digital operations. How do I keep the lights on? That's the most important thing for them. So while the numbers spent on AI, ML is less overall, I still think the AI ML to spend to sort of like a employee experience or the IT ops, AI ops, ML ops, as we talked about, some of those areas actually went up. There are companies, we talked about it, Atlassian had a lot of platform issues till the amount of money people are spending on that is exorbitant and simply because they are offering the solution that was not available other way. So there are companies out there, you can take AoPS or incident management for that matter, right? A lot of companies have a digital insurance, they don't know how to properly manage it. How do you find an intern solve it immediately? That's all using AI ML and some of those areas actually growing unbelievable, the companies in that area. >> So this is a really good point. If you can you bring up that chart again, what Andy's saying is a lot of the companies in the ETR taxonomy that are doing things with AI might not necessarily show up in a granular fashion. And I think the other point I would make is, these are still highly elevated numbers. If you put on like storage and servers, they would read way, way down the list. And, look in the pandemic, we had to deal with work from home, we had to re-architect the network, we had to worry about security. So those are really good points that you made there. Let's, unpack this a little bit and look at the ML AI sector and the ETR data and specifically at the players and get Andy to comment on this. This chart here shows the same x y dimensions, and it just notes some of the players that are specifically have services and products that people spend money on, that CIOs and IT buyers can comment on. So the table insert shows how the companies are plotted, it's net score, and then the ends in the survey. And Andy, the hyperscalers are dominant, as you can see. You see Databricks there showing strong as a specialist, and then you got to pack a six or seven in there. And then Oracle and IBM, kind of the big whales of yester year are in the mix. And to your point, companies like Salesforce that you mentioned to me offline aren't in that mix, but they do a lot in AI. But what are your takeaways from that data? >> If you could put the slide back on please. I want to make quick comments on a couple of those. So the first one is, it's surprising other hyperscalers, right? As you and I talked about this earlier, AWS is more about logo blocks. We discussed that, right? >> Like what? Like a SageMaker as an example. >> We'll give you all the components what do you need. Whether it's MLOps component or whether it's, CodeWhisperer that we talked about, or a oral platform or data or data, whatever you want. They'll give you the blocks and then you'll build things on top of it, right? But Google took a different way. Matter of fact, if we did those numbers a few years ago, Google would've been number one because they did a lot of work with their acquisition of DeepMind and other things. They're way ahead of the pack when it comes to AI for longest time. Now, I think Microsoft's move of partnering and taking a huge competitor out would open the eyes is unbelievable. You saw that everybody is talking about chat GPI, right? And the open AI tool and ChatGPT rather. Remember as Warren Buffet is saying that, when my laundry lady comes and talk to me about stock market, it's heated up. So that's how it's heated up. Everybody's using ChatGPT. What that means is at the end of the day is they're creating, it's still in beta, keep in mind. It's not fully... >> Can you play with it a little bit? >> I have a little bit. >> I have, but it's good and it's not good. You know what I mean? >> Look, so at the end of the day, you take the massive text of all the available text in the world today, mass them all together. And then you ask a question, it's going to basically search through that and figure it out and answer that back. Yes, it's good. But again, as we discussed, if there's no business use case of what problem you're going to solve. This is building hype. But then eventually they'll figure out, for example, all your chats, online chats, could be aided by your AI chat bots, which is already there, which is not there at that level. This could build help that, right? Or the other thing we talked about is one of the areas where I'm more concerned about is that it is able to produce equal enough original text at the level that humans can produce, for example, ChatGPT or the equal enough, the large language transformer can help you write stories as of Shakespeare wrote it. Pretty close to it. It'll learn from that. So when it comes down to it, talk about creating messages, articles, blogs, especially during political seasons, not necessarily just in US, but anywhere for that matter. If people are able to produce at the emission speed and throw it at the consumers and confuse them, the elections can be won, the governments can be toppled. >> Because to your point about chatbots is chatbots have obviously, reduced the number of bodies that you need to support chat. But they haven't solved the problem of serving consumers. Most of the chat bots are conditioned response, which of the following best describes your problem? >> The current chatbot. >> Yeah. Hey, did we solve your problem? No. Is the answer. So that has some real potential. But if you could bring up that slide again, Ken, I mean you've got the hyperscalers that are dominant. You talked about Google and Microsoft is ubiquitous, they seem to be dominant in every ETR category. But then you have these other specialists. How do those guys compete? And maybe you could even, cite some of the guys that you know, how do they compete with the hyperscalers? What's the key there for like a C3 ai or some of the others that are on there? >> So I've spoken with at least two of the CEOs of the smaller companies that you have on the list. One of the things they're worried about is that if they continue to operate independently without being part of hyperscaler, either the hyperscalers will develop something to compete against them full scale, or they'll become irrelevant. Because at the end of the day, look, cloud is dominant. Not many companies are going to do like AI modeling and training and deployment the whole nine yards by independent by themselves. They're going to depend on one of the clouds, right? So if they're already going to be in the cloud, by taking them out to come to you, it's going to be extremely difficult issue to solve. So all these companies are going and saying, "You know what? We need to be in hyperscalers." For example, you could have looked at DataRobot recently, they made announcements, Google and AWS, and they are all over the place. So you need to go where the customers are. Right? >> All right, before we go on, I want to share some other data from ETR and why people adopt AI and get your feedback. So the data historically shows that feature breadth and technical capabilities were the main decision points for AI adoption, historically. What says to me that it's too much focus on technology. In your view, is that changing? Does it have to change? Will it change? >> Yes. Simple answer is yes. So here's the thing. The data you're speaking from is from previous years. >> Yes >> I can guarantee you, if you look at the latest data that's coming in now, those two will be a secondary and tertiary points. The number one would be about ROI. And how do I achieve? I've spent ton of money on all of my experiments. This is the same thing theme I'm seeing across when talking to everybody who's spending money on AI. I've spent so much money on it. When can I get it live in production? How much, how can I quickly get it? Because you know, the board is breathing down their neck. You already spend this much money. Show me something that's valuable. So the ROI is going to become, take it from me, I'm predicting this for 2023, that's going to become number one. >> Yeah, and if people focus on it, they'll figure it out. Okay. Let's take a look at some of the top players that won, some of the names we just looked at and double click on that and break down their spending profile. So the chart here shows the net score, how net score is calculated. So pay attention to the second set of bars that Databricks, who was pretty prominent on the previous chart. And we've annotated the colors. The lime green is, we're bringing the platform in new. The forest green is, we're going to spend 6% or more relative to last year. And the gray is flat spending. The pinkish is our spending's going to be down on AI and ML, 6% or worse. And the red is churn. So you don't want big red. You subtract the reds from the greens and you get net score, which is shown by those blue dots that you see there. So AWS has the highest net score and very little churn. I mean, single low single digit churn. But notably, you see Databricks and DataRobot are next in line within Microsoft and Google also, they've got very low churn. Andy, what are your thoughts on this data? >> So a couple of things that stands out to me. Most of them are in line with my conversation with customers. Couple of them stood out to me on how bad IBM Watson is doing. >> Yeah, bring that back up if you would. Let's take a look at that. IBM Watson is the far right and the red, that bright red is churning and again, you want low red here. Why do you think that is? >> Well, so look, IBM has been in the forefront of innovating things for many, many years now, right? And over the course of years we talked about this, they moved from a product innovation centric company into more of a services company. And over the years they were making, as at one point, you know that they were making about majority of that money from services. Now things have changed Arvind has taken over, he came from research. So he's doing a great job of trying to reinvent themselves as a company. But it's going to have a long way to catch up. IBM Watson, if you think about it, that played what, jeopardy and chess years ago, like 15 years ago? >> It was jaw dropping when you first saw it. And then they weren't able to commercialize that. >> Yeah. >> And you're making a good point. When Gerstner took over IBM at the time, John Akers wanted to split the company up. He wanted to have a database company, he wanted to have a storage company. Because that's where the industry trend was, Gerstner said no, he came from AMEX, right? He came from American Express. He said, "No, we're going to have a single throat to choke for the customer." They bought PWC for relatively short money. I think it was $15 billion, completely transformed and I would argue saved IBM. But the trade off was, it sort of took them out of product leadership. And so from Gerstner to Palmisano to Remedi, it was really a services led company. And I think Arvind is really bringing it back to a product company with strong consulting. I mean, that's one of the pillars. And so I think that's, they've got a strong story in data and AI. They just got to sort of bring it together and better. Bring that chart up one more time. I want to, the other point is Oracle, Oracle sort of has the dominant lock-in for mission critical database and they're sort of applying AI there. But to your point, they're really not an AI company in the sense that they're taking unstructured data and doing sort of new things. It's really about how to make Oracle better, right? >> Well, you got to remember, Oracle is about database for the structure data. So in yesterday's world, they were dominant database. But you know, if you are to start storing like videos and texts and audio and other things, and then start doing search of vector search and all that, Oracle is not necessarily the database company of choice. And they're strongest thing being apps and building AI into the apps? They are kind of surviving in that area. But again, I wouldn't name them as an AI company, right? But the other thing that that surprised me in that list, what you showed me is yes, AWS is number one. >> Bring that back up if you would, Ken. >> AWS is number one as you, it should be. But what what actually caught me by surprise is how DataRobot is holding, you know? I mean, look at that. The either net new addition and or expansion, DataRobot seem to be doing equally well, even better than Microsoft and Google. That surprises me. >> DataRobot's, and again, this is a function of spending momentum. So remember from the previous chart that Microsoft and Google, much, much larger than DataRobot. DataRobot more niche. But with spending velocity and has always had strong spending velocity, despite some of the recent challenges, organizational challenges. And then you see these other specialists, H2O.ai, Anaconda, dataiku, little bit of red showing there C3.ai. But these again, to stress are the sort of specialists other than obviously the hyperscalers. These are the specialists in AI. All right, so we hit the bigger names in the sector. Now let's take a look at the emerging technology companies. And one of the gems of the ETR dataset is the emerging technology survey. It's called ETS. They used to just do it like twice a year. It's now run four times a year. I just discovered it kind of mid-2022. And it's exclusively focused on private companies that are potential disruptors, they might be M&A candidates and if they've raised enough money, they could be acquirers of companies as well. So Databricks would be an example. They've made a number of investments in companies. SNEAK would be another good example. Companies that are private, but they're buyers, they hope to go IPO at some point in time. So this chart here, shows the emerging companies in the ML AI sector of the ETR dataset. So the dimensions of this are similar, they're net sentiment on the Y axis and mind share on the X axis. Basically, the ETS study measures awareness on the x axis and intent to do something with, evaluate or implement or not, on that vertical axis. So it's like net score on the vertical where negatives are subtracted from the positives. And again, mind share is vendor awareness. That's the horizontal axis. Now that inserted table shows net sentiment and the ends in the survey, which informs the position of the dots. And you'll notice we're plotting TensorFlow as well. We know that's not a company, but it's there for reference as open source tooling is an option for customers. And ETR sometimes like to show that as a reference point. Now we've also drawn a line for Databricks to show how relatively dominant they've become in the past 10 ETS surveys and sort of mind share going back to late 2018. And you can see a dozen or so other emerging tech vendors. So Andy, I want you to share your thoughts on these players, who were the ones to watch, name some names. We'll bring that data back up as you as you comment. >> So Databricks, as you said, remember we talked about how Oracle is not necessarily the database of the choice, you know? So Databricks is kind of trying to solve some of the issue for AI/ML workloads, right? And the problem is also there is no one company that could solve all of the problems. For example, if you look at the names in here, some of them are database names, some of them are platform names, some of them are like MLOps companies like, DataRobot (indistinct) and others. And some of them are like future based companies like, you know, the Techton and stuff. >> So it's a mix of those sub sectors? >> It's a mix of those companies. >> We'll talk to ETR about that. They'd be interested in your input on how to make this more granular and these sub-sectors. You got Hugging Face in here, >> Which is NLP, yeah. >> Okay. So your take, are these companies going to get acquired? Are they going to go IPO? Are they going to merge? >> Well, most of them going to get acquired. My prediction would be most of them will get acquired because look, at the end of the day, hyperscalers need these capabilities, right? So they're going to either create their own, AWS is very good at doing that. They have done a lot of those things. But the other ones, like for particularly Azure, they're going to look at it and saying that, "You know what, it's going to take time for me to build this. Why don't I just go and buy you?" Right? Or or even the smaller players like Oracle or IBM Cloud, this will exist. They might even take a look at them, right? So at the end of the day, a lot of these companies are going to get acquired or merged with others. >> Yeah. All right, let's wrap with some final thoughts. I'm going to make some comments Andy, and then ask you to dig in here. Look, despite the challenge of leveraging AI, you know, Ken, if you could bring up the next chart. We're not repeating, we're not predicting the AI winter of the 1990s. Machine intelligence. It's a superpower that's going to permeate every aspect of the technology industry. AI and data strategies have to be connected. Leveraging first party data is going to increase AI competitiveness and shorten time to value. Andy, I'd love your thoughts on that. I know you've got some thoughts on governance and AI ethics. You know, we talked about ChatGBT, Deepfakes, help us unpack all these trends. >> So there's so much information packed up there, right? The AI and data strategy, that's very, very, very important. If you don't have a proper data, people don't realize that AI is, your AI is the morals that you built on, it's predominantly based on the data what you have. It's not, AI cannot predict something that's going to happen without knowing what it is. It need to be trained, it need to understand what is it you're talking about. So 99% of the time you got to have a good data for you to train. So this where I mentioned to you, the problem is a lot of these companies can't afford to collect the real world data because it takes too long, it's too expensive. So a lot of these companies are trying to do the synthetic data way. It has its own set of issues because you can't use all... >> What's that synthetic data? Explain that. >> Synthetic data is basically not a real world data, but it's a created or simulated data equal and based on real data. It looks, feels, smells, taste like a real data, but it's not exactly real data, right? This is particularly useful in the financial and healthcare industry for world. So you don't have to, at the end of the day, if you have real data about your and my medical history data, if you redact it, you can still reverse this. It's fairly easy, right? >> Yeah, yeah. >> So by creating a synthetic data, there is no correlation between the real data and the synthetic data. >> So that's part of AI ethics and privacy and, okay. >> So the synthetic data, the issue with that is that when you're trying to commingle that with that, you can't create models based on just on synthetic data because synthetic data, as I said is artificial data. So basically you're creating artificial models, so you got to blend in properly that that blend is the problem. And you know how much of real data, how much of synthetic data you could use. You got to use judgment between efficiency cost and the time duration stuff. So that's one-- >> And risk >> And the risk involved with that. And the secondary issues which we talked about is that when you're creating, okay, you take a business use case, okay, you think about investing things, you build the whole thing out and you're trying to put it out into the market. Most companies that I talk to don't have a proper governance in place. They don't have ethics standards in place. They don't worry about the biases in data, they just go on trying to solve a business case >> It's wild west. >> 'Cause that's what they start. It's a wild west! And then at the end of the day when they are close to some legal litigation action or something or something else happens and that's when the Oh Shit! moments happens, right? And then they come in and say, "You know what, how do I fix this?" The governance, security and all of those things, ethics bias, data bias, de-biasing, none of them can be an afterthought. It got to start with the, from the get-go. So you got to start at the beginning saying that, "You know what, I'm going to do all of those AI programs, but before we get into this, we got to set some framework for doing all these things properly." Right? And then the-- >> Yeah. So let's go back to the key points. I want to bring up the cloud again. Because you got to get cloud right. Getting that right matters in AI to the points that you were making earlier. You can't just be out on an island and hyperscalers, they're going to obviously continue to do well. They get more and more data's going into the cloud and they have the native tools. To your point, in the case of AWS, Microsoft's obviously ubiquitous. Google's got great capabilities here. They've got integrated ecosystems partners that are going to continue to strengthen through the decade. What are your thoughts here? >> So a couple of things. One is the last mile ML or last mile AI that nobody's talking about. So that need to be attended to. There are lot of players in the market that coming up, when I talk about last mile, I'm talking about after you're done with the experimentation of the model, how fast and quickly and efficiently can you get it to production? So that's production being-- >> Compressing that time is going to put dollars in your pocket. >> Exactly. Right. >> So once, >> If you got it right. >> If you get it right, of course. So there are, there are a couple of issues with that. Once you figure out that model is working, that's perfect. People don't realize, the moment you decide that moment when the decision is made, it's like a new car. After you purchase the value decreases on a minute basis. Same thing with the models. Once the model is created, you need to be in production right away because it starts losing it value on a seconds minute basis. So issue number one, how fast can I get it over there? So your deployment, you are inferencing efficiently at the edge locations, your optimization, your security, all of this is at issue. But you know what is more important than that in the last mile? You keep the model up, you continue to work on, again, going back to the car analogy, at one point you got to figure out your car is costing more than to operate. So you got to get a new car, right? And that's the same thing with the models as well. If your model has reached a stage, it is actually a potential risk for your operation. To give you an idea, if Uber has a model, the first time when you get a car from going from point A to B cost you $60. If the model decayed the next time I might give you a $40 rate, I would take it definitely. But it's lost for the company. The business risk associated with operating on a bad model, you should realize it immediately, pull the model out, retrain it, redeploy it. That's is key. >> And that's got to be huge in security model recency and security to the extent that you can get real time is big. I mean you, you see Palo Alto, CrowdStrike, a lot of other security companies are injecting AI. Again, they won't show up in the ETR ML/AI taxonomy per se as a pure play. But ServiceNow is another company that you have have mentioned to me, offline. AI is just getting embedded everywhere. >> Yep. >> And then I'm glad you brought up, kind of real-time inferencing 'cause a lot of the modeling, if we can go back to the last point that we're going to make, a lot of the AI today is modeling done in the cloud. The last point we wanted to make here, I'd love to get your thoughts on this, is real-time AI inferencing for instance at the edge is going to become increasingly important for us. It's going to usher in new economics, new types of silicon, particularly arm-based. We've covered that a lot on "Breaking Analysis", new tooling, new companies and that could disrupt the sort of cloud model if new economics emerge. 'Cause cloud obviously very centralized, they're trying to decentralize it. But over the course of this decade we could see some real disruption there. Andy, give us your final thoughts on that. >> Yes and no. I mean at the end of the day, cloud is kind of centralized now, but a lot of this companies including, AWS is kind of trying to decentralize that by putting their own sub-centers and edge locations. >> Local zones, outposts. >> Yeah, exactly. Particularly the outpost concept. And if it can even become like a micro center and stuff, it won't go to the localized level of, I go to a single IOT level. But again, the cloud extends itself to that level. So if there is an opportunity need for it, the hyperscalers will figure out a way to fit that model. So I wouldn't too much worry about that, about deployment and where to have it and what to do with that. But you know, figure out the right business use case, get the right data, get the ethics and governance place and make sure they get it to production and make sure you pull the model out when it's not operating well. >> Excellent advice. Andy, I got to thank you for coming into the studio today, helping us with this "Breaking Analysis" segment. Outstanding collaboration and insights and input in today's episode. Hope we can do more. >> Thank you. Thanks for having me. I appreciate it. >> You're very welcome. All right. I want to thank Alex Marson who's on production and manages the podcast. Ken Schiffman as well. Kristen Martin and Cheryl Knight helped get the word out on social media and our newsletters. And Rob Hoof is our editor-in-chief over at Silicon Angle. He does some great editing for us. Thank you all. Remember all these episodes are available as podcast. Wherever you listen, all you got to do is search "Breaking Analysis" podcast. I publish each week on wikibon.com and silicon angle.com or you can email me at david.vellante@siliconangle.com to get in touch, or DM me at dvellante or comment on our LinkedIn posts. Please check out ETR.AI for the best survey data and the enterprise tech business, Constellation Research. Andy publishes there some awesome information on AI and data. This is Dave Vellante for theCUBE Insights powered by ETR. Thanks for watching everybody and we'll see you next time on "Breaking Analysis". (gentle closing tune plays)

>>Welcome to this exclusive discussion. IBM, the next three years of life sciences, innovation, precision medicine, advanced clinical data management and beyond. My name is Dave Volante from the Cuban today, we're going to take a deep dive into some of the most important trends impacting the life sciences industry in the next 60 minutes. Yeah, of course. We're going to hear how IBM is utilizing Watson and some really important in life impacting ways, but we'll also bring in real world perspectives from industry and the independent analyst view to better understand how technology and data are changing the nature of precision medicine. Now, the pandemic has created a new reality for everyone, but especially for life sciences companies, one where digital transformation is no longer an option, but a necessity. Now the upside is the events of the past 22 months have presented an accelerated opportunity for innovation technology and real world data are coming together and being applied to support life science, industry trends and improve drug discovery, clinical development, and treatment commercialization throughout the product life cycle cycle. Now I'd like to introduce our esteemed panel. Let me first introduce Lorraine Marshawn, who is general manager of life sciences at IBM Watson health. Lorraine leads the organization dedicated to improving clinical development research, showing greater treatment value in getting treatments to patients faster with differentiated solutions. Welcome Lorraine. Great to see you. >>Dr. Namita LeMay is the research vice-president of IDC, where she leads the life sciences R and D strategy and technology program, which provides research based advisory and consulting services as well as market analysis. The loan to meta thanks for joining us today. And our third panelist is Greg Cunningham. Who's the director of the RWE center of excellence at Eli Lilly and company. Welcome, Greg, you guys are doing some great work. Thanks for being here. Thanks >>Dave. >>Now today's panelists are very passionate about their work. If you'd like to ask them a question, please add it to the chat box located near the bottom of your screen, and we'll do our best to answer them all at the end of the panel. Let's get started. Okay, Greg, and then Lorraine and meta feel free to chime in after one of the game-changers that you're seeing, which are advancing precision medicine. And how do you see this evolving in 2022 and into the next decade? >>I'll give my answer from a life science research perspective. The game changer I see in advancing precision medicine is moving from doing research using kind of a single gene mutation or kind of a single to look at to doing this research using combinations of genes and the potential that this brings is to bring better drug targets forward, but also get the best product to a patient faster. Um, I can give, uh, an example how I see it playing out in the last decade. Non-oncology real-world evidence. We've seen an evolution in precision medicine as we've built out the patient record. Um, as we've done that, uh, the marketplace has evolved rapidly, uh, with, particularly for electronic medical record data and genomic data. And we were pretty happy to get our hands on electronic medical record data in the early days. And then later the genetic test results were combined with this data and we could do research looking at a single mutation leading to better patient outcomes. But I think where we're going to evolve in 2022 and beyond is with genetic testing, growing and oncology, providing us more data about that patient. More genes to look at, uh, researchers can look at groups of genes to analyze, to look at that complex combination of gene mutations. And I think it'll open the door for things like using artificial intelligence to help researchers plow through the complex number of permutations. When you think about all those genes you can look at in combination, right? Lorraine yes. Data and machine intelligence coming together, anything you would add. >>Yeah. Thank you very much. Well, I think that Greg's response really sets us up nicely, particularly when we think about the ability to utilize real-world data in the farm industry across a number of use cases from discovery to development to commercial, and, you know, in particular, I think with real world data and the comments that Greg just made about clinical EMR data linked with genetic or genomic data, a real area of interest in one that, uh, Watson health in particular is focused on the idea of being able to create a data exchange so that we can bring together claims clinical EMR data, genomics data, increasingly wearables and data directly from patients in order to create a digital health record that we like to call an intelligent patient health record that basically gives us the digital equivalent of a real life patient. And these can be used in use cases in randomized controlled clinical trials for synthetic control arms or natural history. They can be used in order to track patients' response to drugs and look at outcomes after they've been on various therapies as, as Greg is speaking to. And so I think that, you know, the promise of data and technology, the AI that we can apply on that is really helping us advance, getting therapies to market faster, with better information, lower sample sizes, and just a much more efficient way to do drug development and to track and monitor outcomes in patients. >>Great. Thank you for that now to meta, when I joined IDC many, many years ago, I really didn't know much about the industry that I was covering, but it's great to see you as a former practitioner now bringing in your views. What do you see as the big game-changers? >>So, um, I would, I would agree with what both Lorraine and Greg said. Um, but one thing that I'd just like to call out is that, you know, everyone's talking about big data, the volume of data is growing. It's growing exponentially actually about, I think 30% of data that exists today is healthcare data. And it's growing at a rate of 36%. That's huge, but then it's not just about the big, it's also about the broad, I think, um, you know, I think great points that, uh, Lorraine and Greg brought out that it's, it's not just specifically genomic data, it's multi omic data. And it's also about things like medical history, social determinants of health, behavioral data. Um, and why, because when you're talking about precision medicine and we know that we moved away from the, the terminology of personalized to position, because you want to talk about disease stratification and you can, it's really about convergence. >>Um, if you look at a recent JAMA paper in 2021, only 1% of EHS actually included genomic data. So you really need to have that ability to look at data holistically and IDC prediction is seeing that investments in AI to fuel in silico, silicone drug discovery will double by 20, 24, but how are you actually going to integrate all the different types of data? Just look at, for example, diabetes, you're on type two diabetes, 40 to 70% of it is genetically inherited and you have over 500 different, uh, genetic low side, which could be involved in playing into causing diabetes. So the earlier strategy, when you are looking at, you know, genetic risk scoring was really single trait. Now it's transitioning to multi rate. And when you say multi trade, you really need to get that integrated view that converging for you to, to be able to drive a precision medicine strategy. So to me, it's a very interesting contrast on one side, you're really trying to make it specific and focused towards an individual. And on the other side, you really have to go wider and bigger as well. >>Uh, great. I mean, the technology is enabling that convergence and the conditions are almost mandating it. Let's talk about some more about data that the data exchange and building an intelligent health record, as it relates to precision medicine, how will the interoperability of real-world data, you know, create that more cohesive picture for the, for the patient maybe Greg, you want to start, or anybody else wants to chime in? >>I think, um, the, the exciting thing from, from my perspective is the potential to gain access to data. You may be weren't aware of an exchange in implies that, uh, some kind of cataloging, so I can see, uh, maybe things that might, I just had no idea and, uh, bringing my own data and maybe linking data. These are concepts that I think are starting to take off in our field, but it, it really opens up those avenues to when you, you were talking about data, the robustness and richness volume isn't, uh, the only thing is Namita said, I think really getting to a rich high-quality data and, and an exchange offers a far bigger, uh, range for all of us to, to use, to get our work done. >>Yeah. And I think, um, just to chime, chime into that, uh, response from Greg, you know, what we hear increasingly, and it's pretty pervasive across the industry right now, because this ability to create an exchange or the intelligent, uh, patient health record, these are new ideas, you know, they're still rather nascent and it always is the operating model. Uh, that, that is the, uh, the difficult challenge here. And certainly that is the case. So we do have data in various silos. Uh, they're in patient claims, they're in electronic medical records, they might be in labs, images, genetic files on your smartphone. And so one of the challenges with this interoperability is being able to tap into these various sources of data, trying to identify quality data, as Greg has said, and the meta is underscoring as well. Uh, we've gotta be able to get to the depth of data that's really meaningful to us, but then we have to have technology that allows us to pull this data together. >>First of all, it has to be de-identified because of security and patient related needs. And then we've gotta be able to link it so that you can create that likeness in terms of the record, it has to be what we call cleaned or curated so that you get the noise and all the missing this out of it, that's a big step. And then it needs to be enriched, which means that the various components that are going to be meaningful, you know, again, are brought together so that you can create that cohort of patients, that individual patient record that now is useful in so many instances across farm, again, from development, all the way through commercial. So the idea of this exchange is to enable that exact process that I just described to have a, a place, a platform where various entities can bring their data in order to have it linked and integrated and cleaned and enriched so that they get something that is a package like a data package that they can actually use. >>And it's easy to plug into their, into their studies or into their use cases. And I think a really important component of this is that it's gotta be a place where various third parties can feel comfortable bringing their data together in order to match it with other third parties. That is a, a real value, uh, that the industry is increasingly saying would be important to them is, is the ability to bring in those third-party data sets and be able to link them and create these, these various data products. So that's really the idea of the data exchange is that you can benefit from accessing data, as Greg mentioned in catalogs that maybe are across these various silos so that you can do the kind of work that you need. And that we take a lot of the hard work out of it. I like to give an example. >>We spoke with one of our clients at one of the large pharma companies. And, uh, I think he expressed it very well. He said, what I'd like to do is have like a complete dataset of lupus. Lupus is an autoimmune condition. And I've just like to have like the quintessential lupus dataset that I can use to run any number of use cases across it. You know, whether it's looking at my phase one trial, whether it's selecting patients and enriching for later stage trials, whether it's understanding patient responses to different therapies as I designed my studies. And so, you know, this idea of adding in therapeutic area indication, specific data sets and being able to create that for the industry in the meta mentioned, being able to do that, for example, in diabetes, that's how pharma clients need to have their needs met is through taking the hard workout, bringing the data together, having it very therapeutically enriched so that they can use it very easily. >>Thank you for that detail and the meta. I mean, you can't do this with humans at scale in technology of all the things that Lorraine was talking about, the enrichment, the provenance, the quality, and of course, it's got to be governed. You've got to protect the privacy privacy humans just can't do all that at massive scale. Can it really tech that's where technology comes in? Doesn't it and automation. >>Absolutely. >>I, couldn't more, I think the biggest, you know, whether you talk about precision medicine or you talk about decentralized trials, I think there's been a lot of hype around these terms, but what is really important to remember is technology is the game changer and bringing all that data together is really going to be the key enabler. So multimodal data integration, looking at things like security or federated learning, or also when you're talking about leveraging AI, you're not talking about things like bias or other aspects around that are, are critical components that need to be addressed. I think the industry is, uh, it's partly, still trying to figure out the right use cases. So it's one part is getting together the data, but also getting together the right data. Um, I think data interoperability is going to be the absolute game changer for enabling this. Uh, but yes, um, absolutely. I can, I can really couldn't agree more with what Lorraine just said, that it's bringing all those different aspects of data together to really drive that precision medicine strategy. >>Excellent. Hey Greg, let's talk about protocols decentralized clinical trials. You know, they're not new to life silences, but, but the adoption of DCTs is of course sped up due to the pandemic we've had to make trade-offs obviously, and the risk is clearly worth it, but you're going to continue to be a primary approach as we enter 2022. What are the opportunities that you see to improve? How DCTs are designed and executed? >>I see a couple opportunities to improve in this area. The first is, uh, back to technology. The infrastructure around clinical trials has, has evolved over the years. Uh, but now you're talking about moving away from kind of site focus to the patient focus. Uh, so with that, you have to build out a new set of tools that would help. So for example, one would be novel trial, recruitment, and screening, you know, how do you, how do you find patients and how do you screen them to see if are they, are they really a fit for, for this protocol? Another example, uh, very important documents that we have to get is, uh, you know, the e-consent that someone's says, yes, I'm, well, I understand this study and I'm willing to do it, have to do that in a more remote way than, than we've done in the past. >>Um, the exciting area, I think, is the use of, uh, eco, uh, E-Pro where we capture data from the patient using apps, devices, sensors. And I think all of these capabilities will bring a new way of, of getting data faster, uh, in, in this kind of model. But the exciting thing from, uh, our perspective at Lily is it's going to bring more data about the patient from the patient, not just from the healthcare provider side, it's going to bring real data from these apps, devices and sensors. The second thing I think is using real-world data to identify patients, to also improve protocols. We run scenarios today, looking at what's the impact. If you change a cut point on a, a lab or a biomarker to see how that would affect, uh, potential enrollment of patients. So it, it definitely the real-world data can be used to, to make decisions, you know, how you improve these protocols. >>But the thing that we've been at the challenge we've been after that this probably offers the biggest is using real-world data to identify patients as we move away from large academic centers that we've used for years as our sites. Um, you can maybe get more patients who are from the rural areas of our countries or not near these large, uh, uh, academic centers. And we think it'll bring a little more diversity to the population, uh, who who's, uh, eligible, but also we have their data, so we can see if they really fit the criteria and the probability they are a fit for the trial is much higher than >>Right. Lorraine. I mean, your clients must be really pushing you to help them improve DCTs what are you seeing in the field? >>Yes, in fact, we just attended the inaugural meeting of the de-central trials research Alliance in, uh, in Boston about two weeks ago where, uh, all of the industry came together, pharma companies, uh, consulting vendors, just everyone who's been in this industry working to help define de-central trials and, um, think through what its potential is. Think through various models in order to enable it, because again, a nascent concept that I think COVID has spurred into action. Um, but it is important to take a look at the definition of DCT. I think there are those entities that describe it as accessing data directly from the patient. I think that is a component of it, but I think it's much broader than that. To me, it's about really looking at workflows and processes of bringing data in from various remote locations and enabling the whole ecosystem to work much more effectively along the data continuum. >>So a DCT is all around being able to make a site more effective, whether it's being able to administer a tele visit or the way that they're getting data into the electronic data captures. So I think we have to take a look at the, the workflows and the operating models for enabling de-central trials and a lot of what we're doing with our own technology. Greg mentioned the idea of electronic consent of being able to do electronic patient reported outcomes, other collection of data directly from the patient wearables tele-health. So these are all data acquisition, methodologies, and technologies that, that we are enabling in order to get the best of the data into the electronic data capture system. So edit can be put together and processed and submitted to the FDA for regulatory use for clinical trial type submission. So we're working on that. I think the other thing that's happening is the ability to be much more flexible and be able to have more cloud-based storage allows you to be much more inter-operable to allow API APIs in order to bring in the various types of data. >>So we're really looking at technology that can make us much more fluid and flexible and accommodating to all the ways that people live and work and manage their health, because we have to reflect that in the way we collect those data types. So that's a lot of what we're, what we're focused on. And in talking with our clients, we spend also a lot of time trying to understand along the, let's say de-central clinical trials continuum, you know, w where are they? And I know Namita is going to talk a little bit about research that they've done in terms of that adoption curve, but because COVID sort of forced us into being able to collect data in more remote fashion in order to allow some of these clinical trials to continue during COVID when a lot of them had to stop. What we want to make sure is that we understand and can codify some of those best practices and that we can help our clients enable that because the worst thing that would happen would be to have made some of that progress in that direction. >>But then when COVID is over to go back to the old ways of doing things and not bring some of those best practices forward, and we actually hear from some of our clients in the pharma industry, that they worry about that as well, because we don't yet have a system for operationalizing a de-central trial. And so we really have to think about the protocol it's designed, the indication, the types of patients, what makes sense to decentralize, what makes sense to still continue to collect data in a more traditional fashion. So we're spending a lot of time advising and consulting with our patients, as well as, I mean, with our clients, as well as CRS, um, on what the best model is in terms of their, their portfolio of studies. And I think that's a really important aspect of trying to accelerate the adoption is making sure that what we're doing is fit for purpose, just because you can use technology doesn't mean you should, it really still does require human beings to think about the problem and solve them in a very practical way. >>Great, thank you for that. Lorraine. I want to pick up on some things that Lorraine was just saying. And then back to what Greg was saying about, uh, uh, DCTs becoming more patient centric, you had a prediction or IDC, did I presume your fingerprints were on it? Uh, that by 20 25, 70 5% of trials will be patient-centric decentralized clinical trials, 90% will be hybrid. So maybe you could help us understand that relationship and what types of innovations are going to be needed to support that evolution of DCT. >>Thanks, Dave. Yeah. Um, you know, sorry, I, I certainly believe that, uh, you know, uh, Lorraine was pointing out of bringing up a very important point. It's about being able to continue what you have learned in over the past two years, I feel this, you know, it was not really a digital revolution. It was an attitude. The revolution that this industry underwent, um, technology existed just as clinical trials exist as drugs exist, but there was a proof of concept that technology works that this model is working. So I think that what, for example, telehealth, um, did for, for healthcare, you know, transition from, from care, anywhere care, anytime, anywhere, and even becoming predictive. That's what the decentralized clinical trials model is doing for clinical trials today. Great points again, that you have to really look at where it's being applied. You just can't randomly apply it across clinical trials. >>And this is where the industry is maturing the complexity. Um, you know, some people think decentralized trials are very simple. You just go and implement these centralized clinical trials, but it's not that simple as it it's being able to define, which are the right technologies for that specific, um, therapeutic area for that specific phase of the study. It's being also a very important point is bringing in the patient's voice into the process. Hey, I had my first telehealth visit sometime last year and I was absolutely thrilled about it. I said, no time wasted. I mean, everything's done in half an hour, but not all patients want that. Some want to consider going back and you, again, need to customize your de-centralized trials model to, to the, to the type of patient population, the demographics that you're dealing with. So there are multiple factors. Um, also stepping back, you know, Lorraine mentioned they're consulting with, uh, with their clients, advising them. >>And I think a lot of, um, a lot of companies are still evolving in their maturity in DCTs though. There's a lot of boys about it. Not everyone is very mature in it. So it's, I think it, one thing everyone's kind of agreeing with is yes, we want to do it, but it's really about how do we go about it? How do we make this a flexible and scalable modern model? How do we integrate the patient's voice into the process? What are the KPIs that we define the key performance indicators that we define? Do we have a playbook to implement this model to make it a scalable model? And, you know, finally, I think what organizations really need to look at is kind of developing a de-centralized mature maturity scoring model, so that I assess where I am today and use that playbook to define, how am I going to move down the line to me reach the next level of maturity. Those were some of my thoughts. Right? >>Excellent. And now remember you, if you have any questions, use the chat box below to submit those questions. We have some questions coming in from the audience. >>At one point to that, I think one common thread between the earlier discussion around precision medicine and around decentralized trials really is data interoperability. It is going to be a big game changer to, to enable both of these pieces. Sorry. Thanks, Dave. >>Yeah. Thank you. Yeah. So again, put your questions in the chat box. I'm actually going to go to one of the questions from the audience. I get some other questions as well, but when you think about all the new data types that are coming in from social media, omics wearables. So the question is with greater access to these new types of data, what trends are you seeing from pharma device as far as developing capabilities to effectively manage and analyze these novel data types? Is there anything that you guys are seeing, um, that you can share in terms of best practice or advice >>I'll offer up? One thing, I think the interoperability isn't quite there today. So, so what's that mean you can take some of those data sources. You mentioned, uh, some Omix data with, uh, some health claims data and it's the, we spend too much time and in our space putting data to gather the behind the scenes, I think the stat is 80% of the time is assembling the data 20% analyzing. And we've had conversations here at Lilly about how do we get to 80% of the time is doing analysis. And it really requires us to think, take a step back and think about when you create a, uh, a health record, you really have to be, have the same plugins so that, you know, data can be put together very easily, like Lorraine mentioned earlier. And that comes back to investing in as an industry and standards so that, you know, you have some of data standard, we all can agree upon. And then those plugs get a lot easier and we can spend our time figuring out how to make, uh, people's lives better with healthcare analysis versus putting data together, which is not a lot of fun behind the scenes. >>Other thoughts on, um, on, on how to take advantage of sort of novel data coming from things like devices in the nose that you guys are seeing. >>I could jump in there on your end. Did you want to go ahead? Okay. So, uh, I mean, I think there's huge value that's being seen, uh, in leveraging those multiple data types. I think one area you're seeing is the growth of prescription digital therapeutics and, um, using those to support, uh, you know, things like behavioral health issues and a lot of other critical conditions it's really taking you again, it is interlinking real-world data cause it's really taking you to the patient's home. Um, and it's, it's, there's a lot of patients in the city out here cause you can really monitor the patient real-time um, without the patient having coming, you know, coming and doing a site visit once in say four weeks or six weeks. So, um, I, and, uh, for example, uh, suicidal behavior and just to take an example, if you can predict well in advance, based on those behavioral parameters, that this is likely to trigger that, uh, the value of it is enormous. Um, again, I think, uh, Greg made a valid point about the industry still trying to deal with resolving the data interoperability issue. And there are so many players that are coming in the industry right now. There are really few that have the maturity and the capability to address these challenges and provide intelligence solutions. >>Yeah. Maybe I'll just, uh, go ahead and, uh, and chime into Nikita's last comment there. I think that's what we're seeing as well. And it's very common, you know, from an innovation standpoint that you have, uh, a nascent industry or a nascent innovation sort of situation that we have right now where it's very fragmented. You have a lot of small players, you have some larger entrenched players that have the capability, um, to help to solve the interoperability challenge, the standards challenge. I mean, I think IBM Watson health is certainly one of the entities that has that ability and is taking a stand in the industry, uh, in order to, to help lead in that way. Others are too. And, uh, but with, with all of the small companies that are trying to find interesting and creative ways to gather that data, it does create a very fragmented, uh, type of environment and ecosystem that we're in. >>And I think as we mature, as we do come forward with the KPIs, the operating models, um, because you know, the devil's in the detail in terms of the operating models, it's really exciting to talk these trends and think about the future state. But as Greg pointed out, if you're spending 80% of your time just under the hood, you know, trying to get the engine, all the spark plugs to line up, um, that's, that's just hard grunt work that has to be done. So I think that's where we need to be focused. And I think bringing all the data in from these disparate tools, you know, that's fine, we need, uh, a platform or the API APIs that can enable that. But I think as we, as we progress, we'll see more consolidation, uh, more standards coming into play, solving the interoperability types of challenges. >>And, um, so I think that's where we should, we should focus on what it's going to take and in three years to really codify this and make it, so it's a, it's a well hum humming machine. And, you know, I do know having also been in pharma that, uh, there's a very pilot oriented approach to this thing, which I think is really healthy. I think large pharma companies tend to place a lot of bets with different programs on different tools and technologies, to some extent to see what's gonna stick and, you know, kind of with an innovation mindset. And I think that's good. I think that's kind of part of the process of figuring out what is going to work and, and helping us when we get to that point of consolidating our model and the technologies going forward. So I think all of the efforts today are definitely driving us to something that feels much more codified in the next three to five years. >>Excellent. We have another question from the audience it's sort of related to the theme of this discussion, given the FDA's recent guidance on using claims and electronic health records, data to support regulatory decision-making what advancements do you think we can expect with regards to regulatory use of real-world data in the coming years? It's kind of a two-parter so maybe you guys can collaborate on this one. What role that, and then what role do you think industry plays in influencing innovation within the regulatory space? >>All right. Well, it looks like you've stumped the panel there. Uh, Dave, >>It's okay to take some time to think about it, right? You want me to repeat it? You guys, >>I, you know, I I'm sure that the group is going to chime into this. I, so the FDA has issued a guidance. Um, it's just, it's, it's exactly that the FDA issues guidances and says that, you know, it's aware and supportive of the fact that we need to be using real-world data. We need to create the interoperability, the standards, the ways to make sure that we can include it in regulatory submissions and the like, um, and, and I sort of think about it akin to the critical path initiative, probably, I don't know, 10 or 12 years ago in pharma, uh, when the FDA also embrace this idea of the critical path and being able to allow more in silico modeling of clinical trial, design and development. And it really took the industry a good 10 years, um, you know, before they were able to actually adopt and apply and take that sort of guidance or openness from the FDA and actually apply it in a way that started to influence the way clinical trials were designed or the in silico modeling. >>So I think the second part of the question is really important because while I think the FDA is saying, yes, we recognize it's important. Uh, we want to be able to encourage and support it. You know, when you look for example, at synthetic control arms, right? The use of real-world data in regulatory submissions over the last five or six years, all of the use cases have been in oncology. I think there've been about maybe somewhere between eight to 10 submissions. And I think only one actually was a successful submission, uh, in all those situations, the real-world data arm of that oncology trial that synthetic control arm was actually rejected by the FDA because of lack of completeness or, you know, equalness in terms of the data. So the FDA is not going to tell us how to do this. So I think the second part of the question, which is what's the role of industry, it's absolutely on industry in order to figure out exactly what we're talking about, how do we figure out the interoperability, how do we apply the standards? >>How do we ensure good quality data? How do we enrich it and create the cohort that is going to be equivalent to the patient in the real world, uh, in the end that would otherwise be in the clinical trial and how do we create something that the FDA can agree with? And we'll certainly we'll want to work with the FDA in order to figure out this model. And I think companies are already doing that, but I think that the onus is going to be on industry in order to figure out how you actually operationalize this and make it real. >>Excellent. Thank you. Um, question on what's the most common misconception that clinical research stakeholders with sites or participants, et cetera might have about DCTs? >>Um, I could jump in there. Right. So, sure. So, um, I think in terms of misconceptions, um, I think the communist misconceptions that sites are going away forever, which I do not think is really happening today. Then the second, second part of it is that, um, I think also the perspective that patients are potentially neglected because they're moving away. So we'll pay when I, when I, what I mean by that neglected, perhaps it was not the appropriate term, but the fact that, uh, will patients will, will, will patient engagement continue, will retention be strong since the patients are not interacting in person with the investigator quite as much. Um, so site retention and patient retention or engagement from both perspectives, I think remains a concern. Um, but actually if you look at, uh, look at, uh, assessments that have been done, I think patients are more than happy. >>Majority of the patients have been really happy about, about the new model. And in fact, sites are, seem to increase, have increased investments in technology by 50% to support this kind of a model. So, and the last thing is that, you know, decentralized trials is a great model and it can be applied to every possible clinical trial. And in another couple of weeks, the whole industry will be implementing only decentralized trials. I think we are far away from that. It's just not something that you would implement across every trial. And we discussed that already. So you have to find the right use cases for that. So I think those were some of the key misconceptions I'd say in the industry right now. Yeah. >>Yeah. And I would add that the misconception I hear the most about is, uh, the, the similar to what Namita said about the sites and healthcare professionals, not being involved to the level that they are today. Uh, when I mentioned earlier in our conversation about being excited about capturing more data, uh, from the patient that was always in context of, in addition to, you know, healthcare professional opinion, because I think both of them bring that enrichment and a broader perspective of that patient experience, whatever disease they're faced with. So I, I think some people think is just an all internet trial with just someone, uh, putting out there their own perspective. And, and it's, it's a combination of both to, to deliver a robust data set. >>Yeah. Maybe I'll just comment on, it reminds me of probably 10 or 15 years ago, maybe even more when, um, really remote monitoring was enabled, right? So you didn't have to have the study coordinator traveled to the investigative site in order to check the temperature of the freezer and make sure that patient records were being completed appropriately because they could have a remote visit and they could, they could send the data in a via electronic data and do the monitoring visit, you know, in real time, just the way we're having this kind of communication here. And there was just so much fear that you were going to replace or supplant the personal relationship between the sites between the study coordinators that you were going to, you know, have to supplant the role of the monitor, which was always a very important role in clinical trials. >>And I think people that really want to do embrace the technology and the advantages that it provided quickly saw that what it allowed was the monitor to do higher value work, you know, instead of going in and checking the temperature on a freezer, when they did have their visit, they were able to sit and have a quality discussion for example, about how patient recruitment was going or what was coming up in terms of the consent. And so it created a much more high touch, high quality type of interaction between the monitor and the investigative site. And I think we should be looking for the same advantages from DCT. We shouldn't fear it. We shouldn't think that it's going to supplant the site or the investigator or the relationship. It's our job to figure out where the technology fits and clinical sciences always got to be high touch combined with high-tech, but the high touch has to lead. And so getting that balance right? And so that's going to happen here as well. We will figure out other high value work, meaningful work for the site staff to do while they let the technology take care of the lower quality work, if you will, or the lower value work, >>That's not an, or it's an, and, and you're talking about the higher value work. And it, it leads me to something that Greg said earlier about the 80, 20, 80% is assembly. 20% is actually doing the analysis and that's not unique to, to, to life sciences, but, but sort of question is it's an organizational question in terms of how we think about data and how we approach data in the future. So Bamyan historically big data in life sciences in any industry really is required highly centralized and specialized teams to do things that the rain was talking about, the enrichment, the provenance, the data quality, the governance, the PR highly hyper specialized teams to do that. And they serve different constituencies. You know, not necessarily with that, with, with context, they're just kind of data people. Um, so they have responsibility for doing all those things. Greg, for instance, within literally, are you seeing a move to, to, to democratize data access? We've talked about data interoperability, part of that state of sharing, um, that kind of breaks that centralized hold, or is that just too far in the future? It's too risky in this industry? >>Uh, it's actually happening now. Uh, it's a great point. We, we try to classify what people can do. And, uh, the example would be you give someone who's less analytically qualified, uh, give them a dashboard, let them interact with the data, let them better understand, uh, what, what we're seeing out in the real world. Uh, there's a middle user, someone who you could give them, they can do some analysis with the tool. And the nice thing with that is you have some guardrails around that and you keep them in their lane, but it allows them to do some of their work without having to go ask those centralized experts that, that you mentioned their precious resources. And that's the third group is those, uh, highly analytical folks that can, can really deliver, uh, just value beyond. But when they're doing all those other things, uh, it really hinders them from doing what we've been talking about is the high value stuff. So we've, we've kind of split into those. We look at people using data in one of those three lanes and it, and it has helped I think, uh, us better not try to make a one fit solution for, for how we deliver data and analytic tools for people. Right. >>Okay. I mean, DCT hot topic with the, the, the audience here. Another question, um, what capabilities do sponsors and CRS need to develop in-house to pivot toward DCT? >>Should I jump in here? Yeah, I mean, um, I think, you know, when, when we speak about DCTs and when I speak with, uh, folks around in the industry, I, it takes me back to the days of risk-based monitoring. When it was first being implemented, it was a huge organizational change from the conventional monitoring models to centralize monitoring and risk-based monitoring, it needs a mental reset. It needs as Lorraine had pointed out a little while ago, restructuring workflows, re redefining processes. And I think that is one big piece. That is, I think the first piece, when, you know, when you're implementing a new model, I think organizational change management is a big piece of it because you are disturbing existing structures, existing methods. So getting that buy-in across the organization towards the new model, seeing what the value add in it. And where do you personally fit into that story? >>How do your workflows change, or how was your role impacted? I think without that this industry will struggle. So I see organizations, I think, first trying to work on that piece to build that in. And then of course, I also want to step back for the second to the, uh, to the point that you brought out about data democratization. And I think Greg Greg gave an excellent point, uh, input about how it's happening in the industry. But I would also say that the data democratization really empowerment of, of, of the stakeholders also includes the sites, the investigators. So what is the level of access to data that you know, that they have now, and is it, uh, as well as patients? So see increasingly more and more companies trying to provide access to patients finally, it's their data. So why shouldn't they have some insights to it, right. So access to patients and, uh, you know, the 80, 20 part of it. Uh, yes, he's absolutely right that, uh, we want to see that flip from, uh, 20%, um, you know, focusing on, on actually integrating the data 80% of analytics, but the real future will be coming in when actually the 20 and 18 has gone. And you actually have analysts the insights out on a silver platter. That's kind of wishful thinking, some of the industries is getting there in small pieces, but yeah, then that's just why I should, why we share >>Great points. >>And I think that we're, we're there in terms that like, I really appreciate the point around democratizing the data and giving the patient access ownership and control over their own data. I mean, you know, we see the health portals that are now available for patients to view their own records, images, and labs, and claims and EMR. We have blockchain technology, which is really critical here in terms of the patient, being able to pull all of their own data together, you know, in the blockchain and immutable record that they can own and control if they want to use that to transact clinical trial types of opportunities based on their data, they can, or other real world scenarios. But if they want to just manage their own data because they're traveling and if they're in a risky health situation, they've got their own record of their health, their health history, uh, which can avoid, you know, medical errors occurring. So, you know, even going beyond life sciences, I think this idea of democratizing data is just good for health. It's just good for people. And we definitely have the technology that can make it a reality. Now >>You're here. We have just about 10 minutes left and now of course, now all the questions are rolling in like crazy from the crowd. Would it be curious to know if there would be any comments from the panel on cost comparison analysis between traditional clinical trials in DCTs and how could the outcome effect the implementation of DCTs any sort of high-level framework you can share? >>I would say these are still early days to, to drive that analysis because I think many companies are, um, are still in the early stages of implementation. They've done a couple of trials. The other part of it that's important to keep in mind is, um, is for organizations it's, they're at a stage of, uh, of being on the learning curve. So when you're, you're calculating the cost efficiencies, if ideally you should have had two stakeholders involved, you could have potentially 20 stakeholders involved because everyone's trying to learn the process and see how it's going to be implemented. So, um, I don't think, and the third part of it, I think is organizations are still defining their KPIs. How do you measure it? What do you measure? So, um, and even still plugging in the pieces of technology that they need to fit in, who are they partnering with? >>What are the pieces of technology they're implementing? So I don't think there is a clear cut as answered at this stage. I think as you scale this model, the efficiencies will be seen. It's like any new technology or any new solution that's implemented in the first stages. It's always a little more complex and in fact sometimes costs extra. But as, as you start scaling it, as you establish your workflows, as you streamline it, the cost efficiencies will start becoming evident. That's why the industry is moving there. And I think that's how it turned out on the long run. >>Yeah. Just make it maybe out a comment. If you don't mind, the clinical trials are, have traditionally been costed are budgeted is on a per patient basis. And so, you know, based on the difficulty of the therapeutic area to recruit a rare oncology or neuromuscular disease, there's an average that it costs in order to find that patient and then execute the various procedures throughout the clinical trial on that patient. And so the difficulty of reaching the patient and then the complexity of the trial has led to what we might call a per patient stipend, which is just the metric that we use to sort of figure out what the average cost of a trial will be. So I think to point, we're going to have to see where the ability to adjust workflows, get to patients faster, collect data more easily in order to make the burden on the site, less onerous. I think once we start to see that work eases up because of technology, then I think we'll start to see those cost equations change. But I think right now the system isn't designed in order to really measure the economic benefit of de-central models. And I think we're going to have to sort of figure out what that looks like as we go along and since it's patient oriented right now, we'll have to say, well, you know, how does that work, ease up? And to those costs actually come down and then >>Just scale, it's going to be more, more clear as the media was saying, next question from the audiences, it's kind of a best fit question. You all have touched on this, but let me just ask it is what examples in which, in which phases suit DCT in its current form, be it fully DCT or hybrid models, none of our horses for courses question. >>Well, I think it's kind of, uh, it's, it's it's has its efficiencies, obviously on the later phases, then the absolute early phase trials, those are not the ideal models for DCTs I would say so. And again, the logic is also the fact that, you know, when you're, you're going into the later phase trials, the volume of number of patients is increasing considerably to the point that Lorraine brought up about access to the patients about patient selection. The fact, I think what one should look at is really the advantages that it brings in, in terms of, you know, patient access in terms of patient diversity, which is a big piece that, um, the cities are enabling. So, um, if you, if, if you, if you look at the spectrum of, of these advantages and, and just to step back for a moment, if you, if you're looking at costs, like you're looking at things like remote site monitoring, um, is, is a big, big plus, right? >>I mean, uh, site monitoring alone accounts for around a third of the trial costs. So there are so many pieces that fall in together. The challenge actually that comes when you're in defining DCTs and there are, as Rick pointed out multiple definitions of DCTs that are existing, uh, you know, in the industry right now, whether you're talking of what Detroit is doing, or you're talking about acro or Citi or others. But the point is it's a continuum, it's a continuum of different pieces that have been woven together. And so how do you decide which pieces you're plugging in and how does that impact the total cost or the solution that you're implementing? >>Great, thank you. Last question we have in the audience, excuse me. What changes have you seen? Are there others that you can share from the FDA EU APAC, regulators and supporting DCTs precision medicine for approval processes, anything you guys would highlight that we should be aware of? >>Um, I could quickly just add that. I think, um, I'm just publishing a report on de-centralized clinical trials should be published shortly, uh, perspective on that. But I would say that right now, um, there, there was a, in the FDA agenda, there was a plan for a decentralized clinical trials guidance, as far as I'm aware, one has not yet been published. There have been significant guidances that have been published both by email and by, uh, the FDA that, um, you know, around the implementation of clinical trials during the COVID pandemic, which incorporate various technology pieces, which support the DCD model. Um, but I, and again, I think one of the reasons why it's not easy to publish a well-defined guidance on that is because there are so many moving pieces in it. I think it's the Danish, uh, regulatory agency, which has per se published a guidance and revised it as well on decentralized clinical trials. >>Right. Okay. Uh, we're pretty much out of time, but I, I wonder Lorraine, if you could give us some, some final thoughts and bring us home things that we should be watching or how you see the future. >>Well, I think first of all, let me, let me thank the panel. Uh, we really appreciate Greg from Lily and the meta from IDC bringing their perspectives to this conversation. And, uh, I hope that the audience has enjoyed the, uh, the discussion that we've had around the future state of real world data as, as well as DCT. And I think, you know, some of the themes that we've talked about, number one, I think we have a vision and I think we have the right strategies in terms of the future promise of real-world data in any number of different applications. We certainly have talked about the promise of DCT to be more efficient, to get us closer to the patient. I think that what we have to focus on is how we come together as an industry to really work through these very vexing operational issues, because those are always the things that hang us up and whether it's clinical research or whether it's later stage, uh, applications of data. >>We, the healthcare system is still very fragmented, particularly in the us. Um, it's still very, state-based, uh, you know, different states can have different kinds of, uh, of, of cultures and geographic, uh, delineations. And so I think that, you know, figuring out a way that we can sort of harmonize and bring all of the data together, bring some of the models together. I think that's what you need to look to us to do both industry consulting organizations, such as IBM Watson health. And we are, you know, through DTRA and, and other, uh, consortia and different bodies. I think we're all identifying what the challenges are in terms of making this a reality and working systematically on those. >>It's always a pleasure to work with such great panelists. Thank you, Lorraine Marshawn, Dr. Namita LeMay, and Greg Cunningham really appreciate your participation today and your insights. The next three years of life sciences, innovation, precision medicine, advanced clinical data management and beyond has been brought to you by IBM in the cube. You're a global leader in high tech coverage. And while this discussion has concluded, the conversation continues. So please take a moment to answer a few questions about today's panel on behalf of the entire IBM life sciences team and the cube decks for your time and your feedback. And we'll see you next time.

>> From theCUBE Studios in Palo Alto and Boston, bringing you data-driven insights from theCUBE and ETR. This is breaking analysis with Dave Vellante. >> Moore's Law is dead, right? Think again. Massive improvements in processing power combined with data and AI will completely change the way we think about designing hardware, writing software and applying technology to businesses. Every industry will be disrupted. You hear that all the time. Well, it's absolutely true and we're going to explain why and what it all means. Hello everyone, and welcome to this week's Wikibon Cube Insights powered by ETR. In this breaking analysis, we're going to unveil some new data that suggests we're entering a new era of innovation that will be powered by cheap processing capabilities that AI will exploit. We'll also tell you where the new bottlenecks will emerge and what this means for system architectures and industry transformations in the coming decade. Moore's Law is dead, you say? We must have heard that hundreds, if not, thousands of times in the past decade. EE Times has written about it, MIT Technology Review, CNET, and even industry associations that have lived by Moore's Law. But our friend Patrick Moorhead got it right when he said, "Moore's Law, by the strictest definition of doubling chip densities every two years, isn't happening anymore." And you know what, that's true. He's absolutely correct. And he couched that statement by saying by the strict definition. And he did that for a reason, because he's smart enough to know that the chip industry are masters at doing work arounds. Here's proof that the death of Moore's Law by its strictest definition is largely irrelevant. My colleague, David Foyer and I were hard at work this week and here's the result. The fact is that the historical outcome of Moore's Law is actually accelerating and in quite dramatically. This graphic digs into the progression of Apple's SoC, system on chip developments from the A9 and culminating with the A14, 15 nanometer bionic system on a chip. The vertical axis shows operations per second and the horizontal axis shows time for three processor types. The CPU which we measure here in terahertz, that's the blue line which you can't even hardly see, the GPU which is the orange that's measured in trillions of floating point operations per second and then the NPU, the neural processing unit and that's measured in trillions of operations per second which is that exploding gray area. Now, historically, we always rushed out to buy the latest and greatest PC, because the newer models had faster cycles or more gigahertz. Moore's Law would double that performance every 24 months. Now that equates to about 40% annually. CPU performance is now moderated. That growth is now down to roughly 30% annual improvements. So technically speaking, Moore's Law as we know it was dead. But combined, if you look at the improvements in Apple's SoC since 2015, they've been on a pace that's higher than 118% annually. And it's even higher than that, because the actual figure for these three processor types we're not even counting the impact of DSPs and accelerator components of Apple system on a chip. It would push this even higher. Apple's A14 which is shown in the right hand side here is quite amazing. It's got a 64 bit architecture, it's got many, many cores. It's got a number of alternative processor types. But the important thing is what you can do with all this processing power. In an iPhone, the types of AI that we show here that continue to evolve, facial recognition, speech, natural language processing, rendering videos, helping the hearing impaired and eventually bringing augmented reality to the palm of your hand. It's quite incredible. So what does this mean for other parts of the IT stack? Well, we recently reported Satya Nadella's epic quote that "We've now reached peak centralization." So this graphic paints a picture that was quite telling. We just shared the processing powers exploding. The costs consequently are dropping like a rock. Apple's A14 cost the company approximately 50 bucks per chip. Arm at its v9 announcement said that it will have chips that can go into refrigerators. These chips are going to optimize energy usage and save 10% annually on your power consumption. They said, this chip will cost a buck, a dollar to shave 10% of your refrigerator electricity bill. It's just astounding. But look at where the expensive bottlenecks are, it's networks and it's storage. So what does this mean? Well, it means the processing is going to get pushed to the edge, i.e., wherever the data is born. Storage and networking are going to become increasingly distributed and decentralized. Now with custom silicon and all that processing power placed throughout the system, an AI is going to be embedded into software, into hardware and it's going to optimize a workloads for latency, performance, bandwidth, and security. And remember, most of that data, 99% is going to stay at the edge. And we love to use Tesla as an example. The vast majority of data that a Tesla car creates is never going to go back to the cloud. Most of it doesn't even get persisted. I think Tesla saves like five minutes of data. But some data will connect occasionally back to the cloud to train AI models and we're going to come back to that. But this picture says if you're a hardware company, you'd better start thinking about how to take advantage of that blue line that's exploding, Cisco. Cisco is already designing its own chips. But Dell, HPE, who kind of does maybe used to do a lot of its own custom silicon, but Pure Storage, NetApp, I mean, the list goes on and on and on either you're going to get start designing custom silicon or you're going to get disrupted in our view. AWS, Google and Microsoft are all doing it for a reason as is IBM and to Sarbjeet Johal said recently this is not your grandfather's semiconductor business. And if you're a software engineer, you're going to be writing applications that take advantage of all the data being collected and bringing to bear this processing power that we're talking about to create new capabilities like we've never seen it before. So let's get into that a little bit and dig into AI. You can think of AI as the superset. Just as an aside, interestingly in his book, "Seeing Digital", author David Moschella says, there's nothing artificial about this. He uses the term machine intelligence, instead of artificial intelligence and says that there's nothing artificial about machine intelligence just like there's nothing artificial about the strength of a tractor. It's a nuance, but it's kind of interesting, nonetheless, words matter. We hear a lot about machine learning and deep learning and think of them as subsets of AI. Machine learning applies algorithms and code to data to get "smarter", make better models, for example, that can lead to augmented intelligence and help humans make better decisions. These models improve as they get more data and are iterated over time. Now deep learning is a more advanced type of machine learning. It uses more complex math. But the point that we want to make here is that today much of the activity in AI is around building and training models. And this is mostly happening in the cloud. But we think AI inference will bring the most exciting innovations in the coming years. Inference is the deployment of that model that we were just talking about, taking real time data from sensors, processing that data locally and then applying that training that has been developed in the cloud and making micro adjustments in real time. So let's take an example. Again, we love Tesla examples. Think about an algorithm that optimizes the performance and safety of a car on a turn, the model take data on friction, road condition, angles of the tires, the tire wear, the tire pressure, all this data, and it keeps testing and iterating, testing and iterating, testing iterating that model until it's ready to be deployed. And then the intelligence, all this intelligence goes into an inference engine which is a chip that goes into a car and gets data from sensors and makes these micro adjustments in real time on steering and braking and the like. Now, as you said before, Tesla persist the data for very short time, because there's so much of it. It just can't push it back to the cloud. But it can now ever selectively store certain data if it needs to, and then send back that data to the cloud to further train them all. Let's say for instance, an animal runs into the road during slick conditions, Tesla wants to grab that data, because they notice that there's a lot of accidents in New England in certain months. And maybe Tesla takes that snapshot and sends it back to the cloud and combines it with other data and maybe other parts of the country or other regions of New England and it perfects that model further to improve safety. This is just one example of thousands and thousands that are going to further develop in the coming decade. I want to talk about how we see this evolving over time. Inference is where we think the value is. That's where the rubber meets the road, so to speak, based on the previous example. Now this conceptual chart shows the percent of spend over time on modeling versus inference. And you can see some of the applications that get attention today and how these applications will mature over time as inference becomes more and more mainstream, the opportunities for AI inference at the edge and in IOT are enormous. And we think that over time, 95% of that spending is going to go to inference where it's probably only 5% today. Now today's modeling workloads are pretty prevalent and things like fraud, adtech, weather, pricing, recommendation engines, and those kinds of things, and now those will keep getting better and better and better over time. Now in the middle here, we show the industries which are all going to be transformed by these trends. Now, one of the point that Moschella had made in his book, he kind of explains why historically vertically industries are pretty stovepiped, they have their own stack, sales and marketing and engineering and supply chains, et cetera, and experts within those industries tend to stay within those industries and they're largely insulated from disruption from other industries, maybe unless they were part of a supply chain. But today, you see all kinds of cross industry activity. Amazon entering grocery, entering media. Apple in finance and potentially getting into EV. Tesla, eyeing insurance. There are many, many, many examples of tech giants who are crossing traditional industry boundaries. And the reason is because of data. They have the data. And they're applying machine intelligence to that data and improving. Auto manufacturers, for example, over time they're going to have better data than insurance companies. DeFi, decentralized finance platforms going to use the blockchain and they're continuing to improve. Blockchain today is not great performance, it's very overhead intensive all that encryption. But as they take advantage of this new processing power and better software and AI, it could very well disrupt traditional payment systems. And again, so many examples here. But what I want to do now is dig into enterprise AI a bit. And just a quick reminder, we showed this last week in our Armv9 post. This is data from ETR. The vertical axis is net score. That's a measure of spending momentum. The horizontal axis is market share or pervasiveness in the dataset. The red line at 40% is like a subjective anchor that we use. Anything above 40% we think is really good. Machine learning and AI is the number one area of spending velocity and has been for awhile. RPA is right there. Very frankly, it's an adjacency to AI and you could even argue. So it's cloud where all the ML action is taking place today. But that will change, we think, as we just described, because data's going to get pushed to the edge. And this chart will show you some of the vendors in that space. These are the companies that CIOs and IT buyers associate with their AI and machine learning spend. So it's the same XY graph, spending velocity by market share on the horizontal axis. Microsoft, AWS, Google, of course, the big cloud guys they dominate AI and machine learning. Facebook's not on here. Facebook's got great AI as well, but it's not enterprise tech spending. These cloud companies they have the tooling, they have the data, they have the scale and as we said, lots of modeling is going on today, but this is going to increasingly be pushed into remote AI inference engines that will have massive processing capabilities collectively. So we're moving away from that peak centralization as Satya Nadella described. You see Databricks on here. They're seen as an AI leader. SparkCognition, they're off the charts, literally, in the upper left. They have extremely high net score albeit with a small sample. They apply machine learning to massive data sets. DataRobot does automated AI. They're super high in the y-axis. Dataiku, they help create machine learning based apps. C3.ai, you're hearing a lot more about them. Tom Siebel's involved in that company. It's an enterprise AI firm, hear a lot of ads now doing AI and responsible way really kind of enterprise AI that's sort of always been IBM. IBM Watson's calling card. There's SAP with Leonardo. Salesforce with Einstein. Again, IBM Watson is right there just at the 40% line. You see Oracle is there as well. They're embedding automated and tele or machine intelligence with their self-driving database they call it that sort of machine intelligence in the database. You see Adobe there. So a lot of typical enterprise company names. And the point is that these software companies they're all embedding AI into their offerings. So if you're an incumbent company and you're trying not to get disrupted, the good news is you can buy AI from these software companies. You don't have to build it. You don't have to be an expert at AI. The hard part is going to be how and where to apply AI. And the simplest answer there is follow the data. There's so much more to the story, but we just have to leave it there for now and I want to summarize. We have been pounding the table that the post x86 era is here. It's a function of volume. Arm volumes are a way for volumes are 10X those of x86. Pat Gelsinger understands this. That's why he made that big announcement. He's trying to transform the company. The importance of volume in terms of lowering the cost of semiconductors it can't be understated. And today, we've quantified something that we haven't really seen much of and really haven't seen before. And that's that the actual performance improvements that we're seeing in processing today are far outstripping anything we've seen before, forget Moore's Law being dead that's irrelevant. The original finding is being blown away this decade and who knows with quantum computing what the future holds. This is a fundamental enabler of AI applications. And this is most often the case the innovation is coming from the consumer use cases first. Apple continues to lead the way. And Apple's integrated hardware and software model we think increasingly is going to move into the enterprise mindset. Clearly the cloud vendors are moving in this direction, building their own custom silicon and doing really that deep integration. You see this with Oracle who kind of really a good example of the iPhone for the enterprise, if you will. It just makes sense that optimizing hardware and software together is going to gain momentum, because there's so much opportunity for customization in chips as we discussed last week with Arm's announcement, especially with the diversity of edge use cases. And it's the direction that Pat Gelsinger is taking Intel trying to provide more flexibility. One aside, Pat Gelsinger he may face massive challenges that we laid out a couple of posts ago with our Intel breaking analysis, but he is right on in our view that semiconductor demand is increasing. There's no end in sight. We don't think we're going to see these ebbs and flows as we've seen in the past that these boom and bust cycles for semiconductor. We just think that prices are coming down. The market's elastic and the market is absolutely exploding with huge demand for fab capacity. Now, if you're an enterprise, you should not stress about and trying to invent AI, rather you should put your focus on understanding what data gives you competitive advantage and how to apply machine intelligence and AI to win. You're going to be buying, not building AI and you're going to be applying it. Now data as John Furrier has said in the past is becoming the new development kit. He said that 10 years ago and he seems right. Finally, if you're an enterprise hardware player, you're going to be designing your own chips and writing more software to exploit AI. You'll be embedding custom silicon in AI throughout your product portfolio and storage and networking and you'll be increasingly bringing compute to the data. And that data will mostly stay where it's created. Again, systems and storage and networking stacks they're all being completely re-imagined. If you're a software developer, you now have processing capabilities in the palm of your hand that are incredible. And you're going to rewriting new applications to take advantage of this and use AI to change the world, literally. You'll have to figure out how to get access to the most relevant data. You have to figure out how to secure your platforms and innovate. And if you're a services company, your opportunity is to help customers that are trying not to get disrupted are many. You have the deep industry expertise and horizontal technology chops to help customers survive and thrive. Privacy? AI for good? Yeah well, that's a whole another topic. I think for now, we have to get a better understanding of how far AI can go before we determine how far it should go. Look, protecting our personal data and privacy should definitely be something that we're concerned about and we should protect. But generally, I'd rather not stifle innovation at this point. I'd be interested in what you think about that. Okay. That's it for today. Thanks to David Foyer, who helped me with this segment again and did a lot of the charts and the data behind this. He's done some great work there. Remember these episodes are all available as podcasts wherever you listen, just search breaking it analysis podcast and please subscribe to the series. We'd appreciate that. Check out ETR's website at ETR.plus. We also publish a full report with more detail every week on Wikibon.com and siliconangle.com, so check that out. You can get in touch with me. I'm dave.vellante@siliconangle.com. You can DM me on Twitter @dvellante or comment on our LinkedIn posts. I always appreciate that. This is Dave Vellante for theCUBE Insights powered by ETR. Stay safe, be well. And we'll see you next time. (bright music)

>>Hello, and welcome to today's event, dealing government emergency responses beyond the pandemic. This is Bob Wooley, senior fellow for the center for digital government and formerly the chief tech clerk for the state of Utah. I'm excited to serve as moderator for today's event. And just want to say, thank you for joining us. I know we're in for an informative session over the next 60 minutes before we begin a couple of brief housekeeping notes or recording of this presentation will be emailed to all registrants within 48 hours. You can use the recording for your reference or feel free to pass it along to colleagues. This webcast is designed to be interactive and you can participate in Q and a with us by asking questions at any time during the presentation, you should see a Q and a box on the bottom left of the presentation panel. >>Please send in your questions as they come out throughout the presentation, our speakers will address as many of these questions as we can during the Q and a portion of the close of our webinar today, if you would like to download the PDF of the slides for this presentation, you can do so by clicking the webinar resources widget at the bottom of the console. Also during today's webinar, you'll be able to connect with your peers by LinkedIn, Twitter and Facebook. Please use the hashtag gov tech live to connect with your peers across the government technology platform, via Twitter. At the close of the webinar, we encourage you to complete a brief survey about the presentation. We would like to hear what you think if you're unable to see with us for the entire webinar, but we're just like to complete the survey. As much as you're able, please click the survey widget at the bottom of the screen to launch the survey. Otherwise it will pop up once the webinar concludes at this time, we recommend that you disable your pop-up blockers, and if you experiencing any media player issues or have any other problems, please visit our webcast help guide by clicking on the help button at the bottom of the console. >>Joining me today to discuss this very timely topic are Karen revolt and Tim Burch, Kim Berge currently serves as the administrator of human services for Clark County Nevada. He's invested over 20 years in improving health and human service systems of care or working in the private public and nonprofit sectors. 18 of those years have been in local government in Clark County, Las Vegas, where you served in a variety of capacities, including executive leadership roles as the director of department of social services, as well as the director for the department of family services. He has also served as CEO for provider of innovative hosted software solutions, as well as chief strategy officer for a boutique public sector consulting firm. Karen real-world is the social program management offering lead for government health and human services with IBM Watson health. Karen focuses delivering exciting new offerings by focusing on market opportunities, determining unmet needs and identifying innovative solutions. >>Much of her career has been in health and human services focused on snap, TANIF, Medicaid, affordable care act, and child welfare prior to joining IBM. Karen was the senior director of product management for a systems integrator. She naturally fell in love with being a project manager. She can take her user requirements and deliver offerings. Professionals would use to make their job easier and more productive. Karen has also found fulfillment in working in health and human services on challenges that could possibly impact the outcome of people's lives. Now, before we begin our discussion of the presentation, I want to one, we'd like to learn a little more about you as an audience. So I'm going to ask you a polling question. Please take a look at this. Give us an idea of what is your organization size. I won't bother to read all these to you, but there are other a range of sizes zero to 250 up to 50,000. Please select the one that is most appropriate and then submit. >>It looks like the vast majority are zero to two 50. Don't have too many over 250,000. So this is a very, very interesting piece of information. Now, just to set up our discussion today, what I want to do is just spend just a moment and talk about the issue that we're dealing with. So when you look the COVID-19 pandemic, it's put immense pressure on States. I've been a digital state judge and had been judging a lot of the responses from States around the country. It's been very interesting to me because they bifurcate really into two principle kinds of reactions to the stress providing services that COVID environment present. One is we're in a world of hurt. We don't have enough money. I think I'm going to go home and engage as little as I have to. Those are relatively uncommon. Thankfully, most of them have taken the COVID-19 pandemic has immense opportunity for them to really do a lot more with telework, to do more with getting people, employees, and citizens involved with government services. >>And I've done some really, really creative things along the way. I find that to be a really good thing, but in many States systems have been overloaded as individuals and families throughout the country submitted just an unprecedented number of benefit applications for social services. At the same time, government agencies have had to contend with social distance and the need for a wholly different approach to engage with citizens. Um, overall most public agencies, regardless of how well they've done with technology have certainly felt some strain. Now, today we have the opportunity to go into a discussion with our speakers, have some wonderful experience in these areas, and I'm going to be directing questions to them. And again, we encourage you as you hear what they have to say. Be sure and submit questions that we can pick up later at the time. So Tim, let's start with you. Given that Las Vegas is a hub for hospitality. An industry hit severely as a result of this pandemic. How's the County doing right now and how are you prioritizing the growing needs of the County? >>Thanks Bob. Thanks for having me. Let me start off by giving just a little, maybe context for Clark County too, to our audience today. So, uh, Clark County is, you know, 85% of the state of Nevada if we serve not just as a regional County by way of service provision, but also direct municipal services. Well, if, uh, the famous Las Vegas strip is actually in unincorporated Clark County, and if we were incorporated, we would be the largest city in the state. So I say all of that to kind of help folks understand that we provide a mix of services, not just regional services, like health and human services, the direct and, and missable, uh, services as well as we work with our other five jurisdiction partners, uh, throughout the area. Uh, we are very much, um, I think during the last recession we were called the Detroit of the West. >>And, uh, that was because we're very much seen as a one industry town. Uh, so most like when the car plants, the coal plants closed back East and in the communities fuel that very rapidly, the same thing happens to us when tourism, uh, it's cut. Uh, so of course, when we went into complete shutdown and March, uh, we felt it very rapidly, not just on, uh, uh, tax receipts and collectibles, but the way in which we could deliver services. So of course our first priority was to, uh, like I think you mentioned mobilized staff. We, we mobilized hundreds of staff overnight with laptops and phones and cars and the things they needed to do to get mobile and still provide the priority services that we're mandated to provide from a safety standpoint. Um, and then we got busy working for our clients and that's really where our partnership with IBM and Watson, uh, came in and began planning that in July. And we're able to open that portal up in October to, to really speed up the way in which we're giving assistance to, to our residents. Um, re focus has been on making sure that people stay housed. We have, uh, an estimated, uh, 2.5 million residents and over 150,000 of those households are anticipated to be facing eviction, uh, as of January one. So we, we've got a, a big task ahead of us. >>All of this sounds kind of expensive. Uh, one of the common threads as you know, runs throughout government is, ah, I don't really have the money for that. I think I'd be able to afford that a diaper too, as well. So what types of funding has been made available for counties, a result of a pandemic, >>Primarily our funding stream that we're utilizing to get these services out the door has been the federal cares act. Uh, now we had some jurisdictions regionally around us and even locally that prioritize those funds in a different way. Um, our board of County commissioners, uh, took, um, a sum total of about $85 million of our 240 million that said, this will go directly to residents in the form of rental assistance and basic needs support. No one should lose their home or go hungry during this pandemic. Uh, so we've really been again working through our community partners and through our IBM tools to make sure that happens. >>So how does, how does, how does the cares act funding then support Clark County? Cause it seems to me that the needs would be complex, diverse >>Pretty much so. So as you, as folks may know him a call there's several tronches of the cares act, the original cares act funding that has come down to us again, our board, uh, identified basic needs or rental assistance and, and gave that the department of social service to go to the tunicate, uh, through the community. We then have the cares act, uh, uh, coronavirus relief funds that have, uh, impacted our CDBG and our emergency solutions grants. We've taken those. And that's what we was going to keep a lot of the programs and services, uh, like our IBM Watson portal open past January one when the cares act dollars expire. Uh, our initial response was a very manual one, uh, because even though we have a great home grown homeless management information system, it does not do financials. Uh, so we had 14 local nonprofits adjudicating, uh, this rental assistance program. >>And so we could get our social service visitor portal up, uh, to allow us to take applications digitally and run that through our program. Uh, and, uh, so those partners were obviously very quickly overwhelmed and were able to stand up our portal, uh, which for the reason we were driving so hard, even from, uh, beginning of the conversations where after going into lockdown into contracting in July and getting the portal open in October, which was an amazing turnaround. Uh, so the kudos that IBM team, uh, for getting us up and out the door so quickly, uh, was a tie in, uh, to our, uh, Curam IBM, uh, case management system that we utilize to adjudicate benefits on daily basis in Clark County for all our local indigent population, uh, and high needs folks. Uh, and then that ties into our SAP IBM platform, which gets the checks out the door. >>So what, what we've been able to do with these dollars is created in Lucian, uh, that has allowed us in the last 60 days to get as much money out the door, as our nonprofits were able go out the door in the first six months pandemic. So it really has helped us. Uh, so I'm really grateful to our board of County commissioners for recognizing the investment in technology to, to not only get our teams mobile, but to create ease of access for our constituents and our local residents to give them the help they need quickly and the way that they need it. >>Just to follow up question to that, Tim, that I'm curious about having done a lot of work like this in government, sometimes getting procurement through in a timely way is a bit challenging. How were you able to work through those issues and getting this up and provision so quickly? >>Uh, yeah, so we, we put together a, what we call a pandemic playbook, which is kind of lessons learned. And what we've seen is the folks who were essential workers in the first 60 days of the, uh, pandemic. We were able to get a lot done quickly because we were taking full advantage of the emergency. Uh, it may sound a little crass to folks not inside the service world, but it was, uh, you know, don't want you to crisis. It was things we've been planning or trying to do for years. We need them yesterday. We should have had them yesterday, but let's get them tomorrow and get it moving very quickly. Uh, this IBM procurement was something we were able to step through very quickly because of our longstanding relationship. Our countywide, uh, system of record for our financials is SAP. Uh, we've worked with Curam, uh, solution, uh, for years. >>So we've got this long standing relationship and trust in the product and the teams, which helped us build the business case of why we did it, no need to go out for competitive procurement that we didn't have time. And we needed something that would integrate very quickly into our existing systems. Uh, so that part was there. Now when the folks who were non essential came back in June and the reopening, it was whiplash, uh, the speed at which we were moving, went back to the pace of normal business, uh, which feels like hitting a wall, doing a hundred miles an hour when you're used to having that, uh, mode of doing business. Uh, so that's certainly been a struggle, uh, for all of those involved, uh, in trying to continue to get things up. Um, but, uh, once again, the teams have been great because we've probably tripled our licensure on this portal since we opened it, uh, because of working with outside vendors, uh, to, uh, literally triple the size of our staff that are processing these applications by bringing on temporary staff, uh, and short-term professionals. Uh, and so we've been able to get those things through, uh, because we'd already built the purchasing vehicle during the early onset of the crisis. >>That's very helpful. Karen, IBM has played a really pivotal role in all of this. Uh, IBM Watson health works with a number of global government agencies, raging from counties like Clark County to federal governments. What are some of the major challenges you've seen with your clients as a result of the pandemic and how is technology supporting them in a time of need and give us some background Watson health too. So we kind of know a little more about it because this is really a fascinating area. >>Yeah. Thank you, Bob. And thanks Tim for the background on Clark County, because I think Clark County is definitely also an example of what federal governments and global governments are doing worldwide today. So, um, Watson health is our division within IBM where we really focus on health and human services. And our goal is to really focus in on, um, the outcomes that we're providing to individuals and families and looking at how we use data and insights to really make that impact and that change. And within that division, we have our government health and human services area, which is the focus of where we are with our clients around social program. But it also allows us to work with, um, different agencies and really look at how we can really move the ball in terms of, um, effecting change and outcomes for, um, really moving the needle of how we can, uh, make an impact on individuals and families. >>So as we look at the globe globally as well, you know, everything that Tim had mentioned about how the pandemic has really changed the way that government agencies operate and how they do services, I think it's amazing that you have that pandemic playbook because a lot of agencies in the same way also had these set of activities that they always wanted to go and take part on, but there was no impetus to really allow for that to happen. And with the pandemic, it allowed that to kind of open and say, okay, we can try this. And unfortunately I'm in a very partial house way to do that. And, um, what Tim has mentioned about the new program that they set up for the housing, some of those programs could take a number of years to really get a program online and get through and allowing, uh, the agencies to be able to do that in a matter of weeks is amazing. >>And I think that's really gonna set a precedent as we go forward and how you can bring on programs such as the housing and capability in Canada with the economic, uh, social, um, uh, development and, and Canada need that the same thing. They actually had a multi benefit delivery system that was designed to deliver benefits for three programs. And as part of the department of fisheries and oceans Canada, the, um, the state had an emergency and they really need to set up on how they could provide benefits to the fishermen who had been at that impacted, um, from that. And they also did set up a digital front-end using IBM citizen engagement to start to allow the applications that benefits, um, and they set it up in a matter of weeks. And as I mentioned, we, uh, Clark County had a backend legacy system where they could connect to and process those applications. And this case, this is a brand new program and the case management system that they brought up was on cloud. And they had to set up a new one, but allow them to set up a, what we used to call straight through processing, I think has been now turned, turned or coined contact less, uh, processing and allowing us to really start to move those benefits and get those capabilities out to the citizens in even a faster way than has been imagined. Uh, pre pandemic. >>Karen, I have one follow-up question. I want to ask you, having had a lot of experience with large projects in government. Sometimes there's a real gap between getting to identified real requirements and then actions. How do you, how do you work with clients to make sure that process time to benefit is shortened? >>So we really focus on the user themselves and we take a human centered design focus and really prioritizing what those needs are. Um, so working with the clients, uh, effectively, and then going through agile iterations of brain, that capability out as, um, in, in a phased approach to, so the idea of getting what we can bring out that provides quality and capability to the users, and then over time starting to really roll out additional functions and, um, other, uh, things that citizens or individuals and families would need >>Very helpful. Tim, this is an interesting partnership. It's always good to see partnerships between private sector and government. Tell us a little bit about how the partnership with IBM Watson health was established and what challenges or they were brought into assist, where they brought into assist with back to requirements. Again, within the requirements definitely shifted on us. You know, we had the con looking at, uh, Watson on our child welfare, uh, side of the house that I'm responsible for and how that we could, uh, increase access to everything from tele-health to, to, uh, foster parent benefit, uh, kinship, placement benefits, all those types of things that, that right now are very manual, uh, on the child welfare side. Uh, and then the pandemic kid. And we very quickly realized that we needed, uh, to stand up a, um, a new program because, uh, a little bit for context, uh, the park County, we don't administer TANIF or Medicaid at the County level. >>It is done at the state level. So we don't have, uh, unemployment systems or Medicaid, 10 of snap benefits systems to be able to augment and enroll out. We provide, uh, the indigent supports the, the homelessness prevention, referee housing continuum of care, long-term care, really deep emergency safety net services for our County, which is a little bit different and how those are done. So that was really our focus, which took a lot of in-person investigation. We're helping people qualify for disability benefits so they can get into permanent supportive housing, uh, things that are very intensive. And yet now we have a pandemic where we need things to happen quickly because the cares act money expires at the end of December. And people were facing eviction and eviction can help spread exposure to, to COVID. Uh, so, uh, be able to get in and very rapidly, think about what is the minimal pelvis to MVP. >>What's the minimum viable product that we can get out the door that will help people, uh, entrance to a system as contactless as possible, which again was a complete one 80 from how we had been doing business. Um, and, uh, so the idea that you could get on and you have this intelligent chat bot that can walk you through questions, help you figure out if you look like you might be eligible, roll you right into an application where you can upload the few documents that we're going to require to help verify your coat would impact and do that from a smartphone and under, you know, 20 minutes. Um, it, it, it is amazing. And the fact that we've stood that up and got it out the door in 90 days, it's just amazing to me, uh, when it shows the, uh, strength of partnership. Um, I think we can, we have some shared language because we had that ongoing partnership, but we were able to actually leverage some system architects that we had that were familiar with our community and our other products. So it really helped expedite, uh, getting this, uh, getting this out to the citizens. >>So, uh, I assume that there are some complexities in doing this. So overall, how has this deployment of citizen engagement with Watson gone and how do you measure success other than you got it out quick? How do you know if it's working? >>Yeah. Right. So it's the adage of, you know, quick, fast and good, right. Um, or fast, good and cheap. So, uh, we measure success in this way. Um, how are we getting access as our number one quality measurement here? So we were able to collect, uh, about 13,000 applications, uh, manual NRC, manually folks had to go onto our website, download a PDF, fill it out, email it, or physically drop it off along with their backup. One of their choice of 14 non-profits in town, whichever is closest to them. Um, and, uh, and then wait for that process. And they were able to get 13,000 of those, uh, process for the last six months. Uh, we have, I think we had about 8,000 applications the first month come into the portal and about an equal amount of folks who could not provide the same documentation that it was needed. >>And self-selected out. If we had not had the, the tool in place, we would have had 16,000 applications, half of which would have been non-eligible would have been jamming up the system, uh, when we don't have the bandwidth to deal to deal with that, we, we need to be able to focus in on, uh, Judy Kenny applications that we believe are like a 95% success rate from the moment our staff gets them, but because we have the complex and he was on already being dependent upon the landlord, having to verify the rent amount and be willing to work with us, um, which is a major hurdle. Um, but, uh, so w we knew we could not do is go, just reinvent the manual process digitally that that would have been an abject failure on our behalf. So, uh, the ideas that, uh, folks had can go on a very, had this very intuitive conversation to the chat bot, answer some questions and find out if they're eligible. >>And then self-select out was critical for us to not only make sure that the citizens got the help they needed, but not so burnt out and overload our workforce, which is already feeling the strain of the COVID pandemic on their own personal lives and in their homes and in the workplace. Um, so that was really critical for us. So it's not just about speed, ease of access was important. Uh, the ability to quickly automate things on the fly, uh, we have since changed, uh, the area median income, a qualifier for the rental assistance, because we were able to reallocate more money, uh, to the program. So we were able to open it up to more people. We were able to make that, uh, change to the system very quickly. Uh, the idea that we can go on the home page and put updates, uh, we recognized that, uh, some of our monolingual Hispanic residents were having difficulty even with some guidance getting through the system. >>So we're able to record a, a Spanish language walkthrough and get done on the home page the next day, right into the fordable, there'll be a fine, so they could literally run the YouTube video while they're walking through their application. Side-by-side so things like that, that those are how we are able to, for us measured success, not just in the raw dollars out the door, not just in the number of applications that have come in, but our ability to be responsive when we hear from our constituents and our elected officials that, Hey, I want, I appreciate the 15,000 applications as you all, a process and record time, I've got three, four, five, six, 10 constituents that having this type of problem and be able to go back and retool our systems to make them more intuitive, to do, be able to keep them responsive for us is definitely a measure of success and all of this, probably more qualitative than here we're looking >>For, but, uh, that's for us, that's important. Actually the qualitative side is what usually gets ignored. Uh, Karen, I've got a question that's a follow up for you on the same topic. How does IBM facilitate reporting within this kind of an environment given the different needs of stakeholders, online managers and citizens? What kinds of things do you, are you able to do >>So with, um, the influx of digitalization? I think it allows us to really take a more data-driven approach to start looking at that. So, as, as Tim was mentioning, you can see where potentially users are spending more time on certain questions, or if they're stuck on a question, you can see where the abandoned rate is. So using a more data-driven approach to go in to identify, you know, how do we actually go and, um, continue to drive that user experience that may not be something that we drive directly from the users. So I would say that analytics is really, uh, I think going to continue to be a driving force as government agencies go forward, because now they are capturing the data. But one thing that they have to be careful of is making sure that the data that they're getting is the right data to give them the information, to make the right next steps and decisions. >>And Tim, you know, use a really good example with, um, the chatbot in terms of, you know, with the influx of everything going on with COVID, the citizens are completely flooded with information and how do they get the right information to actually help them decide, can I apply for this chap program? Or should I, you know, not even try and what Tim mentioned just saved the citizens, you know, the people that may not be eligible a lot of time and going through and applying, and then getting denied by having that upfront, I have questions and I need answers. Um, so again, more data-driven of how do we provide that information? And, you know, we've seen traditionally citizens having to go on multiple website, web pages to get an answer to the question, because they're like, I think I have a question in this area, but I'm not exactly sure. And they, then they're starting to hunt and hunt and hunt and not even potentially get an answer. So the chocolate really like technology-wise helps to drive, you know, more data-driven answers to what, um, whether it's a citizen, whether it's, um, Tim who needs to understand how and where my citizens getting stuck, are they able to complete the application where they are? Can we really get the benefits to, um, this individual family for the housing needs >>Too many comments on the same thing. I know you have to communicate measures of success to County executives and others. How do you do that? I mean, are you, do you have enough information to do it? Yeah, we're able to, we actually have a standup meeting every morning where the first thing I learn is how many new applications came in overnight. How many of those were completed with full documentation? How many will be ported over into our system, assigned the staff to work, where they're waiting >>On landlord verification. So I can see the entire pipeline of applications, which helps us then determine, um, Oh, it's, it's not, you know, maybe urban legend is that folks are having difficulty accessing the system. When I see really the bottleneck there, it got gotten the system fine, the bottlenecks laying with our landlord. So let's do a landlord, a town hall and iterate and reeducate them about what their responsibilities are and how easy it is for them to respond with the form they need to attest to. And so it lets us see in real time where we're having difficulties, uh, because, uh, there's a constant pressure on this system. Not just that, uh, we don't want anyone to lose their home, uh, but these dollars also go away within a December. So we've got this dual pressure of get it right and get it right now. >>Uh, and so th the ability to see these data and these metrics on, on a daily basis is critical for us to, to continue to, uh, ModuLite our response. Um, and, and not just get comfortable are baked into well, that's why we developed the flowchart during requirements, and that's just the way things are gonna stay. Uh, that's not how you respond to a pandemic. Uh, and so having a tool and a partner that helps us, uh, stay flexible, state agile, I guess, to, to, to leverage some terminology, uh, is important. And, and it's, it's paid dividends for our citizens. Karen, again, is another up to the same thing. I'm kind of curious about one of the problems of government from time to time. And Tim, I think attest to this is how do you know when Dunn has been reached? How did you go about defining what done would look like for the initial rollout with this kind of a customer? >>So I think Doug, I guess in this case, um, is, is this, isn't able to get the benefits that they're looking for and how do we, uh, you know, starting from, I think what we were talking about earlier, like in terms of requirements and what is the minimum viable, um, part of that, and then you start to add on the bells and whistles that we're really looking to do. So, um, you know, our team worked with him to really define what are those requirements. I know it's a new program. So some of those policy decisions were still also being worked out as the requirements were being defined as well. So making sure that you are staying on top of, okay, what are the key things and what do we really need to do from a compliance standpoint, from a functionality, and obviously, um, the usability of how, uh, an assistant can come on and apply and, um, have those, uh, requirements, make sure that you can meet that, that version before you start adding on additional scope. >>Very helpful. Jim, what's your comment on this since I know done matters to you? Yeah. And look, I I've lived through a, again, multiple, uh, county-wide it implementations and some department wide initiatives as well. So I think we know that our staff always want more so nothing's ever done, uh, which is a challenge and that's on our side of the customer. Um, but, uh, for this, it really was our, our experience of recognizing the, the time was an essence. We didn't have a chance. We didn't have, uh, the space to get into these endless, uh, conversations, uh, the agile approach, rather than doing the traditional waterfall, where we would have been doing requirements tracking for months before we ever started coding, it was what do we need minimally to get a check in the hands of a landlord on behalf of a client, so they don't get evicted. >>And we kept just re honing on that. That's nice. Let's put that in the parking lot. We'll come back to it because again, we want to leverage this investment long term, uh, because we've got a we, and we've got the emergency solutions and CDBG, and then our, uh, mainstream, uh, services we brought on daily basis, but we will come back to those things speed and time are of the essence. So what do we need, uh, to, to get this? So a chance to really, um, educate our staff about the concepts of agile iteration, um, and say, look, this is not just on the it side. We're gonna roll a policy out today around how you're doing things. And we may figure out through data and metrics that it's not working next week, and we'll have to have that. You want it. And you're going to get the same way. >>You're getting updated guidance from the CDC on what to do and what not to do. Uh, health wise, you're getting the same from us, uh, and really to helping the staff understand that process from the beginning was key. And, uh, so, and, and that's, again, partnering with, with our development team in that way was helpful. Um, because once we gave them that kind of charter as I am project champion, this is what we're saying. They did an equally good job of staying on task and getting to the point of is this necessary or nice. And if it wasn't necessary, we put it in the nice category and we'll come back to it. So I think that's really helpful. My experience having done several hundred sheet applications also suggest the need for MBP matters, future stages really matter and not getting caught. My flying squirrels really matters. So you don't get distracted. So let's move on to, let's do a polling question before we go on to some of our other questions. So for our audience, do you have a digital front ends for your benefit delivery? Yes, no. Or we're planning to a lot of response here yet. There we go. Looks like about half, have one and half note. So that's an interesting question. What's going to one more polling question, learn a little more here. Has COVID-19 >>Accelerated or moved cloud. Yes, no. We already run a majority of applications on cloud. Take a moment and respond if you would, please. So this is interesting. No real acceleration was taken place and in terms of moving to cloud is not what I was expecting, but that's interesting. So let's go onto another question then. And Karen, let me direct this one to you, given that feedback, how do you envision technologies such as citizen engagement and watching the system will be used, respond to emergency situations like the pandemic moving forward? I mean, what should government agencies consider given the challenges? This kind of a pandemic is brought upon government and try to tie this in, if you would, what, what is the role of cloud in all of this for making this happen in a timely way? Karen, take it away. >>Okay. Thanks Bob. So as we started the discussion around the digital expansion, you know, we definitely see additional programs and additional capabilities coming online as we continue on. Um, I think, uh, agencies have really seen a way to connect with their citizens and families and landlords, um, in this case an additional way. And he prepared them like there were, uh, presuppose assumptions that the, um, the citizens or landlords really wanted to interact with agency face-to-face and have that high touch part. And I think, um, through this, the governments have really learned that there is a way to still have an impact on the citizen without having a slow, do a face to face. And so I think that's a big realization for them to now really explore other ways to digitally explain, expand their programs and capabilities. Another area that we touched on was around the AI and chat bot piece. >>So as we start to see capabilities like this, the reason why Clark County was able to bring it up quickly and everything was because it was housed on cloud, we are seeing the push of starting to move some of the workloads. I know from a polling question perspective that it's been, um, lighter in terms of getting, uh, moving to the cloud. But we have seen the surge of really chatbots. I think we've been talking about chatbots for a while now. And, um, agencies hadn't really had the ability to start to implement that and really put it into effect. But with the pandemic, they were able to bring things up and, you know, very short amount of time to solve, um, a big challenge of not having the call center be flooded and have a different way to direct that engagement between the citizen and the government. >>So really building a different type of channel for them to engage rather than having to call or to come into an office, which wasn't really allowed in terms of, um, the pandemic. Um, the other thing I'll touch on is, um, 10 mentioned, you know, the backlog of applications that are coming in and we're starting to see the, um, the increase in automation. How do we automate areas where it's administratively highly burdened, but it's really a way that we can start to automate those processes, to give our workers the ability to focus on more of those complex situations that really need attention. So we're starting to see where the trends of trying to push there of can we automate some of those processes, um, uh, uploading documents and verification documents is another way of like, trying to look at, is there a way that we can make that easier? >>Not only for the applicant that's applying, but also for the caseworker. So there's not having to go through that. Um, does the name match, um, the applicant, uh, information and what we're looking on here, and Bob, you mentioned cloud. So behind the scenes of, you know, why, uh, government agencies are really pushing the cloud is, um, you heard about, I mean, with the pandemic, you see a surge of applicants coming in for those benefits and how do we scale for that kind of demand and how do you do that in an inappropriate way, without the huge pressures that you put on to your data center or your staff who's already trying to help our citizens and applicants, applicants, and families get the benefits they need. And so the cloud, um, you know, proposition of trying, being able to be scalable and elastic is really a key driver that we've seen in terms of, uh, uh, government agencies going to cloud. >>We haven't really seen during a pandemic, the core competencies, some of them moving those to cloud, it's really been around that digital front end, the chat bot area of how do we start to really start with that from a cloud perspective and cloud journey, and then start to work in the other processes and other areas. Um, security is also huge, uh, focus right now with the pandemic and everything going online. And with cloud allows you to be able to make sure that you're secure and be able to apply the right security so that you're always covered in terms of the type of demand and, um, impact, uh, that is coming through >>Very helpful. Tim, I'm going to ask to follow up on this of a practical nature. So you brought this up very quickly. Uh, there's a certain amount of suspicion around state government County government about chatbots. How did you get a chat much and be functional so quickly? And were you able to leverage the cloud in this process? Yeah, so on the trust is important. Uh, and I'll go back to my previous statement about individuals being able to see upfront whether they believe they're eligible or not, because nothing will erode trust more than having someone in hours applying and weeks waiting to find out they were denied because they weren't eligible to begin with, uh, that erodes trust. So being able to let folks know right up front, here's what it looks like to be eligible, actually help us build some of that, uh, cause they don't feel like, uh, someone in the bureaucracy is just putting them through the ringer for no reason. >>Um, now in regard to how do we get the chat bot out? I will say, uh, we have a, uh, dynamic it and leadership, uh, team at the highest level of County government who we have been already having conversations over the last year about what it meant to be smart government, uh, the department of social service and family services that I'm responsible for. We're already, uh, hands up first in line, you know, Guinea pigs volunteering to be on the front end of, uh, certain projects. So w we have primed ourselves for, for some of this readiness in that aspect. Um, but for citizen trust, um, the timeliness of application right now is the biggest element of trust. Uh, so I've applied I've I feel like I put my housing future in your hands. Are you going to deliver and having the ability for us to rapidly scale up? >>Uh, we typically have 120 staff in the department of social service that, that are adjudicating benefits for programs on daily basis. We've doubled that with temporary staff, uh, through some partnerships, uh, we're, we're gonna, as of next week, probably have more temporary per professional staff helping an adjudicator applications. No, do full-time County staff, because again, this rush to get the dollars out, out the door. So having a system where I can easily, uh, ramp on new users and manage them without having to be solely dependent upon an already, uh, overworked it staff who were trying to support 37 other departments in the County, um, around infrastructure needs has been greatly helpful. Sounds to me like a strong outcome focus and one that seems to work. Let's move on now to our audience questions. We're getting close to the end of our time. So let's jump into some questions from the audience. A number of you have been asking about getting copies of today's presentation within the next 48 hours. Government technology will provide all attendees with the link to the recording for your reference, or to share with colleagues. Well, let's go to our first question. So this is an interesting one. And Karen, this is for you did IBM work with other counties and States to provide digital engagement portals. >>We did Bob, uh, we've worked, um, so globally we've provided guidance on this. We work closely with New York city. They've been the integral part of the development also with our citizen engagement offering. Um, we work closely with the States. So we worked with New York city. Um, North Carolina was also another state who, um, improved their, uh, citizen engagement piece, bring up their Medicaid and snap, um, applications along with Medicaid. COVID testing along that. And I mentioned, um, the economic and social development in Canada as well. And we also work with the ministry of social development in Singapore. So a number of our customers had put up, uh, a global, uh, or sorry, a citizen engagement frontend. And during this timeframe, >>Very helpful. I don't know how much did you hear your mom provide you, but how much did it cost for initial deployment and what are the ongoing costs in other words, is this thing going to be sustainable over time? >>Yeah, absolutely. So total, uh, to date, we've spent about a $1.8 million on development implementations and licensure. A big chunk of that again has been the rapid extended of licensure, uh, for this program. Um, I think over a third of that is probably licensing because again, we need to get the dollars out and we need staff to do that and making the short term several hundred thousand dollar investment in a professional support staff and having them be able to work this portal is much cheaper than the long-term investment of bringing on a staff, printing a job, uh, during a financial difficulty that we're facing, uh, the single largest fiscal cliff let's get into that us history. Um, so it's not smart to create jobs that have a 30 year, one way to retirement, uh, inside our in unionized government environment here. So having this, the staff that would come on and do this and get out the door on these federal dollars was critical for us. Um, and there is a $800,000 a year, I believe so ongoing costs associated with licensure and, and the programming support. Uh, but once again, we're going to be moving, um, our traditional services into this digital front end. We'll be continuing this because we're, we're, we're facing, it took us, I think, six and a half, seven years to come back from the previous recession. Undoubtedly, take a little longer to get back >>From this one. Here's another interesting question, I guess really primarily Tim Tim was the solution on primarily on premise or in the cloud. >>So we'll, we've done a mix. Uh, the, and I'm starting a lot of feedbacks. I don't know if you all can hear that or not, but the, uh, I think we went on prem for, uh, some people because of the, uh, bridge into our service case manager system, which is on prem. So we did some management there. I do believe the chat bot piece of it though is in the cloud. So we're bringing it down to, from one system to the other. Uh, and, and part of that was a student negotiations and costs and worrying about what long-term is that we have a very stated goal of moving, uh, our Curam platform, which is on-prem, this is the backend. So how are we? We, we set our IBM Watson, uh, portal up, uh, and moving all of that on cloud, uh, because I mean, we've got, uh, a workforce who, uh, has the ability to retire at a very high rate over the next five years. >>And, uh, having 24 seven support in the cloud is, is as a, someone who would be called to respond to emergency situations like the is, is a much better Cod deal for, for myself and the citizen. So migrating, uh, and, um, our typical on-prem stuff up into the cloud, uh, as we continue on this, uh, evolution of what IBM Watson, uh, and the plug into our Curam, uh, system looks like Karen related question for another user is the portal provided with Clara County and others linked to other third-party backend office apps, or can it be, >>Yeah, the answer is it can be it's interoperable. So through APIs, uh, rest, uh, however, um, assistance that they need to be integrated with can definitely be integrated with, uh, like, uh, Tim mentioned, we, we went to the case management solution, but it can be integrated with other applications as well. >>Tim, did you use some other backend third party apps with yours? Uh, we did not. Uh, again, just for speed of getting, uh, this MVP solution out the door. Uh, now what we do with that on the go forward, it is going to look different and probably will include some, another practical question. Given the cares funding should be expended by December. Can this application even be employed at this late date? And you want to take a cut at that? Yeah, for us, uh, once again, we brought up earlier, um, the emergency solutions grants and the community development block grants, which have a Corona virus, uh, CV traunch, each one of those, and those have two to three year expenditure timeframes on them. Uh, so we were going to leverage those to keep this system and some of these programs going once again, that the housing needs, uh, will outstrip our capacity for years to come. >>I guess probably I should have said upfront Las Vegas has one of the worst affordable housing inventories in the nation. Uh, so we know we're going to be facing a housing issue, um, because of this for, for a long time. So we'll be using those two traunches of dollars, ESE, ESPs, uh, CV CDBG, CB funds, uh, in addition to dollars earmarked through some, uh, recreational marijuana license fees that have been dedicated to our homelessness. And when you consider this housing, uh, stability program was part of that homelessness prevention. That's our funding mix locally. Very helpful. So questions maybe for bolts for you on this one, you can probably also teach respond is the system has been set up helping the small business community. Um, this user's been canvassing and the general feeling is that small businesses have been left behind and they've been unable to access funds. What's your response on that? Karen, do you want to take that first? >>Um, yes. So in terms of, uh, the security and sorry. Um, but, uh, can you repeat the last part of that? I just missed the last part when you >>Behind it, but unable to access funds. >>Uh, yeah, so I think from a funding perspective, there's different types of, I think what Tim mentioned in terms of the cares funding, there was different types of funding that came out from a government perspective. Uh, I think there were also other grants and things that are coming out one, uh, that we're still looking at. And I think as we go into the new year, it'll be interesting to see, you know, what additional funding, um, hopefully is, is provided. Uh, but in terms of creativity, we've seen other creative ways that organizations come together to kind of, uh, help with the different agencies, to provide some, some guidance to the community, um, and helping to, uh, provide efforts and, uh, maybe looking at different ways of, um, providing, uh, some of the capabilities that the, either at the County or at the state level that they're able to leverage. But Tim happy to maybe have you chime in here too. >>Yeah. So I'll first start with my wheelhouse and I'll expand out to, to some of my partners. Uh, so the primary, small business, we knew the idea was a daily basis inside this realm is going to be landlords. Uh, so actually this afternoon, we're doing a town hall with folks to be able to roll out, uh, which they will go to our portal to find a corporate landlord program. Uh, so that I seem a landlord for Camille the application pack and on behalf of a hundred residents, rather than us having to adjudicate a hundred individual applications and melon a hundred checks. Uh, so that is because we were listening to that particular segment of the, uh, the business community. Now I know early on, we were, we were really hoping that the, the paycheck protection program federally would have, uh, been dispersed in a way that helped our local small businesses. >>Uh, more we did a, our economic development team did a round of small business supports through our cares act. Uh, our quarterly unfortunate was not open yet. It was just about 15, 20 days shy. So we use, uh, another traditional grant mechanism that we have in place to dedicate that. Uh, but on a go forward board, willing to Congress passes something over the next 30 days, um, that if there's a round two of cares or some other programs, we absolutely now have a tool that we know we can create a digital opening for individuals to come figure out if they're eligible or not for whatever program it is, the it housing, the it, uh, small business operations supports, uh, and it would apply through that process and in a very lightweight, so we're looking forward to how we can expand our footprint to help all of the needs that are present in our community. This leads to another question which may be our last one, but this is an interesting question. How can agencies use COVID-19 as a proof point providing a low cost configurable solutions that can scale across government. Karen, do you want to respond to that? And then Tim also, >>Thanks, Bob. So I believe like, you know, some of the things that we've said in terms of examples of how we were able to bring up the solution quicker, I definitely see that scaling as you go forward and trying to really, um, focus in on the needs and getting that MVP out the door. Uh, and then Tim alluded to this as well. A lot of the change management processes that went into re-imagining what these processes look like. I definitely see a additional, you know, growth mindset of how do we get better processes in place, or really focusing on the core processes so that we can really move the ball forward and continuing to go that path of delivering on a quicker path, uh, leveraging cloud, as we mentioned of, um, some, some of the capabilities around the chat bot and other things to really start to push, um, uh, the capabilities out to those citizens quicker and really reduce that timeline that we have to take on the backend side, um, that that would be our hope and goal, um, given, you know, sort of what we've been able to accomplish and hoping using that as a proof point of how we can do this for other types of, uh, either programs or other processes. >>Yeah, I think, um, the, you know, the tool has given us capability now there, whether we use local leaders leverage that to the fullest really becomes a coming upon us. So do we take a beat, uh, when we can catch our breath and then, you know, work through our executive leadership to say, look, here's all the ways you can use this tool. You've made an enterprise investment in. Um, and I know for us, uh, at Clark County, we've stood up, uh, enterprise, uh, kind of governance team where we can come and talk through all of our enterprise solutions, uh, encourage our other department head peers, uh, to, to examine how you might be able to use this. Is there a way that, um, you know, parks and rec might use this to better access their scholarship programs to make sure that children get into youth sports leagues and don't get left out, uh, because we know youth suicide on the rise and they need something positive to do when this pandemic is clear, I'm there for them to get out and do those things. >>So the possibilities really are out there. It really becomes, um, how do we mind those internally? And I know that being a part of listservs and, uh, you know, gov tech and all the magazines and things are out there to help us think about how do we better use our solutions, um, as well as our IBM partners who are always eager to say, Hey, have you seen how they're using this? Um, it is important for us to continue to keep our imaginations open, um, so that we continue to iterate through this process. Um, cause I, I would hate to see the culture of, um, iteration go away with this pandemic. >>Okay. We have time for one final question. We've already addressed this in part two, and this one is probably for you and that you've used the cares act to eliminate some of the procurement red tape that's shown up. Well, how do you somehow that's been very positive. How do you see that impacting you going forward? What happens when the red tape all comes back? >>Yeah, so I think I mentioned a little bit, uh, about that when some of the folks who are deemed non essential came back during our reopening phases and they're operating at the speed of prior business and red tape where we had all been on this, these green tape, fast tracks, uh, it, it was a bit of a organizational whiplash. Uh, but it, for us, we've had the conversation with executive management of like, we cannot let this get in the way of what our citizens need. So like keep that pressure on our folks to think differently. Don't and, uh, we've gone so far as to, uh, even, uh, maybe take it a step further and investigate what had been done in, in, in Canada. Some other places around, um, like, like going right from in a 48 hour period, going from a procurement statement through a proof of concept and doing purchasing on the backside, like how can we even get this even more streamlined so that we can get the things we need quickly, uh, because the citizens don't understand, wait, we're doing our best, uh, your number 3000 and queue on the phone line that that's not what they need to hear or want to hear during times of crisis. >>Very helpful. Well, I want to be respectful of our one hour commitment, so we'll have to wrap it up here in closing. I want to thank everyone for joining us for today's event and especially a big, thank you goes to Karen and Tim. You've done a really great job of answering a lot of questions and laying this out for us and a special thanks to our partners at IBM for enabling us to bring this worthwhile discussion to our audience. Thanks once again, and we look forward to seeing you at another government technology event,

>>from around the globe. >>It's the >>Cube with digital coverage of AWS reinvent 2020 sponsored by Intel and AWS. Yeah. Hey, welcome back to the Cube. Virtual coverage of AWS reinvent 2020. I'm John for your host of the Cube. Virtual not there in person. We're doing remote interviews, bringing that content to you virtually obviously with the virtual vent over three weeks, Walter Wall coverage Got a great guest here. Norman Quijano, chief market officer for acoustic. Normally great to have you on the Cube. Great story. Want to get into independent marketing? Cloud all that good stuff? Thanks for joining me. >>It's a pleasure to be here, John. I'm excited to chat with you, and it's exciting during reinvent. >>Yeah, a lot of great stuff. I mean, just every year I just get kind of nerdy, and I nerd out on all the massive new stuff and some of its kind of, you know, futuristic not yet available, but most is. But let's get into what you guys do. So first tell me the story about acoustic and you guys were originally part of IBM. Spun out. And now independent Take us through what happened. >>Yeah, sure it's It's actually a super fascinating story overall, because in short, acoustic was created last year, July 2019, as a carve out from IBM. The interesting history is that over the course of about a decade, IBM said, this marketing technology space is pretty interesting. So it went and acquired a number of companies across multiple years. Hold it all together in what it called IBM Watson marketing ultimately and said, We're in the marketing technology space, unfortunately. Turns out that's probably not a core business for IBM. So a few years ago, someone said, Maybe we're not in this space. Let's see if we can put this car of this out. And so we were born last July were private equity owned and from, Ah, great history became a great new beginning. >>Awesome. So talk about the value proposition. You guys living here says You guys air the independent marketing cloud. Does that mean independent in the sense of you don't take a position on certain technologies or independent as a company? Just what does that mean? >>Why independent used to be a simple word, but it doesn't have so it's not so simple anymore. Now is it. You know what we mean by that is very straightforward one. We are private, and we are focused on marketing and marketers, and we are not beholden to other parts of the business that may be trying to serve back, office or finance or other elements in a business. And what we think that the marketer today which, as you know, marketers usually have the or one of the biggest I T budgets in a company. We think they need providers that are focused on their needs and their needs Home. >>Yeah, it's interesting. The Martek stack and I just had a conversation with the venture capitalists. Live on the kickoff of the program for the show Review it this pre cloud this cloud transition. Now you got all in cloud benefits of being cloud native. Right? So you kind of 2021. I think we're in this post covert era. You got to see a whole new set of advantages. Yeah, they'll still be hybrid. They'll still be on premise. But if you look at the all the Martek marketing technology stuff, it's just so much stuff and Salesforce just bought slack. You have Microsoft tea and the big guys, all these things, and you only have a departments don't have a lot >>of staff. It's not like eso. You need >>technology to try. Great. Do the heavy lifting. This is a big theme of of the Amazon reinvent culture. Using tech creates the customer value. Reduce the heavy lifting. How are you guys doing that? How do you serve customers >>in that competitive landscape? It's a well set up job, because the reality is that we have a lot. There's a lot of companies in the marketing technology space you can look at charts online there, 8000 companies evidently on. But the reality is that very few of those companies are trying to provide big sort of anti and solutions the way that we are and some of our large competitors are. But they're all at different stages of the revolution in the cloud, because most of the bigger companies in this space got their Martek capabilities through acquisition, and they may have to sort of carry forward a pre cloud, uh, technology stack with him. What we're trying to do is really two things. One. We moved our technology to the cloud and in particular, over 90% of our workload is on AWS now. And we're trying to find the integration points with our customers with their equally moving to public cloud like AWS, and give them the capability of being able to bring up capabilities quickly, particularly in something like email Be able to scale. Right? We're in the middle of the holiday season is the busiest time of the year for businesses to send email, and we wanna make sure that our customers can scale up. We want them to have that capability, and we wanna be able to take advantage of that so we don't have toe over invest in back end technology. We want marketers to feel as empowered as the CEO Who's yours. Oh, I'm all in on the cloud. Well, what about the marketers? They're the ones who should be using that, And and I think something like AWS and continue to grow and me and the capabilities that every part of AWS will continue to provide value to the marketers to the customer experience team as well as to the I T >>team. How are you guys using data and AI? Because I see seeing that huge part of every single product. It's one of those things that you see on and we've been saying for years Now it's kind of mainstream, the benefit of clouds. You get horizontal scalability of infrastructure. Now you get lamb Daniel containers and then you got data you can get vertically specialized within the app. So if you do the micro services or deconstruct the monolith, you could really provide point value and still get that data scale. So this opens up massive data intelligence opportunities, which every marketer wants to be data driven. S O R O r. Use the data to make a great user experience or customer experience. How do you guys see that? And acoustic. And what do you guys doing in the clouds around that you >>share? Well, first of all, somehow you got ahold of our are confidential roadmap because you just laid it out right there. And what you said, it's not so confidential. But the reality is it's market >>leading for sure. I mean, I think you can. That's the Holy Grail. I mean, >>it's where everyone wants to be. And we had at acoustic have a very specific philosophy. Is that we want to. We want to embrace data, and we mean, of course, on behalf of our customers. And we want to bring data to empower every every application in every part of the marketers business. And for better or worse, there's some marketing technology sort of have a little bit of, ah, little hands off with data, particularly if it's not their own data. We believe that whether it's first party, second party, third party data it needs to be brought into the marketing life cycle, and we are building or have built capabilities to do that. We believe in being open, believe in being ableto bring in all sorts of different data types, and then use that to build the best marketing campaigns and experiences for our customers and for their customer. And if you're not embracing all the types of data out there in creating a unique formula for each particular customer, you're not gonna deliver the best marketing >>experience. Yeah, I totally agree. And I think one of these things where modern applications there's two themes here. Modern applications and then completely programmable infrastructure for Amazon. And this again, I've been covering cloud for many, many years since the beginning of Cloud and I've looked at all the big three and I see Amazon's been clearly winning on the infrastructure of the service platform is a service. They Yeah, they have sass apps out there, but they have an ecosystem. Microsoft has their own strategy. Google the other you picked Amazon is a preferred partner. Could you share? Um Why? Why Amazon And what specifically does that enable you to dio a za company? Because, um, yeah, Amazon's huge and some people get nervous like Okay, I'm just gonna You're gonna eat me up and you're in a marketing focus, not a not a court. I don't have a core building block out there called Marketing Cloud like Oracle does or other companies by Amazon. >>Yeah, I think that you really sort of laid the landscape out well, and Amazon is very much a a full stack. And and there's so much maturity in AWS overall, which you don't necessarily see the sort of top to bottom maturity that you see in the other of the clouds and Amazon and all clouds, right? We we all want to be able to tap into micro services, so when we were trying to figure out what gave us the scalability that we needed, we were really focused on the ability to integrate at multiple touch points Theobald iti to scale up really fast because, like during the holiday season, were transacting billions of transactions. Whether it be emails that our customers are sending or SMS messages that they're sending so billions of transactions over a fixed period of time, we need to be able to scale quickly at an affordable price on We also believe that actually, a lot of marketing departments are going to start to realize the value of plugging into the service is available in a public cloud, particularly as they see things such as taking data from 33rd parties. Right? How did they get that into the system or taking their marketing stacks and ultimately may potentially putting those stacks in containers, right. How do you move it into a container and be able to quickly connect other micro services to that container? So we think that this is the absolute future of where the marketing department is gonna end up and we think Amazon and AWS could be a great partner because it gives you that global footprint gives you that ability to scale and gives you the richest set of services available right now. That was a really easy decision for us. >>Awesome stuff. Thanks for coming on. Normal. Really appreciate you laying out your vision of the cloud. Take a minute, real quick. We got a couple of minutes left. Put the plug out for acoustic. What do you guys looking to do? What's the value proposition? Give a plug for the company. >>Yeah, we we left talking about acoustic, and you can certainly visit us it acoustic dot com Acoustic is a full service marketing platform. We are modern, we are cloud based, and one of the things that we do is we specialize and focus on marketing and the marketing function. And if anybody out there is interested in finding out more, you can not only come to acoustic dot com. You can ping me because we believe that marketers are key decision makers and myself is our CMO wants to talk to every potential >>client number. Thanks for coming on. The Moncada you Chief market officer acoustic here featured on the Cube, but Adam's reinvent thanks for coming on. Thanks. It >>was a pleasure, John. >>Thank you. I'm John for hosting the Cube. More coverage after this short break. Stay with us form or Cube. Live coverage.

>>Hi, I'm Peter Burress. And welcome to another cute conversation. This one from BMC Helix is immersion days in Santa Clara Marriott in Santa Clara, California One of the biggest challenges that every IittIe organization faces. In fact, every business is how to start merging greater control through I t sm as well as greater change and evolve ability of systems through Dev ops. It's a big topic. A lot of folks looking at how best to do it. We've got a great person here to talk to us about it. Dick Stark is the president CEO of right star Dick. Welcome to the Cube. >>Well, thanks very much for having me. I really appreciate the opportunity beyond the Cube here. >>Excellent. Well, why don't we start? Tell us a little about right start? >>Sure. Right. Stars in I t sm consultancy and we happen to be a dev Ops consulted to say at the same time, we're also a BMC solution provider and lasting solution provider. Now, we've been a BMC solution provider for for 16 years, so we've been in this space a long time and we've earned several accolades up along the way. We made it into the Forrester I t s m service provider. It's not called a Magic Quadrant because that's what God gardener uses. But instead it's a wave report. And so we made it sort of into the far right hand quadrant there. And if you added up all the points we ended up in North America being rated number five out of all the different idea Sam Consultancy. So it's very proud about that. And then last year with BMC, we were the North American Solution provider of the year in the D S. M space. >>Well is an export person, I can tell you Congratulations. Those waves very seriously. Let's jump into this question, though off what does I t. S m from a technology and people in process standpoint have to do to accommodate some of the changes that are being founded and defusing out of the Hole Dev Ops world, which is just having an enormous impact on our I t thinks and does >>it really has. And you know, we've been in the space a long time and I t s m Sometimes I tell the words are interchangeable and there are about if you can believe this about three million people That ended up getting an Idol certification of some short like an Idol Foundation certificate. And over time, that's been have been a really a big, big deal. However, Idol now is lost, its luster just a little bit. And it's allowed Dev ops to sort of sneak in or add dollar whatever you won't want to call it, and I'd listen. Standing still, though, they've bounced back and bounce back in a hard way. And they've they've come up with what's now called Idle for an Idol For was just released this this year, and it takes some of those Dev ops principles, and it has its own value stream as well and is a result Idle for or agile idol or whatever you wanna call it now is taking a little bit stronger position. And when I say Dev ops principles, it's things like Collaborate. It's things like promote, it's It's things like operate and automate. It's It's It's all about it again. It's all about collaboration in some of these other values that that you'll see in Dev ops. I guess what what happened is we spent a lot of time on the Idol side of things, and we did things for process sake and a good example would be changed management and spent a lot of time putting together is change management processes per this idol framework. Okay, And what what happened is that a lot of the users then rebelled a little bit because it might take longer to go through and fill out all the paperwork of It's not paperwork the online tool set then to do a change than to actually perform the change itself. So I don't got a little bit of a bad rap. And so that's where this whole Dev ops thing has come in. And the whole idea right now is to get Dev and Ops under the Shame umbrella, because that's not typically very used to do. But it's, but it's certainly happening. >>Well, let's talk about why that intersections happening, right? So I'm gonna I'm gonna show a little bit of history from my perspective as well, you know, I told began, First of all, it started in some government agencies many years ago, but it started as the basis of it was How do we take better care of the assets with an I T. Which at the time were mainly hardware. In many respects, what we've seen happen over the last 25 30 years that Idol has been an extent. Is that the nature of the assets that I t recognizes? His acknowledges delivering value for the business has changed. We've gone from hardware to infrastructure is code. That's where Dev Ops is so many respects. What you're saying is that Iittle is now trying to bring the best of what it means to do a good job of asset management with a new class of assets. Namely, software is code infrastructures code, and that's where we have to have that marriage. I got that right. >>That's that's correct. And you don't want to have silent silos. You want to be a silo buster if if anything else. And I just wanted to mention something else that I think is kind of fun along with this Idol. Four. We now do what's called the Mars Lander simulation traded it replaced. If you've heard of the Apollo 13 simulation, will Mars four, even though it's idle for specific, it's really all about Dev ops, and I took the Mars board just about a month or so ago, and it's a lot of fun. You sit down and the whole objective is to get get to Mars and you're a business. So and you're going to be selling the data that you're going to collect along along the way. And so the whole idea is to is to make a profit, and you have all these different roles that you play. When I went through it, I was the release manager then. But you might have a business analyst. You might have a service desk person. You have vendors and a it's it's really it's very realistic that and typically like a lot of large enterprises, you start playing the game and it's just chaos, and you have to go back and try this over and over again until essentially you get it right. And I was surprised how easy it is to get sucked in. If you're in a big enterprise, your silent, you have a specific role that you have to d'oh and you have instructions how you're supposed to do that and you want to stick to it. Whatever you know, whatever your assignment is, you have to do that. But that's not the right thing to Dio. Remember, it's about collaboration. It's about transparency. It's been it's about posting your goals, posting the results and moving forward from from there. And so I was surprised how I got sucked into it. And so I can understand why we need to make some progress in this space. And it's all about getting people to change their behavior a little bit in some of these new tool set certainly help >>well, as well. You're going back to what you said. He used to be the three R's of any regime or rolls responsibilities and relationships, and so the roles have are evolving. But often it's just in name only the responsibilities. You know today it's still code. It still has to run on hard, where it's not a bunch of hamsters, they're doing things. But as you said, it's really the relationships amongst the various actors as we introduce more business people. As technology gets put into position to generate more revenue or to do more with customer experience, the relationships are being pressured, are being really pushed to evolve. So how do you see in your practice in right stars practice. How do you see the relationships between Dev ops and I T s M and the business starting to evolve so that you can have amore coherent, comprehensive view of how you make sister? Well, >>I think in that particular case, it's gonna take some time. I mean, it's not gonna happen overnight. I mean, that's why you have agile coaches, or that's while you have the scales agile, or the safe framework is because people don't get it. And they need to understand how to work together better with others. And so it's not gonna happen by just implementing a new new tool set turning the key and then say, OK, everything's gonna be fine. It's good to get the integration between the different tool sets. And the technology is certainly there to do that. But without having some instruction to begin with and having the door in users cooperate. You're not going to see that kind of kind of performance improvement or cost statements or whatever it is that you're looking for. You're not going to see that >>they're one of the biggest challenges in any changes. Abandonment. The user's ultimately abandoned. So as you look a tte. The ideas M tool set that you're utilizing mainly from being right is it is that there's a degree of there's always a degree of pedagogic tool away, it says. Here's how you should do things. What you're discovering is that tool set is really catalyzing. Helping to catalyze positive changes in your mind within a lot of your customer base is, well, the >>thing about Helix, and I'm very excited about this because we're making a lot of good progress with. He likes our customer base that we have right now and give you a good example. George Washing University were based in a D C. Area day. If they are, too, they've been a long time remedy customer. We've moved them to Helix, and then, just recently, when I say recently started a year ago in August, they moved to the BMC Chap Cat box platform. Then, this past August, they totally went cold turkey with chatbots throughout the entire university. That makes a tremendous difference in the performance and not just performance, but also on the cost and the efficiency that the university, particularly from a service management perspective, is providing to its university employees and to its students, just like you mentioned today in the keynote session that it's all about mobility. And practically practically all the students there rely on their their cellphone day in and day out. And so when they have a question at G W. If it's how do I get a new account? How do I get a park parking permit? G on the wireless in my dorm room isn't working. You don't pick up the phone and call. Nobody does that you texted at. And this is a chap off its power by IBM Watson, and it works great. And there's lots of good things that are gonna come out of that. For example, students, I think they probably still have to turn paper sent. You know, maybe that's all Elektronik Lee delivered, but I think you might still have to print out a paper and turn it into your professor. You know, I'm not sure, but bluebirds Anyway, you're probably you're probably gonna do this late at night when the service desk is an open. So what do you do if you can't get the printer to work? Well, you pick up your cell phone, you text in that That the issue and bingo. You've got a response. So those are the sorts of things that are gonna make for a tremendous amount of impact, and it's gonna cause people to change their behavior in really a good way. Another good example. We have another longtime hospital customer. They have a 24 by seven service desk. They're huge, and they pay a lot of money to operate that 24 by seven. But they hardly get any call said at night. Right? Because not that many people work. So why don't they just turn that and you start using chatbots and think of that the r A. Y. It's just incredible. And I think you're going to see more. And that more situations like that as we move forward. >>Dick start President CEO of right Starr. Yep. Thanks very much for being too. >>Thanks very much. Appreciate it. Okay. >>And what's going on? Peter Burress. You've been watching other cube conversation from BMC Helix immersion days in Santa Clara. Thanks very much. Next time

(dramatic music) >> Welcome back, everyone to theCUBE's headquarters in Palo Alto, I'm John Furrier, the host of theCUBE, founder of SiliconANGLE Media, we're here for Cube Conversation with Ryan Welsh, who's the founder of CEO of Kyndi. It's a hot startup, it's a growing startup, doing really well in a hot area, it's in AI, it's where cloud computing, AI, data, all intersect around IoT, RPA's been a hot trend everyone's on, they're in that as well, but really an interesting startup we want to profile here, Ryan, thanks for spending the time to come in and talk about the startup. >> Yeah, thanks for having me. >> So I love getting the startups in, because we get the real scoop, you know, what's real, what's not real, and also, practitioners also tell us the truth too, so we love to have especially founders in. So first, before we get started, tell 'em about the company, how old is your company, what's the core value proposition, what do you guys do? >> Yeah, we're four years old, we were founded in June 2014. The first two, three years were really fundamental research and developing some new AI algorithms. What we focus on is, we focused on building explainable AI products for government customers, pharmaceutical customers and financial services customers. So our-- >> Let's explain the AI, what does that mean, like how do you explain AI? AI works, especially machine learning, well AI doesn't really exist, 'cause it's really machine learning, and what is AI? So what is explainable AI? >> Yeah, for us, it's the ability of a machine to communicate with the user in natural language. So there's kind of two aspects to explainability. Some of the deep learning folks are grabbing onto it, and really what they're talking about with explainability is algorithmic transparency, but where they tell you how the algorithm works, they tell you the parameters that are being used. So I explain to you the algorithm, you can actually interrogate the system. For us, if our system's going to make a recommendation to you, you would want to know why it's making the recommendation, right? So for us, we're able to communicate with users in natural language, like it's another person, of why we make a recommendation, why we bring back a search result, why we do whatever it is as part of the business process. >> And you mentioned deep learning AI is obviously the buzzword everybody's talking about, I mean I'm a big fan of AI in the sense that hyping it up means my kids know what it is, and everybody say, hey Dad, love machine learning. They love AI 'cause it's got a futuristic sound to it, but deep learning is real, deep learning is about learning systems that learn, which means they need to know what's going on, right? So this learning loop, how does that work? Is that kind of where explainable AI needs to go? Is that where it's going, where if you can explain it and it's explainable, you can interrogate it, does it have a learning mechanism to it? >> I think there's two major aspects of intelligence. There's the learning aspect, then there's the reasoning aspect. So if you look back through the history of AI, current machine learning is phenomenal at learning from data, like you're saying, learning the patterns in the data, but its reasoning is actually pretty weak. It can do statistical inferencing, but in the field of symbolic AI, where there's inductive, deductive, abductive, analogical reasoning, kind of advanced reasoning, it's terrible at reasoning. Whereas the symbolic approaches are phenomenal at reasoning but can't learn from data. So what is AI? A sub-group of that is machine learning that can learn from data. Another sub-group of that, it's knowledge-based approaches, which can't learn from data, they are phenomenal at reasoning, and really the trend that we're seeing at the edge in AI, or kind of the cutting edge, is actually fusing those two paradigms together, which is effectively what we've done. You've seen DeepMind and Google Brain publish a paper on that earlier this year, you've seen Gary Marcus start to talk about that, so for us, explainability is kind of bringing together these two paradigms of AI, that can both learn from data, reason about data, and answer questions like, why are you giving me this recommendation. >> Great explanation. And I want to just ask you, what' the impact of that, because we've always talked in the old search world, meta-reasoning, you type in a misspelling on Google, and it says, there's the misspelling, okay, I get that, but what if is misspell the word all the time, can't Google figure out that I really want that word? So reasoning has been a hard nut to crack, big time. >> Well you have to acquire the knowledge first to combine bits of knowledge to then reason, right? But the challenge is acquiring the knowledge. So you have all these systems or knowledge-based approaches, and you have human beings on-site, professional services, building and managing your knowledge base. So that's been one of the hurdles for knowledge-based approaches. Now you have machine learning that can learn from data, one of the problems with that is, that you need a bunch of labeled data. So you're kind of trading off between handcrafted knowledge systems, handcrafted labeled systems which you can then learn from data. So the benefits of fusing the two together is you can use machine learning approaches to acquire the knowledge, as opposed to hand engineering it, and then you can put that in a form or a data model that you can then reason about. So the benefit is really it all comes down to customer. >> Awesome, great area, great concepts, we can go for an hour on this, I love this topic, I think it's super relevant, especially as cloud and automation become the key accelerant to a lot of new value. But let's get back to the company. So four years old, you've done some R and D, give me the stats, where are you guys in the product side, product shipping, what's the value proposition, how do people engage with you, just go down looking on the list. >> Yeah, yeah, shipping product to customers in pharmaceutical, and government use cases. How people engage with us-- >> It's a software product? >> It's a software product. Yeah, yeah. So we can deliver it, surprisingly a lot of customers still want it on-prem. (both laugh) But we can deploy in the cloud as well. Typically, how we work with customers is we'll have close engagements for specific use cases within pharma or government or financial services, because it's a very broad platform an can be applied to any text-based use case. So we work with them closely, develop a use case, they're able to sell that internally to champions >> And what problems are they solving, what specifically is the answer? >> So for pharmaceutical companies, a lot of their internal, historical clinical trial data, they'll develop memos, emails, notes as they bring a drug to market. How do you leverage that data now? Instead of just storing it, how do I find new and innovative ways to use existing drugs that someone in another part of the organization could have developed? How do I manage the risks within that historical clinical trial data? Are there people that are doing research incorrectly? Are they reporting things incorrectly? You know, this entire process of both getting drugs through the pipeline and managing drugs as they move through the pipeline, is a very manual process that revolves around text-based data sources. So how do you develop systems that amplify the productivity of the people that are developing the drugs, then also the people that are managing the process. >> And so what are you guys actually delivering as value? What's the value proposition for them? >> Yeah, so >> Is it time? >> It's saving time, but ultimately increasing their productivity of getting that work done. It's not replacing individuals, because there's so much work to do. >> So all the... The loose stuff like the paper, they can discover it faster, so they have more access to the data. >> That's right. >> Using your tools >> That's right >> and your software. >> You can classify things in certain ways, saying there's data integrity issues, you need to look at this closer, but ultimately managing that data. >> And that's where machine learning and some of these AI techniques matter, because you want to essentially throw software at that problem, accelerate that process of getting the data, bringing it in, assessing it. >> Yeah, I mean we spend most of our time looking for the information to then analyze. I mean we spend 80% of our time doing it, right? Where it's like are there ways to automate that process, so we can spend 80% of our time actually doing our job? >> So Ryan, who's the customer out there? So is it someone, someone's watching this video, and what's their pain point, when do they call you, why do they call you? What's some of the signals that might tell someone, hey I want to give these guys a call, I need this solution? >> Yeah, a lot of it comes down to the amount of manual labor that you're doing. So we see a lot of big expenses around people, because you haven't traditionally been able to automate that process, or to use software in that process. So if you actually look at your income statement and you say where am I spending my most money, on tons of people, and I'm just throwing people at the problem, that's typically where people engage with us and say, how do I amplify the productivity of these people so I can get more out of them, hopefully make them more efficient? >> And it's not just so much to reduce the head count issue, it's more of increasing the automation for saying value in top-line revenue, because if you have to reproduce people all the time, why not replicate that in software? So I think what I'm seeing is, get that right? >> That's exactly right. And the job consistently changes too, so it's not like this robotic process that you can just automate away. They're looking for certain things one day, then they're looking for certain things the next day, but you need a capability that kind of matches their expertise. >> You know, I was talking to a CIO the other day and we were talking about some of the things around reproducing things, replicating, and the notion of how things get scaled or moved along, growth, is, and the expression was "Throw a body at that". That's been IT. Outsource it. So throwing a body, or throw bodies at it, you know, throw that problem at me, that doesn't really end well. With software automation you can say, you don't just throw a body at it, you can say, if it can be automated, automate it. >> Yeah, here's what I think most people miss, is that we are the bottleneck in the modern production process because we can't read and understand information any faster than our parents or grandparents. And there's not enough people on the planet to increase our capacity, to push things through. So if we were to compare the modern knowledge economy, it's interesting, to the manufacturing process, you have raw materials, manufacture it, and end product. All these technologies that we have effectively stack information and raw materials at the front of it. We haven't actually automated that process. >> You nailed it, and in fact one of the things I would say that would support that is, in interviewed Dave Redskin, who's a site reliable engineer at Google, and we were talking about the history of how Google scaled, and they have this whole new program around how to operate large data centers. He said years and years ago at Google, they looked up the growth and said, we're going to need a thousand people per data center, at least, if not, per data center, so that means we need 15,000 people just to manage the servers. 'Cause what they did was they just did the operating cycle on provisioning servers, and essentially, they automated it all away, and they created a lot of the tools that became now Google Cloud. His point was, is that, they now have one person, site reliability engineer, who overlooks the entire automation piece. This is where the action is. That concept of not, to scale down the people focus, scale up the machine base model. Is that kind of the trend that you guys are riding? >> Absolutely. And I think that's why AI is hot right now. I mean, AI's been around since the late 40s, early 50s, but why this time I think it's different is, one, that it's starting to work, given the computational resources and the data that we have, but then also the economic need for it. Businesses are looking, and saying, how I historically address these problems, I can no longer address them that way, I can't hire 15,000 people to run my data center. I need to now automate-- >> You got to get out front on it. >> Yeah, I got to augment those people with better technologies to make them do the work better. >> All right, so how much does the product cost, how do people engage with you guys, what's the engagement cost, is it consulting you come in, POC you ship 'em software, to appliances in the cloud, you mention on-premise. >> Yeah, yeah. >> So what's, how's the product look, how much does it cost? >> Yeah, it costs a good chunk for folks, so typically north of 500K. We do provide a lot of ROI around that, hence the ability to charge such a high price. Typically how we push people through the cycle and how we actually engage with folks is, we do what we demonstration of value. So there's a lot of different, or typically there's about 15 use cases that any given Fortune 500 customer wants to address. We find the ones with the highest ROI, the ones with accessible data >> And they point at it, >> The ones with budget >> They think, that's my problem, they point to it, right? >> Yeah. >> It's not hard to find. >> We have to walk 'em through it a little bit. Hopefully they've engaged with other vendors in the market that have been pushing AI solutions for the last few years, and have had some problems. So they're coached up on that, but we engage with demonstration of value, we typically demonstrate that ROI, and then we transition that into a full operational deployment for them. If they have a private cloud, we can deploy on a private cloud. Typically we provide an appliance to government customers and other folk. >> So is that a pre-sale activity, and you throw bodies at it, on your team. What's the engagement required kind of like a... Then during that workshop if you will, call it workshop. You come in and you show some value. Kind of throw some people at it, right? >> Yeah, you got-- >> You have SE, and sales all that. >> Exactly right. Exactly right. So we'll have our sales person managing the relationship, an SE also interacting with the data, working with the system, working closely with a contact on the customer's side. >> And they typically go, this is amazing, let's get started. Do they break it up, or-- >> They break it up. It's an iterative process, 'cause a lot of times, people don't fully grasp the power of these capabilities, so they'll come through and say, hey can you just help us with this small aspect of it, and once you show 'em that I can manage all of your unstructured text data, I can turn it into this giant knowledge graph, on top of which I can build apps. Then the light kind of goes off and they go, they go, all right, I can see this being used in HR, marketing, I mean legal, everywhere. >> Yeah, I mean you open up a whole new insight engine basically for 'em. >> That's exactly right. >> So, okay, so competition. Who are you competing with? I mean, we've been covering UiPath, they just had an event in Miami. This is the hot area, who's competing with you, who are you up against, and how are you guys winning, why are you winning? >> Yeah, we don't compete with the RPA folks. You know there's interesting aspects there, and I think we'll chat about that. Mainly there are incumbents like IBM Watson that are out there, we think IBM has done phenomenal research over the last 60 years in the field of AI. But we do run into the IBMs, big consulting companies, a lot of the AI deployments that we see, candidly are from all the big consulting shops. >> And they're weak, or... They're weaker than yours. >> Yeah, I would argue yes. (both laugh) >> It's okay, get that out of your sleigh. >> I think one of the big challenges-- >> Is it because they just don't have the chops, or they're just recycling old tech into a-- >> We do have new novel algorithms. I mean, what's interesting is, and this has actually been quite hard for us, is coming out saying, we've taken a step beyond deep learning. We've take a step beyond existing approaches. And really it's fusing those two paradigms of AI together, 'cause what I want to do is to be able to acquire the knowledge from the data, build a giant knowledge graph, and use that knowledge graph for different applications. So yeah, we deploy our systems way faster than everyone else out there, and our system's fully explainable. >> Well I mean it's a good position to be in. At least from a marketing standpoint, you can have a leadership strategy, you don't need to differentiate in anyway 'cause you're different, right, so... >> Yeah, yeah >> Looks like you're in good shape. So easy marketing playbook there, just got to pound the pavement. RPA, you brought that up and I think that's certainly been an area. You mentioned you guys kind of dip into that. How do you, I mean that's not an area you would, you would fit well in there, so, I want to get you, well you're not positioning yourself as an RPA solution, but you can solve RPA challenges or those kinds of... Explain why you're not an RPA but you will play in it. >> Here's what's so fascinating about this market is, a lot of people in AI will knock the RPA guys as not being sophisticated approaches. Those guys are solving real business problems, providing real value to enterprises, and they are automating processes. Then you have sophisticated AI companies like ours, that are solving really really high-level white-collar worker tasks, and it's interesting, I feel like the AI community needs to kind of come down a step of sophistication, and the RPA companies are starting to come up a level of sophistication, and that's where you're starting to see that overlap. RPA companies moving from RPA to intelligence process automation, where AI companies can actually add value in the analysis of unstructured text data. So around natural language processing, natural language understanding. RPA companies no longer need to look at specific structured aspects and forms, but can actually move into more sophisticated extraction of things from text data and other-- >> Well I think it's not a mutually exclusive scenario anymore, as you mentioned earlier, there's a blending of the two machine learning and symbolics coming together in this new reasoning model. If you look at RPA, my view is it's kind of a dogmatic view of certain things. They're there to replace people, right (laughs) >> Yeah, totally. >> We got robotics, we don't need people on the manufacturing line, we just put some robotics on as an example. And AI's always been about getting the best out of the software and the data, so if you look at the new RPA that we see that's relevant is to your point, let's use machines to augment humans. A different, that's a cultural thing. So I think you're right, I think it's coming together in new ground where most people who are succeeding in data, if you will, data driven or AI, really have the philosophy that humans have to be getting the value. Like that SRE example, Google, so that's a fundamental thing. >> Absolutely. >> And okay, so what's next for you guys? Business is good? >> Business is good. >> Hiring, I'm imagining with your kind of community >> Always hiring phenomenal AI and ML expertise, if you have it, >> Good luck competing with Google >> Shoot us an email. >> And Google will think that you're hiring 'em all. How do you handle that, I mean... >> Yeah I mean they actually get to work on novel algorithms. I mean what's fascinating is a lot of the AI out there, I mean you can date it all the way back to Rumelhart and Hinton's paper from 1986. So I mean, we've had backprop for a while. If you want to come work on new, novel algorithms, that are really pushing the limit of what's possible, >> Yeah, if you're bored at Google or Facebook, check these guys out. >> Check us out. >> Okay, so funding, you got plenty of money in the bank, strategic partners, what's the vision, what's your goal for the next 12 months or so, what's your objective? >> Yeah, focusing big on the customers that we have now. I'm always big on having customers, get a viral factor within the B2B enterprise software space, get customers that are screaming from the mountaintop that this is the best stuff ever, then you can kind of take care of it. >> How about biz dev, partnerships, are you guys looking at an ecosystem? Obviously rising tide floats all boats, I mean I can almost imagine might salivate for some of the software you're talking about, like we have all this data, here inside theCUBE, we have all kinds of processes that are, we're trying to streamline, I mean, we need more software, I mean, can I buy your stuff? I mean we don't have half a million bucks, can I get a discount? I mean how do I >> We'll see. We'll see how we end up. >> I mean is there like a biz dev partner program? >> No, not... >> Forgetting about theCUBE, we'd love if that's so, but if it's to partner, do you guys partner? >> So not yet in exposing APIs to third parties. So I mean I would love if I had the balance sheet to go to market horizontally, but I don't. So it's go to market vertically, focus on specific solutions. >> Industries. >> Industries, pharma >> So you're sort of, you're industry-focused >> government, financial services. >> That's the ones you've got right now. >> They're the three. >> For now. >> Yep. >> Okay, so once you nail an industry, you move onto the next one. >> Yeah, then I would love expose APIs for tab partners to work on this stuff. I mean we see that every day someone wants to use certain engines that we have, or to embed them within applications. >> Well I mean you've got a nice vertical strategy. You've knocked down maybe one or two verticals. Then you kind of lay down a foundational... >> Yeah. >> Yeah, development platform. >> Yeah, that's right. >> That's your strategy. >> And we can be, I mean at Kyndi I think we can be embedded in every application out there that's looking at unstructured data >> Which is also the mark of maturity, you got to go where the customers are, and you know the vision of having this global platform could be a great vision, but you've got to meet the customers where they are, and where they are now is, solve my vertical problem. (laughs) >> Yeah, and for us, with new technologies, well, show me that they're better than other approaches. I can't go to market horizontally and just say, I have better AI than Google. Who's going to come beyond the Kyndi person? >> Well IBM's been trying to do it with Watson, and that's hard. >> It's very hard. >> And they end up specializing in industries. Well Ryan, thanks for coming on theCUBE, appreciate it. Kyndi, great company, check 'em out, they're hiring. We're going to keep an eye on these guys 'cause they're really hitting a part of the market that we think, here at theCUBE, is going to be super-powerful, it's really the intersection of a lot of major markets, cloud, AIs, soon to be blockchain, supply chain, data center of course, storage networking, this is IoT security and data at the center of all the action. New models can emerge, with you guys in the center, so thanks for coming and sharing your story, appreciate it. >> Thank you very much. >> I'm John Furrier, here in theCUBE studios in Palo Alto. Thanks for watching. (dramatic music)