(soft music) >> Hello, everyone. Welcome to theCUBE's presentation of the AWS Startup Showcase. AI and Machine Learning: Top Startups Building Foundational Model Infrastructure. This is season 3, episode 1 of the ongoing series covering the exciting stuff from the AWS ecosystem, talking about machine learning and AI. I'm your host, John Furrier and today we are excited to be joined by Luis Ceze who's the CEO of OctoML and Anna Connolly, VP of customer success and experience OctoML. Great to have you on again, Luis. Anna, thanks for coming on. Appreciate it. >> Thank you, John. It's great to be here. >> Thanks for having us. >> I love the company. We had a CUBE conversation about this. You guys are really addressing how to run foundational models faster for less. And this is like the key theme. But before we get into it, this is a hot trend, but let's explain what you guys do. Can you set the narrative of what the company's about, why it was founded, what's your North Star and your mission? >> Yeah, so John, our mission is to make AI sustainable and accessible for everyone. And what we offer customers is, you know, a way of taking their models into production in the most efficient way possible by automating the process of getting a model and optimizing it for a variety of hardware and making cost-effective. So better, faster, cheaper model deployment. >> You know, the big trend here is AI. Everyone's seeing the ChatGPT, kind of the shot heard around the world. The BingAI and this fiasco and the ongoing experimentation. People are into it, and I think the business impact is clear. I haven't seen this in all of my career in the technology industry of this kind of inflection point. And every senior leader I talk to is rethinking about how to rebuild their business with AI because now the large language models have come in, these foundational models are here, they can see value in their data. This is a 10 year journey in the big data world. Now it's impacting that, and everyone's rebuilding their company around this idea of being AI first 'cause they see ways to eliminate things and make things more efficient. And so now they telling 'em to go do it. And they're like, what do we do? So what do you guys think? Can you explain what is this wave of AI and why is it happening, why now, and what should people pay attention to? What does it mean to them? >> Yeah, I mean, it's pretty clear by now that AI can do amazing things that captures people's imaginations. And also now can show things that are really impactful in businesses, right? So what people have the opportunity to do today is to either train their own model that adds value to their business or find open models out there that can do very valuable things to them. So the next step really is how do you take that model and put it into production in a cost-effective way so that the business can actually get value out of it, right? >> Anna, what's your take? Because customers are there, you're there to make 'em successful, you got the new secret weapon for their business. >> Yeah, I think we just see a lot of companies struggle to get from a trained model into a model that is deployed in a cost-effective way that actually makes sense for the application they're building. I think that's a huge challenge we see today, kind of across the board across all of our customers. >> Well, I see this, everyone asking the same question. I have data, I want to get value out of it. I got to get these big models, I got to train it. What's it going to cost? So I think there's a reality of, okay, I got to do it. Then no one has any visibility on what it costs. When they get into it, this is going to break the bank. So I have to ask you guys, the cost of training these models is on everyone's mind. OctoML, your company's focus on the cost side of it as well as the efficiency side of running these models in production. Why are the production costs such a concern and where specifically are people looking at it and why did it get here? >> Yeah, so training costs get a lot of attention because normally a large number, but we shouldn't forget that it's a large, typically one time upfront cost that customers pay. But, you know, when the model is put into production, the cost grows directly with model usage and you actually want your model to be used because it's adding value, right? So, you know, the question that a customer faces is, you know, they have a model, they have a trained model and now what? So how much would it cost to run in production, right? And now without the big wave in generative AI, which rightfully is getting a lot of attention because of the amazing things that it can do. It's important for us to keep in mind that generative AI models like ChatGPT are huge, expensive energy hogs. They cost a lot to run, right? And given that model usage growth directly, model cost grows directly with usage, what you want to do is make sure that once you put a model into production, you have the best cost structure possible so that you're not surprised when it's gets popular, right? So let me give you an example. So if you have a model that costs, say 1 to $2 million to train, but then it costs about one to two cents per session to use it, right? So if you have a million active users, even if they use just once a day, it's 10 to $20,000 a day to operate that model in production. And that very, very quickly, you know, get beyond what you paid to train it. >> Anna, these aren't small numbers, and it's cost to train and cost to operate, it kind of reminds me of when the cloud came around and the data center versus cloud options. Like, wait a minute, one, it costs a ton of cash to deploy, and then running it. This is kind of a similar dynamic. What are you seeing? >> Yeah, absolutely. I think we are going to see increasingly the cost and production outpacing the costs and training by a lot. I mean, people talk about training costs now because that's what they're confronting now because people are so focused on getting models performant enough to even use in an application. And now that we have them and they're that capable, we're really going to start to see production costs go up a lot. >> Yeah, Luis, if you don't mind, I know this might be a little bit of a tangent, but, you know, training's super important. I get that. That's what people are doing now, but then there's the deployment side of production. Where do people get caught up and miss the boat or misconfigure? What's the gotcha? Where's the trip wire or so to speak? Where do people mess up on the cost side? What do they do? Is it they don't think about it, they tie it to proprietary hardware? What's the issue? >> Yeah, several things, right? So without getting really technical, which, you know, I might get into, you know, you have to understand relationship between performance, you know, both in terms of latency and throughput and cost, right? So reducing latency is important because you improve responsiveness of the model. But it's really important to keep in mind that it often leads diminishing returns. Below a certain latency, making it faster won't make a measurable difference in experience, but it's going to cost a lot more. So understanding that is important. Now, if you care more about throughputs, which is the time it takes for you to, you know, units per period of time, you care about time to solution, we should think about this throughput per dollar. And understand what you want is the highest throughput per dollar, which may come at the cost of higher latency, which you're not going to care about, right? So, and the reality here, John, is that, you know, humans and especially folks in this space want to have the latest and greatest hardware. And often they commit a lot of money to get access to them and have to commit upfront before they understand the needs that their models have, right? So common mistake here, one is not spending time to understand what you really need, and then two, over-committing and using more hardware than you actually need. And not giving yourself enough freedom to get your workload to move around to the more cost-effective choice, right? So this is just a metaphoric choice. And then another thing that's important here too is making a model run faster on the hardware directly translates to lower cost, right? So, but it takes a lot of engineers, you need to think of ways of producing very efficient versions of your model for the target hardware that you're going to use. >> Anna, what's the customer angle here? Because price performance has been around for a long time, people get that, but now latency and throughput, that's key because we're starting to see this in apps. I mean, there's an end user piece. I even seeing it on the infrastructure side where they're taking a heavy lifting away from operational costs. So you got, you know, application specific to the user and/or top of the stack, and then you got actually being used in operations where they want both. >> Yeah, absolutely. Maybe I can illustrate this with a quick story with the customer that we had recently been working with. So this customer is planning to run kind of a transformer based model for tech generation at super high scale on Nvidia T4 GPU, so kind of a commodity GPU. And the scale was so high that they would've been paying hundreds of thousands of dollars in cloud costs per year just to serve this model alone. You know, one of many models in their application stack. So we worked with this team to optimize our model and then benchmark across several possible targets. So that matching the hardware that Luis was just talking about, including the newer kind of Nvidia A10 GPUs. And what they found during this process was pretty interesting. First, the team was able to shave a quarter of their spend just by using better optimization techniques on the T4, the older hardware. But actually moving to a newer GPU would allow them to serve this model in a sub two milliseconds latency, so super fast, which was able to unlock an entirely new kind of user experience. So they were able to kind of change the value they're delivering in their application just because they were able to move to this new hardware easily. So they ultimately decided to plan their deployment on the more expensive A10 because of this, but because of the hardware specific optimizations that we helped them with, they managed to even, you know, bring costs down from what they had originally planned. And so if you extend this kind of example to everything that's happening with generative AI, I think the story we just talked about was super relevant, but the scale can be even higher, you know, it can be tenfold that. We were recently conducting kind of this internal study using GPT-J as a proxy to illustrate the experience of just a company trying to use one of these large language models with an example scenario of creating a chatbot to help job seekers prepare for interviews. So if you imagine kind of a conservative usage scenario where the model generates just 3000 words per user per day, which is, you know, pretty conservative for how people are interacting with these models. It costs 5 cents a session and if you're a company and your app goes viral, so from, you know, beginning of the year there's nobody, at the end of the year there's a million daily active active users in that year alone, going from zero to a million. You'll be spending about $6 million a year, which is pretty unmanageable. That's crazy, right? >> Yeah. >> For a company or a product that's just launching. So I think, you know, for us we see the real way to make these kind of advancements accessible and sustainable, as we said is to bring down cost to serve using these techniques. >> That's a great story and I think that illustrates this idea that deployment cost can vary from situation to situation, from model to model and that the efficiency is so strong with this new wave, it eliminates heavy lifting, creates more efficiency, automates intellect. I mean, this is the trend, this is radical, this is going to increase. So the cost could go from nominal to millions, literally, potentially. So, this is what customers are doing. Yeah, that's a great story. What makes sense on a financial, is there a cost of ownership? Is there a pattern for best practice for training? What do you guys advise cuz this is a lot of time and money involved in all potential, you know, good scenarios of upside. But you can get over your skis as they say, and be successful and be out of business if you don't manage it. I mean, that's what people are talking about, right? >> Yeah, absolutely. I think, you know, we see kind of three main vectors to reduce cost. I think one is make your deployment process easier overall, so that your engineering effort to even get your app running goes down. Two, would be get more from the compute you're already paying for, you're already paying, you know, for your instances in the cloud, but can you do more with that? And then three would be shop around for lower cost hardware to match your use case. So on the first one, I think making the deployment easier overall, there's a lot of manual work that goes into benchmarking, optimizing and packaging models for deployment. And because the performance of machine learning models can be really hardware dependent, you have to go through this process for each target you want to consider running your model on. And this is hard, you know, we see that every day. But for teams who want to incorporate some of these large language models into their applications, it might be desirable because licensing a model from a large vendor like OpenAI can leave you, you know, over provision, kind of paying for capabilities you don't need in your application or can lock you into them and you lose flexibility. So we have a customer whose team actually prepares models for deployment in a SaaS application that many of us use every day. And they told us recently that without kind of an automated benchmarking and experimentation platform, they were spending several days each to benchmark a single model on a single hardware type. So this is really, you know, manually intensive and then getting more from the compute you're already paying for. We do see customers who leave money on the table by running models that haven't been optimized specifically for the hardware target they're using, like Luis was mentioning. And for some teams they just don't have the time to go through an optimization process and for others they might lack kind of specialized expertise and this is something we can bring. And then on shopping around for different hardware types, we really see a huge variation in model performance across hardware, not just CPU vs. GPU, which is, you know, what people normally think of. But across CPU vendors themselves, high memory instances and across cloud providers even. So the best strategy here is for teams to really be able to, we say, look before you leap by running real world benchmarking and not just simulations or predictions to find the best software, hardware combination for their workload. >> Yeah. You guys sound like you have a very impressive customer base deploying large language models. Where would you categorize your current customer base? And as you look out, as you guys are growing, you have new customers coming in, take me through the progression. Take me through the profile of some of your customers you have now, size, are they hyperscalers, are they big app folks, are they kicking the tires? And then as people are out there scratching heads, I got to get in this game, what's their psychology like? Are they coming in with specific problems or do they have specific orientation point of view about what they want to do? Can you share some data around what you're seeing? >> Yeah, I think, you know, we have customers that kind of range across the spectrum of sophistication from teams that basically don't have MLOps expertise in their company at all. And so they're really looking for us to kind of give a full service, how should I do everything from, you know, optimization, find the hardware, prepare for deployment. And then we have teams that, you know, maybe already have their serving and hosting infrastructure up and ready and they already have models in production and they're really just looking to, you know, take the extra juice out of the hardware and just do really specific on that optimization piece. I think one place where we're doing a lot more work now is kind of in the developer tooling, you know, model selection space. And that's kind of an area that we're creating more tools for, particularly within the PyTorch ecosystem to bring kind of this power earlier in the development cycle so that as people are grabbing a model off the shelf, they can, you know, see how it might perform and use that to inform their development process. >> Luis, what's the big, I like this idea of picking the models because isn't that like going to the market and picking the best model for your data? It's like, you know, it's like, isn't there a certain approaches? What's your view on this? 'Cause this is where everyone, I think it's going to be a land rush for this and I want to get your thoughts. >> For sure, yeah. So, you know, I guess I'll start with saying the one main takeaway that we got from the GPT-J study is that, you know, having a different understanding of what your model's compute and memory requirements are, very quickly, early on helps with the much smarter AI model deployments, right? So, and in fact, you know, Anna just touched on this, but I want to, you know, make sure that it's clear that OctoML is putting that power into user's hands right now. So in partnership with AWS, we are launching this new PyTorch native profiler that allows you with a single, you know, one line, you know, code decorator allows you to see how your code runs on a variety of different hardware after accelerations. So it gives you very clear, you know, data on how you should think about your model deployments. And this ties back to choices of models. So like, if you have a set of choices that are equally good of models in terms of functionality and you want to understand after acceleration how are you going to deploy, how much they're going to cost or what are the options using a automated process of making a decision is really, really useful. And in fact, so I think these events can get early access to this by signing up for the Octopods, you know, this is exclusive group for insiders here, so you can go to OctoML.ai/pods to sign up. >> So that Octopod, is that a program? What is that, is that access to code? Is that a beta, what is that? Explain, take a minute and explain Octopod. >> I think the Octopod would be a group of people who is interested in experiencing this functionality. So it is the friends and users of OctoML that would be the Octopod. And then yes, after you sign up, we would provide you essentially the tool in code form for you to try out in your own. I mean, part of the benefit of this is that it happens in your own local environment and you're in control of everything kind of within the workflow that developers are already using to create and begin putting these models into their applications. So it would all be within your control. >> Got it. I think the big question I have for you is when do you, when does that one of your customers know they need to call you? What's their environment look like? What are they struggling with? What are the conversations they might be having on their side of the fence? If anyone's watching this, they're like, "Hey, you know what, I've got my team, we have a lot of data. Do we have our own language model or do I use someone else's?" There's a lot of this, I will say discovery going on around what to do, what path to take, what does that customer look like, if someone's listening, when do they know to call you guys, OctoML? >> Well, I mean the most obvious one is that you have a significant spend on AI/ML, come and talk to us, you know, putting AIML into production. So that's the clear one. In fact, just this morning I was talking to someone who is in life sciences space and is having, you know, 15 to $20 million a year cloud related to AI/ML deployment is a clear, it's a pretty clear match right there, right? So that's on the cost side. But I also want to emphasize something that Anna said earlier that, you know, the hardware and software complexity involved in putting model into production is really high. So we've been able to abstract that away, offering a clean automation flow enables one, to experiment early on, you know, how models would run and get them to production. And then two, once they are into production, gives you an automated flow to continuously updating your model and taking advantage of all this acceleration and ability to run the model on the right hardware. So anyways, let's say one then is cost, you know, you have significant cost and then two, you have an automation needs. And Anna please compliment that. >> Yeah, Anna you can please- >> Yeah, I think that's exactly right. Maybe the other time is when you are expecting a big scale up in serving your application, right? You're launching a new feature, you expect to get a lot of usage or, and you want to kind of anticipate maybe your CTO, your CIO, whoever pays your cloud bills is going to come after you, right? And so they want to know, you know, what's the return on putting this model essentially into my application stack? Am I going to, is the usage going to match what I'm paying for it? And then you can understand that. >> So you guys have a lot of the early adopters, they got big data teams, they're pushed in the production, they want to get a little QA, test the waters, understand, use your technology to figure it out. Is there any cases where people have gone into production, they have to pull it out? It's like the old lemon laws with your car, you buy a car and oh my god, it's not the way I wanted it. I mean, I can imagine the early people through the wall, so to speak, in the wave here are going to be bloody in the sense that they've gone in and tried stuff and get stuck with huge bills. Are you seeing that? Are people pulling stuff out of production and redeploying? Or I can imagine that if I had a bad deployment, I'd want to refactor that or actually replatform that. Do you see that too? >> Definitely after a sticker shock, yes, your customers will come and make sure that, you know, the sticker shock won't happen again. >> Yeah. >> But then there's another more thorough aspect here that I think we likely touched on, be worth elaborating a bit more is just how are you going to scale in a way that's feasible depending on the allocation that you get, right? So as we mentioned several times here, you know, model deployment is so hardware dependent and so complex that you tend to get a model for a hardware choice and then you want to scale that specific type of instance. But what if, when you want to scale because suddenly luckily got popular and, you know, you want to scale it up and then you don't have that instance anymore. So how do you live with whatever you have at that moment is something that we see customers needing as well. You know, so in fact, ideally what we want is customers to not think about what kind of specific instances they want. What they want is to know what their models need. Say, they know the SLA and then find a set of hybrid targets and instances that hit the SLA whenever they're also scaling, they're going to scale with more freedom, right? Instead of having to wait for AWS to give them more specific allocation for a specific instance. What if you could live with other types of hardware and scale up in a more free way, right? So that's another thing that we see customers, you know, like they need more freedom to be able to scale with whatever is available. >> Anna, you touched on this with the business model impact to that 6 million cost, if that goes out of control, there's a business model aspect and there's a technical operation aspect to the cost side too. You want to be mindful of riding the wave in a good way, but not getting over your skis. So that brings up the point around, you know, confidence, right? And teamwork. Because if you're in production, there's probably a team behind it. Talk about the team aspect of your customers. I mean, they're dedicated, they go put stuff into production, they're developers, there're data. What's in it for them? Are they getting better, are they in the beach, you know, reading the book. Are they, you know, are there easy street for them? What's the customer benefit to the teams? >> Yeah, absolutely. With just a few clicks of a button, you're in production, right? That's the dream. So yeah, I mean I think that, you know, we illustrated it before a little bit. I think the automated kind of benchmarking and optimization process, like when you think about the effort it takes to get that data by hand, which is what people are doing today, they just don't do it. So they're making decisions without the best information because it's, you know, there just isn't the bandwidth to get the information that they need to make the best decision and then know exactly how to deploy it. So I think it's actually bringing kind of a new insight and capability to these teams that they didn't have before. And then maybe another aspect on the team side is that it's making the hand-off of the models from the data science teams to the model deployment teams more seamless. So we have, you know, we have seen in the past that this kind of transition point is the place where there are a lot of hiccups, right? The data science team will give a model to the production team and it'll be too slow for the application or it'll be too expensive to run and it has to go back and be changed and kind of this loop. And so, you know, with the PyTorch profiler that Luis was talking about, and then also, you know, the other ways we do optimization that kind of prevents that hand-off problem from happening. >> Luis and Anna, you guys have a great company. Final couple minutes left. Talk about the company, the people there, what's the culture like, you know, if Intel has Moore's law, which is, you know, doubling the performance in few years, what's the culture like there? Is it, you know, more throughput, better pricing? Explain what's going on with the company and put a plug in. Luis, we'll start with you. >> Yeah, absolutely. I'm extremely proud of the team that we built here. You know, we have a people first culture, you know, very, very collaborative and folks, we all have a shared mission here of making AI more accessible and sustainable. We have a very diverse team in terms of backgrounds and life stories, you know, to do what we do here, we need a team that has expertise in software engineering, in machine learning, in computer architecture. Even though we don't build chips, we need to understand how they work, right? So, and then, you know, the fact that we have this, this very really, really varied set of backgrounds makes the environment, you know, it's say very exciting to learn more about, you know, assistance end-to-end. But also makes it for a very interesting, you know, work environment, right? So people have different backgrounds, different stories. Some of them went to grad school, others, you know, were in intelligence agencies and now are working here, you know. So we have a really interesting set of people and, you know, life is too short not to work with interesting humans. You know, that's something that I like to think about, you know. >> I'm sure your off-site meetings are a lot of fun, people talking about computer architectures, silicon advances, the next GPU, the big data models coming in. Anna, what's your take? What's the culture like? What's the company vibe and what are you guys looking to do? What's the customer success pattern? What's up? >> Yeah, absolutely. I mean, I, you know, second all of the great things that Luis just said about the team. I think one that I, an additional one that I'd really like to underscore is kind of this customer obsession, to use a term you all know well. And focus on the end users and really making the experiences that we're bringing to our user who are developers really, you know, useful and valuable for them. And so I think, you know, all of these tools that we're trying to put in the hands of users, the industry and the market is changing so rapidly that our products across the board, you know, all of the companies that, you know, are part of the showcase today, we're all evolving them so quickly and we can only do that kind of really hand in glove with our users. So that would be another thing I'd emphasize. >> I think the change dynamic, the power dynamics of this industry is just the beginning. I'm very bullish that this is going to be probably one of the biggest inflection points in history of the computer industry because of all the dynamics of the confluence of all the forces, which you mentioned some of them, I mean PC, you know, interoperability within internetworking and you got, you know, the web and then mobile. Now we have this, I mean, I wouldn't even put social media even in the close to this. Like, this is like, changes user experience, changes infrastructure. There's going to be massive accelerations in performance on the hardware side from AWS's of the world and cloud and you got the edge and more data. This is really what big data was going to look like. This is the beginning. Final question, what do you guys see going forward in the future? >> Well, it's undeniable that machine learning and AI models are becoming an integral part of an interesting application today, right? So, and the clear trends here are, you know, more and more competitional needs for these models because they're only getting more and more powerful. And then two, you know, seeing the complexity of the infrastructure where they run, you know, just considering the cloud, there's like a wide variety of choices there, right? So being able to live with that and making the most out of it in a way that does not require, you know, an impossible to find team is something that's pretty clear. So the need for automation, abstracting with the complexity is definitely here. And we are seeing this, you know, trends are that you also see models starting to move to the edge as well. So it's clear that we're seeing, we are going to live in a world where there's no large models living in the cloud. And then, you know, edge models that talk to these models in the cloud to form, you know, an end-to-end truly intelligent application. >> Anna? >> Yeah, I think, you know, our, Luis said it at the beginning. Our vision is to make AI sustainable and accessible. And I think as this technology just expands in every company and every team, that's going to happen kind of on its own. And we're here to help support that. And I think you can't do that without tools like those like OctoML. >> I think it's going to be an error of massive invention, creativity, a lot of the format heavy lifting is going to allow the talented people to automate their intellect. I mean, this is really kind of what we see going on. And Luis, thank you so much. Anna, thanks for coming on this segment. Thanks for coming on theCUBE and being part of the AWS Startup Showcase. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(upbeat music) >> Hello everyone. Welcome to theCube's presentation of the "AWS Startup Showcase." The topic this episode is AI and machine learning, top startups building foundational model infrastructure. This is season three, episode one of the ongoing series covering exciting startups from the AWS ecosystem. And this time we're talking about AI and machine learning. I'm your host, John Furrier. I'm excited I'm joined today by Robert Nishihara, who's the co-founder and CEO of a hot startup called Anyscale. He's here to talk about Ray, the open source project, Anyscale's infrastructure for foundation as well. Robert, thank you for joining us today. >> Yeah, thanks so much as well. >> I've been following your company since the founding pre pandemic and you guys really had a great vision scaled up and in a perfect position for this big wave that we all see with ChatGPT and OpenAI that's gone mainstream. Finally, AI has broken out through the ropes and now gone mainstream, so I think you guys are really well positioned. I'm looking forward to to talking with you today. But before we get into it, introduce the core mission for Anyscale. Why do you guys exist? What is the North Star for Anyscale? >> Yeah, like you mentioned, there's a tremendous amount of excitement about AI right now. You know, I think a lot of us believe that AI can transform just every different industry. So one of the things that was clear to us when we started this company was that the amount of compute needed to do AI was just exploding. Like to actually succeed with AI, companies like OpenAI or Google or you know, these companies getting a lot of value from AI, were not just running these machine learning models on their laptops or on a single machine. They were scaling these applications across hundreds or thousands or more machines and GPUs and other resources in the Cloud. And so to actually succeed with AI, and this has been one of the biggest trends in computing, maybe the biggest trend in computing in, you know, in recent history, the amount of compute has been exploding. And so to actually succeed with that AI, to actually build these scalable applications and scale the AI applications, there's a tremendous software engineering lift to build the infrastructure to actually run these scalable applications. And that's very hard to do. So one of the reasons many AI projects and initiatives fail is that, or don't make it to production, is the need for this scale, the infrastructure lift, to actually make it happen. So our goal here with Anyscale and Ray, is to make that easy, is to make scalable computing easy. So that as a developer or as a business, if you want to do AI, if you want to get value out of AI, all you need to know is how to program on your laptop. Like, all you need to know is how to program in Python. And if you can do that, then you're good to go. Then you can do what companies like OpenAI or Google do and get value out of machine learning. >> That programming example of how easy it is with Python reminds me of the early days of Cloud, when infrastructure as code was talked about was, it was just code the infrastructure programmable. That's super important. That's what AI people wanted, first program AI. That's the new trend. And I want to understand, if you don't mind explaining, the relationship that Anyscale has to these foundational models and particular the large language models, also called LLMs, was seen with like OpenAI and ChatGPT. Before you get into the relationship that you have with them, can you explain why the hype around foundational models? Why are people going crazy over foundational models? What is it and why is it so important? >> Yeah, so foundational models and foundation models are incredibly important because they enable businesses and developers to get value out of machine learning, to use machine learning off the shelf with these large models that have been trained on tons of data and that are useful out of the box. And then, of course, you know, as a business or as a developer, you can take those foundational models and repurpose them or fine tune them or adapt them to your specific use case and what you want to achieve. But it's much easier to do that than to train them from scratch. And I think there are three, for people to actually use foundation models, there are three main types of workloads or problems that need to be solved. One is training these foundation models in the first place, like actually creating them. The second is fine tuning them and adapting them to your use case. And the third is serving them and actually deploying them. Okay, so Ray and Anyscale are used for all of these three different workloads. Companies like OpenAI or Cohere that train large language models. Or open source versions like GPTJ are done on top of Ray. There are many startups and other businesses that fine tune, that, you know, don't want to train the large underlying foundation models, but that do want to fine tune them, do want to adapt them to their purposes, and build products around them and serve them, those are also using Ray and Anyscale for that fine tuning and that serving. And so the reason that Ray and Anyscale are important here is that, you know, building and using foundation models requires a huge scale. It requires a lot of data. It requires a lot of compute, GPUs, TPUs, other resources. And to actually take advantage of that and actually build these scalable applications, there's a lot of infrastructure that needs to happen under the hood. And so you can either use Ray and Anyscale to take care of that and manage the infrastructure and solve those infrastructure problems. Or you can build the infrastructure and manage the infrastructure yourself, which you can do, but it's going to slow your team down. It's going to, you know, many of the businesses we work with simply don't want to be in the business of managing infrastructure and building infrastructure. They want to focus on product development and move faster. >> I know you got a keynote presentation we're going to go to in a second, but I think you hit on something I think is the real tipping point, doing it yourself, hard to do. These are things where opportunities are and the Cloud did that with data centers. Turned a data center and made it an API. The heavy lifting went away and went to the Cloud so people could be more creative and build their product. In this case, build their creativity. Is that kind of what's the big deal? Is that kind of a big deal happening that you guys are taking the learnings and making that available so people don't have to do that? >> That's exactly right. So today, if you want to succeed with AI, if you want to use AI in your business, infrastructure work is on the critical path for doing that. To do AI, you have to build infrastructure. You have to figure out how to scale your applications. That's going to change. We're going to get to the point, and you know, with Ray and Anyscale, we're going to remove the infrastructure from the critical path so that as a developer or as a business, all you need to focus on is your application logic, what you want the the program to do, what you want your application to do, how you want the AI to actually interface with the rest of your product. Now the way that will happen is that Ray and Anyscale will still, the infrastructure work will still happen. It'll just be under the hood and taken care of by Ray in Anyscale. And so I think something like this is really necessary for AI to reach its potential, for AI to have the impact and the reach that we think it will, you have to make it easier to do. >> And just for clarification to point out, if you don't mind explaining the relationship of Ray and Anyscale real quick just before we get into the presentation. >> So Ray is an open source project. We created it. We were at Berkeley doing machine learning. We started Ray so that, in order to provide an easy, a simple open source tool for building and running scalable applications. And Anyscale is the managed version of Ray, basically we will run Ray for you in the Cloud, provide a lot of tools around the developer experience and managing the infrastructure and providing more performance and superior infrastructure. >> Awesome. I know you got a presentation on Ray and Anyscale and you guys are positioning as the infrastructure for foundational models. So I'll let you take it away and then when you're done presenting, we'll come back, I'll probably grill you with a few questions and then we'll close it out so take it away. >> Robert: Sounds great. So I'll say a little bit about how companies are using Ray and Anyscale for foundation models. The first thing I want to mention is just why we're doing this in the first place. And the underlying observation, the underlying trend here, and this is a plot from OpenAI, is that the amount of compute needed to do machine learning has been exploding. It's been growing at something like 35 times every 18 months. This is absolutely enormous. And other people have written papers measuring this trend and you get different numbers. But the point is, no matter how you slice and dice it, it' a astronomical rate. Now if you compare that to something we're all familiar with, like Moore's Law, which says that, you know, the processor performance doubles every roughly 18 months, you can see that there's just a tremendous gap between the needs, the compute needs of machine learning applications, and what you can do with a single chip, right. So even if Moore's Law were continuing strong and you know, doing what it used to be doing, even if that were the case, there would still be a tremendous gap between what you can do with the chip and what you need in order to do machine learning. And so given this graph, what we've seen, and what has been clear to us since we started this company, is that doing AI requires scaling. There's no way around it. It's not a nice to have, it's really a requirement. And so that led us to start Ray, which is the open source project that we started to make it easy to build these scalable Python applications and scalable machine learning applications. And since we started the project, it's been adopted by a tremendous number of companies. Companies like OpenAI, which use Ray to train their large models like ChatGPT, companies like Uber, which run all of their deep learning and classical machine learning on top of Ray, companies like Shopify or Spotify or Instacart or Lyft or Netflix, ByteDance, which use Ray for their machine learning infrastructure. Companies like Ant Group, which makes Alipay, you know, they use Ray across the board for fraud detection, for online learning, for detecting money laundering, you know, for graph processing, stream processing. Companies like Amazon, you know, run Ray at a tremendous scale and just petabytes of data every single day. And so the project has seen just enormous adoption since, over the past few years. And one of the most exciting use cases is really providing the infrastructure for building training, fine tuning, and serving foundation models. So I'll say a little bit about, you know, here are some examples of companies using Ray for foundation models. Cohere trains large language models. OpenAI also trains large language models. You can think about the workloads required there are things like supervised pre-training, also reinforcement learning from human feedback. So this is not only the regular supervised learning, but actually more complex reinforcement learning workloads that take human input about what response to a particular question, you know is better than a certain other response. And incorporating that into the learning. There's open source versions as well, like GPTJ also built on top of Ray as well as projects like Alpa coming out of UC Berkeley. So these are some of the examples of exciting projects in organizations, training and creating these large language models and serving them using Ray. Okay, so what actually is Ray? Well, there are two layers to Ray. At the lowest level, there's the core Ray system. This is essentially low level primitives for building scalable Python applications. Things like taking a Python function or a Python class and executing them in the cluster setting. So Ray core is extremely flexible and you can build arbitrary scalable applications on top of Ray. So on top of Ray, on top of the core system, what really gives Ray a lot of its power is this ecosystem of scalable libraries. So on top of the core system you have libraries, scalable libraries for ingesting and pre-processing data, for training your models, for fine tuning those models, for hyper parameter tuning, for doing batch processing and batch inference, for doing model serving and deployment, right. And a lot of the Ray users, the reason they like Ray is that they want to run multiple workloads. They want to train and serve their models, right. They want to load their data and feed that into training. And Ray provides common infrastructure for all of these different workloads. So this is a little overview of what Ray, the different components of Ray. So why do people choose to go with Ray? I think there are three main reasons. The first is the unified nature. The fact that it is common infrastructure for scaling arbitrary workloads, from data ingest to pre-processing to training to inference and serving, right. This also includes the fact that it's future proof. AI is incredibly fast moving. And so many people, many companies that have built their own machine learning infrastructure and standardized on particular workflows for doing machine learning have found that their workflows are too rigid to enable new capabilities. If they want to do reinforcement learning, if they want to use graph neural networks, they don't have a way of doing that with their standard tooling. And so Ray, being future proof and being flexible and general gives them that ability. Another reason people choose Ray in Anyscale is the scalability. This is really our bread and butter. This is the reason, the whole point of Ray, you know, making it easy to go from your laptop to running on thousands of GPUs, making it easy to scale your development workloads and run them in production, making it easy to scale, you know, training to scale data ingest, pre-processing and so on. So scalability and performance, you know, are critical for doing machine learning and that is something that Ray provides out of the box. And lastly, Ray is an open ecosystem. You can run it anywhere. You can run it on any Cloud provider. Google, you know, Google Cloud, AWS, Asure. You can run it on your Kubernetes cluster. You can run it on your laptop. It's extremely portable. And not only that, it's framework agnostic. You can use Ray to scale arbitrary Python workloads. You can use it to scale and it integrates with libraries like TensorFlow or PyTorch or JAX or XG Boost or Hugging Face or PyTorch Lightning, right, or Scikit-learn or just your own arbitrary Python code. It's open source. And in addition to integrating with the rest of the machine learning ecosystem and these machine learning frameworks, you can use Ray along with all of the other tooling in the machine learning ecosystem. That's things like weights and biases or ML flow, right. Or you know, different data platforms like Databricks, you know, Delta Lake or Snowflake or tools for model monitoring for feature stores, all of these integrate with Ray. And that's, you know, Ray provides that kind of flexibility so that you can integrate it into the rest of your workflow. And then Anyscale is the scalable compute platform that's built on top, you know, that provides Ray. So Anyscale is a managed Ray service that runs in the Cloud. And what Anyscale does is it offers the best way to run Ray. And if you think about what you get with Anyscale, there are fundamentally two things. One is about moving faster, accelerating the time to market. And you get that by having the managed service so that as a developer you don't have to worry about managing infrastructure, you don't have to worry about configuring infrastructure. You also, it provides, you know, optimized developer workflows. Things like easily moving from development to production, things like having the observability tooling, the debug ability to actually easily diagnose what's going wrong in a distributed application. So things like the dashboards and the other other kinds of tooling for collaboration, for monitoring and so on. And then on top of that, so that's the first bucket, developer productivity, moving faster, faster experimentation and iteration. The second reason that people choose Anyscale is superior infrastructure. So this is things like, you know, cost deficiency, being able to easily take advantage of spot instances, being able to get higher GPU utilization, things like faster cluster startup times and auto scaling. Things like just overall better performance and faster scheduling. And so these are the kinds of things that Anyscale provides on top of Ray. It's the managed infrastructure. It's fast, it's like the developer productivity and velocity as well as performance. So this is what I wanted to share about Ray in Anyscale. >> John: Awesome. >> Provide that context. But John, I'm curious what you think. >> I love it. I love the, so first of all, it's a platform because that's the platform architecture right there. So just to clarify, this is an Anyscale platform, not- >> That's right. >> Tools. So you got tools in the platform. Okay, that's key. Love that managed service. Just curious, you mentioned Python multiple times, is that because of PyTorch and TensorFlow or Python's the most friendly with machine learning or it's because it's very common amongst all developers? >> That's a great question. Python is the language that people are using to do machine learning. So it's the natural starting point. Now, of course, Ray is actually designed in a language agnostic way and there are companies out there that use Ray to build scalable Java applications. But for the most part right now we're focused on Python and being the best way to build these scalable Python and machine learning applications. But, of course, down the road there always is that potential. >> So if you're slinging Python code out there and you're watching that, you're watching this video, get on Anyscale bus quickly. Also, I just, while you were giving the presentation, I couldn't help, since you mentioned OpenAI, which by the way, congratulations 'cause they've had great scale, I've noticed in their rapid growth 'cause they were the fastest company to the number of users than anyone in the history of the computer industry, so major successor, OpenAI and ChatGPT, huge fan. I'm not a skeptic at all. I think it's just the beginning, so congratulations. But I actually typed into ChatGPT, what are the top three benefits of Anyscale and came up with scalability, flexibility, and ease of use. Obviously, scalability is what you guys are called. >> That's pretty good. >> So that's what they came up with. So they nailed it. Did you have an inside prompt training, buy it there? Only kidding. (Robert laughs) >> Yeah, we hard coded that one. >> But that's the kind of thing that came up really, really quickly if I asked it to write a sales document, it probably will, but this is the future interface. This is why people are getting excited about the foundational models and the large language models because it's allowing the interface with the user, the consumer, to be more human, more natural. And this is clearly will be in every application in the future. >> Absolutely. This is how people are going to interface with software, how they're going to interface with products in the future. It's not just something, you know, not just a chat bot that you talk to. This is going to be how you get things done, right. How you use your web browser or how you use, you know, how you use Photoshop or how you use other products. Like you're not going to spend hours learning all the APIs and how to use them. You're going to talk to it and tell it what you want it to do. And of course, you know, if it doesn't understand it, it's going to ask clarifying questions. You're going to have a conversation and then it'll figure it out. >> This is going to be one of those things, we're going to look back at this time Robert and saying, "Yeah, from that company, that was the beginning of that wave." And just like AWS and Cloud Computing, the folks who got in early really were in position when say the pandemic came. So getting in early is a good thing and that's what everyone's talking about is getting in early and playing around, maybe replatforming or even picking one or few apps to refactor with some staff and managed services. So people are definitely jumping in. So I have to ask you the ROI cost question. You mentioned some of those, Moore's Law versus what's going on in the industry. When you look at that kind of scale, the first thing that jumps out at people is, "Okay, I love it. Let's go play around." But what's it going to cost me? Am I going to be tied to certain GPUs? What's the landscape look like from an operational standpoint, from the customer? Are they locked in and the benefit was flexibility, are you flexible to handle any Cloud? What is the customers, what are they looking at? Basically, that's my question. What's the customer looking at? >> Cost is super important here and many of the companies, I mean, companies are spending a huge amount on their Cloud computing, on AWS, and on doing AI, right. And I think a lot of the advantage of Anyscale, what we can provide here is not only better performance, but cost efficiency. Because if we can run something faster and more efficiently, it can also use less resources and you can lower your Cloud spending, right. We've seen companies go from, you know, 20% GPU utilization with their current setup and the current tools they're using to running on Anyscale and getting more like 95, you know, 100% GPU utilization. That's something like a five x improvement right there. So depending on the kind of application you're running, you know, it's a significant cost savings. We've seen companies that have, you know, processing petabytes of data every single day with Ray going from, you know, getting order of magnitude cost savings by switching from what they were previously doing to running their application on Ray. And when you have applications that are spending, you know, potentially $100 million a year and getting a 10 X cost savings is just absolutely enormous. So these are some of the kinds of- >> Data infrastructure is super important. Again, if the customer, if you're a prospect to this and thinking about going in here, just like the Cloud, you got infrastructure, you got the platform, you got SaaS, same kind of thing's going to go on in AI. So I want to get into that, you know, ROI discussion and some of the impact with your customers that are leveraging the platform. But first I hear you got a demo. >> Robert: Yeah, so let me show you, let me give you a quick run through here. So what I have open here is the Anyscale UI. I've started a little Anyscale Workspace. So Workspaces are the Anyscale concept for interactive developments, right. So here, imagine I'm just, you want to have a familiar experience like you're developing on your laptop. And here I have a terminal. It's not on my laptop. It's actually in the cloud running on Anyscale. And I'm just going to kick this off. This is going to train a large language model, so OPT. And it's doing this on 32 GPUs. We've got a cluster here with a bunch of CPU cores, bunch of memory. And as that's running, and by the way, if I wanted to run this on instead of 32 GPUs, 64, 128, this is just a one line change when I launch the Workspace. And what I can do is I can pull up VS code, right. Remember this is the interactive development experience. I can look at the actual code. Here it's using Ray train to train the torch model. We've got the training loop and we're saying that each worker gets access to one GPU and four CPU cores. And, of course, as I make the model larger, this is using deep speed, as I make the model larger, I could increase the number of GPUs that each worker gets access to, right. And how that is distributed across the cluster. And if I wanted to run on CPUs instead of GPUs or a different, you know, accelerator type, again, this is just a one line change. And here we're using Ray train to train the models, just taking my vanilla PyTorch model using Hugging Face and then scaling that across a bunch of GPUs. And, of course, if I want to look at the dashboard, I can go to the Ray dashboard. There are a bunch of different visualizations I can look at. I can look at the GPU utilization. I can look at, you know, the CPU utilization here where I think we're currently loading the model and running that actual application to start the training. And some of the things that are really convenient here about Anyscale, both I can get that interactive development experience with VS code. You know, I can look at the dashboards. I can monitor what's going on. It feels, I have a terminal, it feels like my laptop, but it's actually running on a large cluster. And I can, with however many GPUs or other resources that I want. And so it's really trying to combine the best of having the familiar experience of programming on your laptop, but with the benefits, you know, being able to take advantage of all the resources in the Cloud to scale. And it's like when, you know, you're talking about cost efficiency. One of the biggest reasons that people waste money, one of the silly reasons for wasting money is just forgetting to turn off your GPUs. And what you can do here is, of course, things will auto terminate if they're idle. But imagine you go to sleep, I have this big cluster. You can turn it off, shut off the cluster, come back tomorrow, restart the Workspace, and you know, your big cluster is back up and all of your code changes are still there. All of your local file edits. It's like you just closed your laptop and came back and opened it up again. And so this is the kind of experience we want to provide for our users. So that's what I wanted to share with you. >> Well, I think that whole, couple of things, lines of code change, single line of code change, that's game changing. And then the cost thing, I mean human error is a big deal. People pass out at their computer. They've been coding all night or they just forget about it. I mean, and then it's just like leaving the lights on or your water running in your house. It's just, at the scale that it is, the numbers will add up. That's a huge deal. So I think, you know, compute back in the old days, there's no compute. Okay, it's just compute sitting there idle. But you know, data cranking the models is doing, that's a big point. >> Another thing I want to add there about cost efficiency is that we make it really easy to use, if you're running on Anyscale, to use spot instances and these preemptable instances that can just be significantly cheaper than the on-demand instances. And so when we see our customers go from what they're doing before to using Anyscale and they go from not using these spot instances 'cause they don't have the infrastructure around it, the fault tolerance to handle the preemption and things like that, to being able to just check a box and use spot instances and save a bunch of money. >> You know, this was my whole, my feature article at Reinvent last year when I met with Adam Selipsky, this next gen Cloud is here. I mean, it's not auto scale, it's infrastructure scale. It's agility. It's flexibility. I think this is where the world needs to go. Almost what DevOps did for Cloud and what you were showing me that demo had this whole SRE vibe. And remember Google had site reliability engines to manage all those servers. This is kind of like an SRE vibe for data at scale. I mean, a similar kind of order of magnitude. I mean, I might be a little bit off base there, but how would you explain it? >> It's a nice analogy. I mean, what we are trying to do here is get to the point where developers don't think about infrastructure. Where developers only think about their application logic. And where businesses can do AI, can succeed with AI, and build these scalable applications, but they don't have to build, you know, an infrastructure team. They don't have to develop that expertise. They don't have to invest years in building their internal machine learning infrastructure. They can just focus on the Python code, on their application logic, and run the stuff out of the box. >> Awesome. Well, I appreciate the time. Before we wrap up here, give a plug for the company. I know you got a couple websites. Again, go, Ray's got its own website. You got Anyscale. You got an event coming up. Give a plug for the company looking to hire. Put a plug in for the company. >> Yeah, absolutely. Thank you. So first of all, you know, we think AI is really going to transform every industry and the opportunity is there, right. We can be the infrastructure that enables all of that to happen, that makes it easy for companies to succeed with AI, and get value out of AI. Now we have, if you're interested in learning more about Ray, Ray has been emerging as the standard way to build scalable applications. Our adoption has been exploding. I mentioned companies like OpenAI using Ray to train their models. But really across the board companies like Netflix and Cruise and Instacart and Lyft and Uber, you know, just among tech companies. It's across every industry. You know, gaming companies, agriculture, you know, farming, robotics, drug discovery, you know, FinTech, we see it across the board. And all of these companies can get value out of AI, can really use AI to improve their businesses. So if you're interested in learning more about Ray and Anyscale, we have our Ray Summit coming up in September. This is going to highlight a lot of the most impressive use cases and stories across the industry. And if your business, if you want to use LLMs, you want to train these LLMs, these large language models, you want to fine tune them with your data, you want to deploy them, serve them, and build applications and products around them, give us a call, talk to us. You know, we can really take the infrastructure piece, you know, off the critical path and make that easy for you. So that's what I would say. And, you know, like you mentioned, we're hiring across the board, you know, engineering, product, go-to-market, and it's an exciting time. >> Robert Nishihara, co-founder and CEO of Anyscale, congratulations on a great company you've built and continuing to iterate on and you got growth ahead of you, you got a tailwind. I mean, the AI wave is here. I think OpenAI and ChatGPT, a customer of yours, have really opened up the mainstream visibility into this new generation of applications, user interface, roll of data, large scale, how to make that programmable so we're going to need that infrastructure. So thanks for coming on this season three, episode one of the ongoing series of the hot startups. In this case, this episode is the top startups building foundational model infrastructure for AI and ML. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(bright upbeat music) >> Hello, everyone. Welcome to SiliconANGLE News breaking story here. Amazon Web Services expanding their relationship with Hugging Face, breaking news here on SiliconANGLE. I'm John Furrier, SiliconANGLE reporter, founder, and also co-host of theCUBE. And I have with me, Swami, from Amazon Web Services, vice president of database, analytics, machine learning with AWS. Swami, great to have you on for this breaking news segment on AWS's big news. Thanks for coming on and taking the time. >> Hey, John, pleasure to be here. >> You know- >> Looking forward to it. >> We've had many conversations on theCUBE over the years, we've watched Amazon really move fast into the large data modeling, SageMaker became a very smashing success, obviously you've been on this for a while. Now with ChatGPT OpenAI, a lot of buzz going mainstream, takes it from behind the curtain inside the ropes, if you will, in the industry to a mainstream. And so this is a big moment, I think, in the industry, I want to get your perspective, because your news with Hugging Face, I think is another tell sign that we're about to tip over into a new accelerated growth around making AI now application aware, application centric, more programmable, more API access. What's the big news about, with AWS Hugging Face, you know, what's going on with this announcement? >> Yeah. First of all, they're very excited to announce our expanded collaboration with Hugging Face, because with this partnership, our goal, as you all know, I mean, Hugging Face, I consider them like the GitHub for machine learning. And with this partnership, Hugging Face and AWS, we'll be able to democratize AI for a broad range of developers, not just specific deep AI startups. And now with this, we can accelerate the training, fine tuning and deployment of these large language models, and vision models from Hugging Face in the cloud. And the broader context, when you step back and see what customer problem we are trying to solve with this announcement, essentially if you see these foundational models, are used to now create like a huge number of applications, suggest like tech summarization, question answering, or search image generation, creative, other things. And these are all stuff we are seeing in the likes of these ChatGPT style applications. But there is a broad range of enterprise use cases that we don't even talk about. And it's because these kind of transformative, generative AI capabilities and models are not available to, I mean, millions of developers. And because either training these elements from scratch can be very expensive or time consuming and need deep expertise, or more importantly, they don't need these generic models, they need them to be fine tuned for the specific use cases. And one of the biggest complaints we hear is that these models, when they try to use it for real production use cases, they are incredibly expensive to train and incredibly expensive to run inference on, to use it at a production scale. So, and unlike web search style applications, where the margins can be really huge, here in production use cases and enterprises, you want efficiency at scale. That's where Hugging Face and AWS share our mission. And by integrating with Trainium and Inferentia, we're able to handle the cost efficient training and inference at scale, I'll deep dive on it. And by teaming up on the SageMaker front, now the time it takes to build these models and fine tune them is also coming down. So that's what makes this partnership very unique as well. So I'm very excited. >> I want to get into the time savings and the cost savings as well on the training and inference, it's a huge issue, but before we get into that, just how long have you guys been working with Hugging Face? I know there's a previous relationship, this is an expansion of that relationship, can you comment on what's different about what's happened before and then now? >> Yeah. So, Hugging Face, we have had a great relationship in the past few years as well, where they have actually made their models available to run on AWS, you know, fashion. Even in fact, their Bloom Project was something many of our customers even used. Bloom Project, for context, is their open source project which builds a GPT-3 style model. And now with this expanded collaboration, now Hugging Face selected AWS for that next generation office generative AI model, building on their highly successful Bloom Project as well. And the nice thing is, now, by direct integration with Trainium and Inferentia, where you get cost savings in a really significant way, now, for instance, Trn1 can provide up to 50% cost to train savings, and Inferentia can deliver up to 60% better costs, and four x more higher throughput than (indistinct). Now, these models, especially as they train that next generation generative AI models, it is going to be, not only more accessible to all the developers, who use it in open, so it'll be a lot cheaper as well. And that's what makes this moment really exciting, because we can't democratize AI unless we make it broadly accessible and cost efficient and easy to program and use as well. >> Yeah. >> So very exciting. >> I'll get into the SageMaker and CodeWhisperer angle in a second, but you hit on some good points there. One, accessibility, which is, I call the democratization, which is getting this in the hands of developers, and/or AI to develop, we'll get into that in a second. So, access to coding and Git reasoning is a whole nother wave. But the three things I know you've been working on, I want to put in the buckets here and comment, one, I know you've, over the years, been working on saving time to train, that's a big point, you mentioned some of those stats, also cost, 'cause now cost is an equation on, you know, bundling whether you're uncoupling with hardware and software, that's a big issue. Where do I find the GPUs? Where's the horsepower cost? And then also sustainability. You've mentioned that in the past, is there a sustainability angle here? Can you talk about those three things, time, cost, and sustainability? >> Certainly. So if you look at it from the AWS perspective, we have been supporting customers doing machine learning for the past years. Just for broader context, Amazon has been doing ML the past two decades right from the early days of ML powered recommendation to actually also supporting all kinds of generative AI applications. If you look at even generative AI application within Amazon, Amazon search, when you go search for a product and so forth, we have a team called MFi within Amazon search that helps bring these large language models into creating highly accurate search results. And these are created with models, really large models with tens of billions of parameters, scales to thousands of training jobs every month and trained on large model of hardware. And this is an example of a really good large language foundation model application running at production scale, and also, of course, Alexa, which uses a large generator model as well. And they actually even had a research paper that showed that they are more, and do better in accuracy than other systems like GPT-3 and whatnot. So, and we also touched on things like CodeWhisperer, which uses generative AI to improve developer productivity, but in a responsible manner, because 40% of some of the studies show 40% of this generated code had serious security flaws in it. This is where we didn't just do generative AI, we combined with automated reasoning capabilities, which is a very, very useful technique to identify these issues and couple them so that it produces highly secure code as well. Now, all these learnings taught us few things, and which is what you put in these three buckets. And yeah, like more than 100,000 customers using ML and AI services, including leading startups in the generative AI space, like stability AI, AI21 Labs, or Hugging Face, or even Alexa, for that matter. They care about, I put them in three dimension, one is around cost, which we touched on with Trainium and Inferentia, where we actually, the Trainium, you provide to 50% better cost savings, but the other aspect is, Trainium is a lot more power efficient as well compared to traditional one. And Inferentia is also better in terms of throughput, when it comes to what it is capable of. Like it is able to deliver up to three x higher compute performance and four x higher throughput, compared to it's previous generation, and it is extremely cost efficient and power efficient as well. >> Well. >> Now, the second element that really is important is in a day, developers deeply value the time it takes to build these models, and they don't want to build models from scratch. And this is where SageMaker, which is, even going to Kaggle uses, this is what it is, number one, enterprise ML platform. What it did to traditional machine learning, where tens of thousands of customers use StageMaker today, including the ones I mentioned, is that what used to take like months to build these models have dropped down to now a matter of days, if not less. Now, a generative AI, the cost of building these models, if you look at the landscape, the model parameter size had jumped by more than thousand X in the past three years, thousand x. And that means the training is like a really big distributed systems problem. How do you actually scale these model training? How do you actually ensure that you utilize these efficiently? Because these machines are very expensive, let alone they consume a lot of power. So, this is where SageMaker capability to build, automatically train, tune, and deploy models really concern this, especially with this distributor training infrastructure, and those are some of the reasons why some of the leading generative AI startups are actually leveraging it, because they do not want a giant infrastructure team, which is constantly tuning and fine tuning, and keeping these clusters alive. >> It sounds like a lot like what startups are doing with the cloud early days, no data center, you move to the cloud. So, this is the trend we're seeing, right? You guys are making it easier for developers with Hugging Face, I get that. I love that GitHub for machine learning, large language models are complex and expensive to build, but not anymore, you got Trainium and Inferentia, developers can get faster time to value, but then you got the transformers data sets, token libraries, all that optimized for generator. This is a perfect storm for startups. Jon Turow, a former AWS person, who used to work, I think for you, is now a VC at Madrona Venture, he and I were talking about the generator AI landscape, it's exploding with startups. Every alpha entrepreneur out there is seeing this as the next frontier, that's the 20 mile stairs, next 10 years is going to be huge. What is the big thing that's happened? 'Cause some people were saying, the founder of Yquem said, "Oh, the start ups won't be real, because they don't all have AI experience." John Markoff, former New York Times writer told me that, AI, there's so much work done, this is going to explode, accelerate really fast, because it's almost like it's been waiting for this moment. What's your reaction? >> I actually think there is going to be an explosion of startups, not because they need to be AI startups, but now finally AI is really accessible or going to be accessible, so that they can create remarkable applications, either for enterprises or for disrupting actually how customer service is being done or how creative tools are being built. And I mean, this is going to change in many ways. When we think about generative AI, we always like to think of how it generates like school homework or arts or music or whatnot, but when you look at it on the practical side, generative AI is being actually used across various industries. I'll give an example of like Autodesk. Autodesk is a customer who runs an AWS and SageMaker. They already have an offering that enables generated design, where designers can generate many structural designs for products, whereby you give a specific set of constraints and they actually can generate a structure accordingly. And we see similar kind of trend across various industries, where it can be around creative media editing or various others. I have the strong sense that literally, in the next few years, just like now, conventional machine learning is embedded in every application, every mobile app that we see, it is pervasive, and we don't even think twice about it, same way, like almost all apps are built on cloud. Generative AI is going to be part of every startup, and they are going to create remarkable experiences without needing actually, these deep generative AI scientists. But you won't get that until you actually make these models accessible. And I also don't think one model is going to rule the world, then you want these developers to have access to broad range of models. Just like, go back to the early days of deep learning. Everybody thought it is going to be one framework that will rule the world, and it has been changing, from Caffe to TensorFlow to PyTorch to various other things. And I have a suspicion, we had to enable developers where they are, so. >> You know, Dave Vellante and I have been riffing on this concept called super cloud, and a lot of people have co-opted to be multicloud, but we really were getting at this whole next layer on top of say, AWS. You guys are the most comprehensive cloud, you guys are a super cloud, and even Adam and I are talking about ISVs evolving to ecosystem partners. I mean, your top customers have ecosystems building on top of it. This feels like a whole nother AWS. How are you guys leveraging the history of AWS, which by the way, had the same trajectory, startups came in, they didn't want to provision a data center, the heavy lifting, all the things that have made Amazon successful culturally. And day one thinking is, provide the heavy lifting, undifferentiated heavy lifting, and make it faster for developers to program code. AI's got the same thing. How are you guys taking this to the next level, because now, this is an opportunity for the competition to change the game and take it over? This is, I'm sure, a conversation, you guys have a lot of things going on in AWS that makes you unique. What's the internal and external positioning around how you take it to the next level? >> I mean, so I agree with you that generative AI has a very, very strong potential in terms of what it can enable in terms of next generation application. But this is where Amazon's experience and expertise in putting these foundation models to work internally really has helped us quite a bit. If you look at it, like amazon.com search is like a very, very important application in terms of what is the customer impact on number of customers who use that application openly, and the amount of dollar impact it does for an organization. And we have been doing it silently for a while now. And the same thing is true for like Alexa too, which actually not only uses it for natural language understanding other city, even national leverages is set for creating stories and various other examples. And now, our approach to it from AWS is we actually look at it as in terms of the same three tiers like we did in machine learning, because when you look at generative AI, we genuinely see three sets of customers. One is, like really deep technical expert practitioner startups. These are the startups that are creating the next generation models like the likes of stability AIs or Hugging Face with Bloom or AI21. And they generally want to build their own models, and they want the best price performance of their infrastructure for training and inference. That's where our investments in silicon and hardware and networking innovations, where Trainium and Inferentia really plays a big role. And we can nearly do that, and that is one. The second middle tier is where I do think developers don't want to spend time building their own models, let alone, they actually want the model to be useful to that data. They don't need their models to create like high school homeworks or various other things. What they generally want is, hey, I had this data from my enterprises that I want to fine tune and make it really work only for this, and make it work remarkable, can be for tech summarization, to generate a report, or it can be for better Q&A, and so forth. This is where we are. Our investments in the middle tier with SageMaker, and our partnership with Hugging Face and AI21 and co here are all going to very meaningful. And you'll see us investing, I mean, you already talked about CodeWhisperer, which is an open preview, but we are also partnering with a whole lot of top ISVs, and you'll see more on this front to enable the next wave of generated AI apps too, because this is an area where we do think lot of innovation is yet to be done. It's like day one for us in this space, and we want to enable that huge ecosystem to flourish. >> You know, one of the things Dave Vellante and I were talking about in our first podcast we just did on Friday, we're going to do weekly, is we highlighted the AI ChatGPT example as a horizontal use case, because everyone loves it, people are using it in all their different verticals, and horizontal scalable cloud plays perfectly into it. So I have to ask you, as you look at what AWS is going to bring to the table, a lot's changed over the past 13 years with AWS, a lot more services are available, how should someone rebuild or re-platform and refactor their application of business with AI, with AWS? What are some of the tools that you see and recommend? Is it Serverless, is it SageMaker, CodeWhisperer? What do you think's going to shine brightly within the AWS stack, if you will, or service list, that's going to be part of this? As you mentioned, CodeWhisperer and SageMaker, what else should people be looking at as they start tinkering and getting all these benefits, and scale up their ups? >> You know, if we were a startup, first, I would really work backwards from the customer problem I try to solve, and pick and choose, bar, I don't need to deal with the undifferentiated heavy lifting, so. And that's where the answer is going to change. If you look at it then, the answer is not going to be like a one size fits all, so you need a very strong, I mean, granted on the compute front, if you can actually completely accurate it, so unless, I will always recommend it, instead of running compute for running your ups, because it takes care of all the undifferentiated heavy lifting, but on the data, and that's where we provide a whole variety of databases, right from like relational data, or non-relational, or dynamo, and so forth. And of course, we also have a deep analytical stack, where data directly flows from our relational databases into data lakes and data virus. And you can get value along with partnership with various analytical providers. The area where I do think fundamentally things are changing on what people can do is like, with CodeWhisperer, I was literally trying to actually program a code on sending a message through Twilio, and I was going to pull up to read a documentation, and in my ID, I was actually saying like, let's try sending a message to Twilio, or let's actually update a Route 53 error code. All I had to do was type in just a comment, and it actually started generating the sub-routine. And it is going to be a huge time saver, if I were a developer. And the goal is for us not to actually do it just for AWS developers, and not to just generate the code, but make sure the code is actually highly secure and follows the best practices. So, it's not always about machine learning, it's augmenting with automated reasoning as well. And generative AI is going to be changing, and not just in how people write code, but also how it actually gets built and used as well. You'll see a lot more stuff coming on this front. >> Swami, thank you for your time. I know you're super busy. Thank you for sharing on the news and giving commentary. Again, I think this is a AWS moment and industry moment, heavy lifting, accelerated value, agility. AIOps is going to be probably redefined here. Thanks for sharing your commentary. And we'll see you next time, I'm looking forward to doing more follow up on this. It's going to be a big wave. Thanks. >> Okay. Thanks again, John, always a pleasure. >> Okay. This is SiliconANGLE's breaking news commentary. I'm John Furrier with SiliconANGLE News, as well as host of theCUBE. Swami, who's a leader in AWS, has been on theCUBE multiple times. We've been tracking the growth of how Amazon's journey has just been exploding past five years, in particular, past three. You heard the numbers, great performance, great reviews. This is a watershed moment, I think, for the industry, and it's going to be a lot of fun for the next 10 years. Thanks for watching. (bright music)

(upbeat music) >> Welcome back, everyone, to theCUBE's coverage here live on the floor at AWS re:MARS 2022. I'm John Furrier, host for theCUBE. Great event, machine learning, automation, robotics, space, that's MARS. It's part of the re-series of events, re:Invent's the big event at the end of the year, re:Inforce, security, re:MARS, really intersection of the future of space, industrial, automation, which is very heavily DevOps machine learning, of course, machine learning, which is AI. We have Luis Ceze here, who's the CEO co-founder of OctoML. Welcome to theCUBE. >> Thank you very much for having me in the show, John. >> So we've been following you guys. You guys are a growing startup funded by Madrona Venture Capital, one of your backers. You guys are here at the show. This is a, I would say small show relative what it's going to be, but a lot of robotics, a lot of space, a lot of industrial kind of edge, but machine learning is the centerpiece of this trend. You guys are in the middle of it. Tell us your story. >> Absolutely, yeah. So our mission is to make machine learning sustainable and accessible to everyone. So I say sustainable because it means we're going to make it faster and more efficient. You know, use less human effort, and accessible to everyone, accessible to as many developers as possible, and also accessible in any device. So, we started from an open source project that began at University of Washington, where I'm a professor there. And several of the co-founders were PhD students there. We started with this open source project called Apache TVM that had actually contributions and collaborations from Amazon and a bunch of other big tech companies. And that allows you to get a machine learning model and run on any hardware, like run on CPUs, GPUs, various GPUs, accelerators, and so on. It was the kernel of our company and the project's been around for about six years or so. Company is about three years old. And we grew from Apache TVM into a whole platform that essentially supports any model on any hardware cloud and edge. >> So is the thesis that, when it first started, that you want to be agnostic on platform? >> Agnostic on hardware, that's right. >> Hardware, hardware. >> Yeah. >> What was it like back then? What kind of hardware were you talking about back then? Cause a lot's changed, certainly on the silicon side. >> Luis: Absolutely, yeah. >> So take me through the journey, 'cause I could see the progression. I'm connecting the dots here. >> So once upon a time, yeah, no... (both chuckling) >> I walked in the snow with my bare feet. >> You have to be careful because if you wake up the professor in me, then you're going to be here for two hours, you know. >> Fast forward. >> The average version here is that, clearly machine learning has shown to actually solve real interesting, high value problems. And where machine learning runs in the end, it becomes code that runs on different hardware, right? And when we started Apache TVM, which stands for tensor virtual machine, at that time it was just beginning to start using GPUs for machine learning, we already saw that, with a bunch of machine learning models popping up and CPUs and GPU's starting to be used for machine learning, it was clear that it come opportunity to run on everywhere. >> And GPU's were coming fast. >> GPUs were coming and huge diversity of CPUs, of GPU's and accelerators now, and the ecosystem and the system software that maps models to hardware is still very fragmented today. So hardware vendors have their own specific stacks. So Nvidia has its own software stack, and so does Intel, AMD. And honestly, I mean, I hope I'm not being, you know, too controversial here to say that it kind of of looks like the mainframe era. We had tight coupling between hardware and software. You know, if you bought IBM hardware, you had to buy IBM OS and IBM database, IBM applications, it all tightly coupled. And if you want to use IBM software, you had to buy IBM hardware. So that's kind of like what machine learning systems look like today. If you buy a certain big name GPU, you've got to use their software. Even if you use their software, which is pretty good, you have to buy their GPUs, right? So, but you know, we wanted to help peel away the model and the software infrastructure from the hardware to give people choice, ability to run the models where it best suit them. Right? So that includes picking the best instance in the cloud, that's going to give you the right, you know, cost properties, performance properties, or might want to run it on the edge. You might run it on an accelerator. >> What year was that roughly, when you were going this? >> We started that project in 2015, 2016 >> Yeah. So that was pre-conventional wisdom. I think TensorFlow wasn't even around yet. >> Luis: No, it wasn't. >> It was, I'm thinking like 2017 or so. >> Luis: Right. So that was the beginning of, okay, this is opportunity. AWS, I don't think they had released some of the nitro stuff that the Hamilton was working on. So, they were already kind of going that way. It's kind of like converging. >> Luis: Yeah. >> The space was happening, exploding. >> Right. And the way that was dealt with, and to this day, you know, to a large extent as well is by backing machine learning models with a bunch of hardware specific libraries. And we were some of the first ones to say, like, know what, let's take a compilation approach, take a model and compile it to very efficient code for that specific hardware. And what underpins all of that is using machine learning for machine learning code optimization. Right? But it was way back when. We can talk about where we are today. >> No, let's fast forward. >> That's the beginning of the open source project. >> But that was a fundamental belief, worldview there. I mean, you have a world real view that was logical when you compare to the mainframe, but not obvious to the machine learning community. Okay, good call, check. Now let's fast forward, okay. Evolution, we'll go through the speed of the years. More chips are coming, you got GPUs, and seeing what's going on in AWS. Wow! Now it's booming. Now I got unlimited processors, I got silicon on chips, I got, everywhere >> Yeah. And what's interesting is that the ecosystem got even more complex, in fact. Because now you have, there's a cross product between machine learning models, frameworks like TensorFlow, PyTorch, Keras, and like that and so on, and then hardware targets. So how do you navigate that? What we want here, our vision is to say, folks should focus, people should focus on making the machine learning models do what they want to do that solves a value, like solves a problem of high value to them. Right? So another deployment should be completely automatic. Today, it's very, very manual to a large extent. So once you're serious about deploying machine learning model, you got a good understanding where you're going to deploy it, how you're going to deploy it, and then, you know, pick out the right libraries and compilers, and we automated the whole thing in our platform. This is why you see the tagline, the booth is right there, like bringing DevOps agility for machine learning, because our mission is to make that fully transparent. >> Well, I think that, first of all, I use that line here, cause I'm looking at it here on live on camera. People can't see, but it's like, I use it on a couple couple of my interviews because the word agility is very interesting because that's kind of the test on any kind of approach these days. Agility could be, and I talked to the robotics guys, just having their product be more agile. I talked to Pepsi here just before you came on, they had this large scale data environment because they built an architecture, but that fostered agility. So again, this is an architectural concept, it's a systems' view of agility being the output, and removing dependencies, which I think what you guys were trying to do. >> Only part of what we do. Right? So agility means a bunch of things. First, you know-- >> Yeah explain. >> Today it takes a couple months to get a model from, when the model's ready, to production, why not turn that in two hours. Agile, literally, physically agile, in terms of walk off time. Right? And then the other thing is give you flexibility to choose where your model should run. So, in our deployment, between the demo and the platform expansion that we announced yesterday, you know, we give the ability of getting your model and, you know, get it compiled, get it optimized for any instance in the cloud and automatically move it around. Today, that's not the case. You have to pick one instance and that's what you do. And then you might auto scale with that one instance. So we give the agility of actually running and scaling the model the way you want, and the way it gives you the right SLAs. >> Yeah, I think Swami was mentioning that, not specifically that use case for you, but that use case generally, that scale being moving things around, making them faster, not having to do that integration work. >> Scale, and run the models where they need to run. Like some day you want to have a large scale deployment in the cloud. You're going to have models in the edge for various reasons because speed of light is limited. We cannot make lights faster. So, you know, got to have some, that's a physics there you cannot change. There's privacy reasons. You want to keep data locally, not send it around to run the model locally. So anyways, and giving the flexibility. >> Let me jump in real quick. I want to ask this specific question because you made me think of something. So we're just having a data mesh conversation. And one of the comments that's come out of a few of these data as code conversations is data's the product now. So if you can move data to the edge, which everyone's talking about, you know, why move data if you don't have to, but I can move a machine learning algorithm to the edge. Cause it's costly to move data. I can move computer, everyone knows that. But now I can move machine learning to anywhere else and not worry about integrating on the fly. So the model is the code. >> It is the product. >> Yeah. And since you said, the model is the code, okay, now we're talking even more here. So machine learning models today are not treated as code, by the way. So do not have any of the typical properties of code that you can, whenever you write a piece of code, you run a code, you don't know, you don't even think what is a CPU, we don't think where it runs, what kind of CPU it runs, what kind of instance it runs. But with machine learning model, you do. So what we are doing and created this fully transparent automated way of allowing you to treat your machine learning models if you were a regular function that you call and then a function could run anywhere. >> Yeah. >> Right. >> That's why-- >> That's better. >> Bringing DevOps agility-- >> That's better. >> Yeah. And you can use existing-- >> That's better, because I can run it on the Artemis too, in space. >> You could, yeah. >> If they have the hardware. (both laugh) >> And that allows you to run your existing, continue to use your existing DevOps infrastructure and your existing people. >> So I have to ask you, cause since you're a professor, this is like a masterclass on theCube. Thank you for coming on. Professor. (Luis laughing) I'm a hardware guy. I'm building hardware for Boston Dynamics, Spot, the dog, that's the diversity in hardware, it's tends to be purpose driven. I got a spaceship, I'm going to have hardware on there. >> Luis: Right. >> It's generally viewed in the community here, that everyone I talk to and other communities, open source is going to drive all software. That's a check. But the scale and integration is super important. And they're also recognizing that hardware is really about the software. And they even said on stage, here. Hardware is not about the hardware, it's about the software. So if you believe that to be true, then your model checks all the boxes. Are people getting this? >> I think they're starting to. Here is why, right. A lot of companies that were hardware first, that thought about software too late, aren't making it. Right? There's a large number of hardware companies, AI chip companies that aren't making it. Probably some of them that won't make it, unfortunately just because they started thinking about software too late. I'm so glad to see a lot of the early, I hope I'm not just doing our own horn here, but Apache TVM, the infrastructure that we built to map models to different hardware, it's very flexible. So we see a lot of emerging chip companies like SiMa.ai's been doing fantastic work, and they use Apache TVM to map algorithms to their hardware. And there's a bunch of others that are also using Apache TVM. That's because you have, you know, an opening infrastructure that keeps it up to date with all the machine learning frameworks and models and allows you to extend to the chips that you want. So these companies pay attention that early, gives them a much higher fighting chance, I'd say. >> Well, first of all, not only are you backable by the VCs cause you have pedigree, you're a professor, you're smart, and you get good recruiting-- >> Luis: I don't know about the smart part. >> And you get good recruiting for PhDs out of University of Washington, which is not too shabby computer science department. But they want to make money. The VCs want to make money. >> Right. >> So you have to make money. So what's the pitch? What's the business model? >> Yeah. Absolutely. >> Share us what you're thinking there. >> Yeah. The value of using our solution is shorter time to value for your model from months to hours. Second, you shrink operator, op-packs, because you don't need a specialized expensive team. Talk about expensive, expensive engineers who can understand machine learning hardware and software engineering to deploy models. You don't need those teams if you use this automated solution, right? Then you reduce that. And also, in the process of actually getting a model and getting specialized to the hardware, making hardware aware, we're talking about a very significant performance improvement that leads to lower cost of deployment in the cloud. We're talking about very significant reduction in costs in cloud deployment. And also enabling new applications on the edge that weren't possible before. It creates, you know, latent value opportunities. Right? So, that's the high level value pitch. But how do we make money? Well, we charge for access to the platform. Right? >> Usage. Consumption. >> Yeah, and value based. Yeah, so it's consumption and value based. So depends on the scale of the deployment. If you're going to deploy machine learning model at a larger scale, chances are that it produces a lot of value. So then we'll capture some of that value in our pricing scale. >> So, you have direct sales force then to work those deals. >> Exactly. >> Got it. How many customers do you have? Just curious. >> So we started, the SaaS platform just launched now. So we started onboarding customers. We've been building this for a while. We have a bunch of, you know, partners that we can talk about openly, like, you know, revenue generating partners, that's fair to say. We work closely with Qualcomm to enable Snapdragon on TVM and hence our platform. We're close with AMD as well, enabling AMD hardware on the platform. We've been working closely with two hyperscaler cloud providers that-- >> I wonder who they are. >> I don't know who they are, right. >> Both start with the letter A. >> And they're both here, right. What is that? >> They both start with the letter A. >> Oh, that's right. >> I won't give it away. (laughing) >> Don't give it away. >> One has three, one has four. (both laugh) >> I'm guessing, by the way. >> Then we have customers in the, actually, early customers have been using the platform from the beginning in the consumer electronics space, in Japan, you know, self driving car technology, as well. As well as some AI first companies that actually, whose core value, the core business come from AI models. >> So, serious, serious customers. They got deep tech chops. They're integrating, they see this as a strategic part of their architecture. >> That's what I call AI native, exactly. But now there's, we have several enterprise customers in line now, we've been talking to. Of course, because now we launched the platform, now we started onboarding and exploring how we're going to serve it to these customers. But it's pretty clear that our technology can solve a lot of other pain points right now. And we're going to work with them as early customers to go and refine them. >> So, do you sell to the little guys, like us? Will we be customers if we wanted to be? >> You could, absolutely, yeah. >> What we have to do, have machine learning folks on staff? >> So, here's what you're going to have to do. Since you can see the booth, others can't. No, but they can certainly, you can try our demo. >> OctoML. >> And you should look at the transparent AI app that's compiled and optimized with our flow, and deployed and built with our flow. That allows you to get your image and do style transfer. You know, you can get you and a pineapple and see how you look like with a pineapple texture. >> We got a lot of transcript and video data. >> Right. Yeah. Right, exactly. So, you can use that. Then there's a very clear-- >> But I could use it. You're not blocking me from using it. Everyone's, it's pretty much democratized. >> You can try the demo, and then you can request access to the platform. >> But you get a lot of more serious deeper customers. But you can serve anybody, what you're saying. >> Luis: We can serve anybody, yeah. >> All right, so what's the vision going forward? Let me ask this. When did people start getting the epiphany of removing the machine learning from the hardware? Was it recently, a couple years ago? >> Well, on the research side, we helped start that trend a while ago. I don't need to repeat that. But I think the vision that's important here, I want the audience here to take away is that, there's a lot of progress being made in creating machine learning models. So, there's fantastic tools to deal with training data, and creating the models, and so on. And now there's a bunch of models that can solve real problems there. The question is, how do you very easily integrate that into your intelligent applications? Madrona Venture Group has been very vocal and investing heavily in intelligent applications both and user applications as well as enablers. So we say an enable of that because it's so easy to use our flow to get a model integrated into your application. Now, any regular software developer can integrate that. And that's just the beginning, right? Because, you know, now we have CI/CD integration to keep your models up to date, to continue to integrate, and then there's more downstream support for other features that you normally have in regular software development. >> I've been thinking about this for a long, long, time. And I think this whole code, no one thinks about code. Like, I write code, I'm deploying it. I think this idea of machine learning as code independent of other dependencies is really amazing. It's so obvious now that you say it. What's the choices now? Let's just say that, I buy it, I love it, I'm using it. Now what do I got to do if I want to deploy it? Do I have to pick processors? Are there verified platforms that you support? Is there a short list? Is there every piece of hardware? >> We actually can help you. I hope we're not saying we can do everything in the world here, but we can help you with that. So, here's how. When you have them all in the platform you can actually see how this model runs on any instance of any cloud, by the way. So we support all the three major cloud providers. And then you can make decisions. For example, if you care about latency, your model has to run on, at most 50 milliseconds, because you're going to have interactivity. And then, after that, you don't care if it's faster. All you care is that, is it going to run cheap enough. So we can help you navigate. And also going to make it automatic. >> It's like tire kicking in the dealer showroom. >> Right. >> You can test everything out, you can see the simulation. Are they simulations, or are they real tests? >> Oh, no, we run all in real hardware. So, we have, as I said, we support any instances of any of the major clouds. We actually run on the cloud. But we also support a select number of edge devices today, like ARMs and Nvidia Jetsons. And we have the OctoML cloud, which is a bunch of racks with a bunch Raspberry Pis and Nvidia Jetsons, and very soon, a bunch of mobile phones there too that can actually run the real hardware, and validate it, and test it out, so you can see that your model runs performant and economically enough in the cloud. And it can run on the edge devices-- >> You're a machine learning as a service. Would that be an accurate? >> That's part of it, because we're not doing the machine learning model itself. You come with a model and we make it deployable and make it ready to deploy. So, here's why it's important. Let me try. There's a large number of really interesting companies that do API models, as in API as a service. You have an NLP model, you have computer vision models, where you call an API and then point in the cloud. You send an image and you got a description, for example. But it is using a third party. Now, if you want to have your model on your infrastructure but having the same convenience as an API you can use our service. So, today, chances are that, if you have a model that you know that you want to do, there might not be an API for it, we actually automatically create the API for you. >> Okay, so that's why I get the DevOps agility for machine learning is a better description. Cause it's not, you're not providing the service. You're providing the service of deploying it like DevOps infrastructure as code. You're now ML as code. >> It's your model, your API, your infrastructure, but all of the convenience of having it ready to go, fully automatic, hands off. >> Cause I think what's interesting about this is that it brings the craftsmanship back to machine learning. Cause it's a craft. I mean, let's face it. >> Yeah. I want human brains, which are very precious resources, to focus on building those models, that is going to solve business problems. I don't want these very smart human brains figuring out how to scrub this into actually getting run the right way. This should be automatic. That's why we use machine learning, for machine learning to solve that. >> Here's an idea for you. We should write a book called, The Lean Machine Learning. Cause the lean startup was all about DevOps. >> Luis: We call machine leaning. No, that's not it going to work. (laughs) >> Remember when iteration was the big mantra. Oh, yeah, iterate. You know, that was from DevOps. >> Yeah, that's right. >> This code allowed for standing up stuff fast, double down, we all know the history, what it turned out. That was a good value for developers. >> I could really agree. If you don't mind me building on that point. You know, something we see as OctoML, but we also see at Madrona as well. Seeing that there's a trend towards best in breed for each one of the stages of getting a model deployed. From the data aspect of creating the data, and then to the model creation aspect, to the model deployment, and even model monitoring. Right? We develop integrations with all the major pieces of the ecosystem, such that you can integrate, say with model monitoring to go and monitor how a model is doing. Just like you monitor how code is doing in deployment in the cloud. >> It's evolution. I think it's a great step. And again, I love the analogy to the mainstream. I lived during those days. I remember the monolithic propriety, and then, you know, OSI model kind of blew it. But that OSI stack never went full stack, and it only stopped at TCP/IP. So, I think the same thing's going on here. You see some scalability around it to try to uncouple it, free it. >> Absolutely. And sustainability and accessibility to make it run faster and make it run on any deice that you want by any developer. So, that's the tagline. >> Luis Ceze, thanks for coming on. Professor. >> Thank you. >> I didn't know you were a professor. That's great to have you on. It was a masterclass in DevOps agility for machine learning. Thanks for coming on. Appreciate it. >> Thank you very much. Thank you. >> Congratulations, again. All right. OctoML here on theCube. Really important. Uncoupling the machine learning from the hardware specifically. That's only going to make space faster and safer, and more reliable. And that's where the whole theme of re:MARS is. Let's see how they fit in. I'm John for theCube. Thanks for watching. More coverage after this short break. >> Luis: Thank you. (gentle music)

(bright upbeat music) >> We're back at the Red Hat Summit at the Seaport in Boston, theCUBE's coverage. This is day two. Dave Vellante and Paul Gillin. Chris Wright is here, the chief technology officer at Red Hat. Chris, welcome back to theCUBE. Good to see you. >> Yeah, likewise. Thanks for having me. >> You're very welcome. So, you were saying today in your keynote. We got a lot of ground to cover here, Chris. You were saying that, you know, software, Andreessen's software is eating the world. Software ate the world, is what you said. And now we have to think about AI. AI is eating the world. What does that mean? What's the implication for customers and developers? >> Well, a lot of implications. I mean, to start with, just acknowledging that software isn't this future dream. It is the reality of how businesses run today. It's an important part of understanding what you need to invest in to make yourself successful, essentially, as a software company, where all companies are building technology to differentiate themselves. Take that, all that discipline, everything we've learned in that context, bring in AI. So, we have a whole new set of skills to learn, tools to create and discipline processes to build around delivering data-driven value into the company, just the way we've built software value into companies. >> I'm going to cut right to the chase because I would say data is eating software. Data and AI, to me, are like, you know, kissing cousins. So here's what I want to ask you as a technologist. So we have the application development stack, if you will. And it's separate from the data and analytics stack. All we talk about is injecting AI into applications, making them data-driven. You just used that term. But they're totally two totally separate stacks, organizationally and technically. Are those worlds coming together? Do they have to come together in order for the AI vision to be real? >> Absolutely, so, totally agree with you on the data piece. It's inextricably linked to AI and analytics and all of the, kind of, machine learning that goes on in creating intelligence for applications. The application connection to a machine learning model is fundamental. So, you got to think about not just the software developer or the data scientist, but also there's a line of business in there that's saying, "Here's the business outcomes I'm looking for." It's that trifecta that has to come together to make advancements and really make change in the business. So, you know, some of the folks we had on stage today were talking about exactly that. Which is, how do you bring together those three different roles? And there's technology that can help bridge gaps. So, we look at what we call intelligent applications. Embed intelligence into the application. That means you surface a machine learning model with APIs to make it accessible into applications, so that developers can query a machine learning model. You need to do that with some discipline and rigor around, you know, what does it mean to develop this thing and life cycle it and integrate it into this bigger picture. >> So the technology is capable of coming together. You know, Amanda Purnell is coming on next. >> Oh, great. >> 'Cause she was talking about, you know, getting, you know, insights in the hands of nurses and they're not coders. >> That's right. >> But they need data. But I feel like it's, well, I feel very strongly that it's an organizational challenge, more so. I think you're confirming. It's not really a technical challenge. I can insert a column into the application development stack and bring TensorFlow in or AI or data, whatever it is. It's not a technical issue. Is that fair? >> Well, there are some technical challenges. So, for example, data scientists. Kind of a scarce kind of skillset within any business. So, how do you scale data scientists into the developer population? Which will be a large population within an organization. So, there's tools that we can use to bring those worlds together. So, you know, it's not just TensorFlow but it's the entire workflow and platform of how you share the data, the data training models and then just deploying models into a runtime production environment. That looks similar to software development processes but it's slightly different. So, that's where a common platform can help bridge the gaps between that developer world and the data science world. >> Where is Red Hat's position in this evolving AI stack? I mean, you're not into developing tool sets like TensorFlow, right? >> Yeah, that's right. If you think about a lot of what we do, it's aggregate content together, bring a distribution of tools, giving flexibility to the user. Whether that's a developer, a system administrator, or a data scientist. So our role here is, one, make sure we work with our hardware partners to create accelerated environments for AI. So, that's sort of an enablement thing. The other is bring together those disparate tools into a workflow and give a platform that enables data scientists to choose which, is it PyTorch, is it TensorFlow? What's the best tool for you? And assemble that tool into your workflow and then proceed training, doing inference, and, you know, tuning and lather, rinse, repeat. >> So, to make your platform then, as receptive as possible, right? You're not trying to pick winners in what languages to work with or what frameworks? >> Yeah, that's right. I mean, picking winners is difficult. The world changes so rapidly. So we make big bets on key areas and certainly TensorFlow would be a great example. A lot of community attraction there. But our goal isn't to say that's the one tool that everybody should use. It's just one of the many tools in your toolbox. >> There are risks of not pursuing this, from an organization's perspective. A customer, they kind of get complacent and, you know, they could get disrupted, but there's also an industry risk. If the industry can't deliver this capability, what are the implications if the industry doesn't step up? I believe the industry will, just 'cause it always does. But what about customer complacency? We certainly saw that a lot with digital transformation and COVID sort of forced us to march to digital. What should we be thinking about of the implications of not leaning in? >> Well, I think that the disruption piece is key because there's always that spectrum of businesses. Some are more leaning in, invested in the future. Some are more laggards and kind of wait and see. Those leaning in tend to be separating themselves, wheat from the chaff. So, that's an important way to look at it. Also, if you think about it, many data science experiments fail within businesses. I think part of that is not having the rigor and discipline around connecting, not just the tools and data scientists together, but also looking at what business outcomes are you trying to drive? If you don't bring those things together then it sort of can be too academic and the business doesn't see the value. And so there's also the question of transparency. How do you understand why is a model predicting you should take a certain action or do a certain thing? As an industry, I think we need to focus on bringing tools together, bringing data together, and building better transparency into how models work. >> There's also a lot of activity around governance right now, AI governance. Particularly removing bias from ML models. Is that something that you are guiding your customers on? Or, how important do you feel this is at this point of AI's development? >> It's really important. I mean, the challenge is finding it and understanding, you know, we bring data that maybe already carrying a bias into a training process and building a model around that. How do you understand what the bias is in that model? There's a lot of open questions there and academic research to try to understand how you can ferret out, you know, essentially biased data and make it less biased or unbiased. Our role is really just bringing the toolset together so that you have the ability to do that as a business. So, we're not necessarily building the next machine learning algorithm or models or ways of building transparency into models, as much as building the platform and bringing the tools together that can give you that for your own organization. >> So, it brings up the question of architectures. I've been sort of a casual or even active observer of data architectures over the last, whatever, 15 years. They've been really centralized. Our data teams are highly specialized. You mentioned data scientists, but there's data engineers and there's data analysts and very hyper specialized roles that don't really scale that well. So there seems to be a move, talk about edge. We're going to talk about edge. The ultimate edge, which is space, very cool. But data is distributed by its very nature. We have this tendency to try to force it into this, you know, monolithic system. And I know that's a pejorative, but for good reason. So I feel like there's this push in organizations to enable scale, to decentralize data architectures. Okay, great. And put data in the hands of those business owners that you talked about earlier. The domain experts that have business context. Two things, two problems that brings up, is you need infrastructure that's self-service, in that instance. And you need, to your point, automated and computational governance. Those are real challenges. What do you see in terms of the trends to decentralize data architectures? Is it even feasible that everybody wants a single version of the truth, centralized data team, right? And they seem to be at odds. >> Yeah, well I think we're coming from a history informed by centralization. That's what we understand. That's what we kind of gravitate towards, but the reality, as you put it, the world's just distributed. So, what we can do is look at federation. So, it's not necessarily centralization but create connections between data sources which requires some policy and governance. Like, who gets access to what? And also think about those domain experts maybe being the primary source of surfacing a model that you don't necessarily have to know how it was trained or what the internals are. You're using it more to query it as a, you know, the domain expert produces this model, you're in a different part of the organization just leveraging some work that somebody else has done. Which is how we build software, reusable components in software. So, you know, I think building that mindset into data and the whole process of creating value from data is going to be a really critical part of how we roll forward. >> So, there are two things in your keynote. One, that I was kind of in awe of. You wanted to be an astronaut when you were a kid. You know, I mean, I watched the moon landing and I was like, "I'm never going up into space." So, I'm in awe of that. >> Oh, I got the space helmet picture and all that. >> That's awesome, really, you know, hat's off to you. The other one really pissed me off, which was that you're a better skier 'cause you got some device in your boot. >> Oh, it's amazing. >> And the reason it angered me is 'cause I feel like it's the mathematicians taking over baseball, you know. Now, you're saying, you're a better skier because of that. But those are two great edge examples and there's a billion of them, right? So, talk about your edge strategy. Kind of, your passion there, how you see that all evolving. >> Well, first of all, we see the edge as a fundamental part of the future of computing. So in that centralization, decentralization pendulum swing, we're definitely on the path towards distributed computing and that is edge and that's because of data. And also because of the compute capabilities that we have in hardware. Hardware gets more capable, lower power, can bring certain types of accelerators into the mix. And you really create this world where what's happening in a virtual context and what's happening in a physical context can come together through this distributed computing system. Our view is, that's hybrid. That's what we've been working on for years. Just the difference was maybe, originally it was focused on data center, cloud, multi-cloud and now we're just extending that view out to the edge and you need the same kind of consistency for development, for operations, in the edge that you do in that hybrid world. So that's really where we're placing our focus and then it gets into all the different use cases. And you know, really, that's the fun part. >> I'd like to shift gears a little bit 'cause another remarkable statistic you cited during your keynote was, it was a Forrester study that said 99% of all applications now have open source in them. What are the implications of that for those who are building applications? In terms of license compliance and more importantly, I think, confidence in the code that they're borrowing from open source projects. >> Well, I think, first and foremost, it says open source has won. We see that that was audited code bases which means there's mission critical code bases. We see that it's pervasive, it's absolutely everywhere. And that means developers are pulling dependencies into their applications based on all of the genius that's happening in open source communities. Which I think we should celebrate. Right after we're finished celebrating we got to look at what are the implications, right? And that shows up as, are there security vulnerabilities that become ubiquitous because we're using similar dependencies? What is your process for vetting code that you bring into your organization and push into production? You know that process for the code you author, what about your dependencies? And I think that's an important part of understanding and certainly there are some license implications. What are you required to do when you use that code? You've been given that code on a license from the open source community, are you compliant with that license? Some of those are reasonably well understood. Some of those are, you know, newer to the enterprise. So I think we have to look at this holistically and really help enterprises build safe application code that goes into production and runs their business. >> We saw Intel up in the keynotes today. We heard from Nvidia, both companies are coming on. We know you've done a lot of work with ARM over the years. I think Graviton was one of the announcements this week. So, love to see that. I want to run something by you as a technologist. The premise is, you know, we used to live in this CPU centric world. We marched to the cadence of Moore's Law and now we're seeing the combinatorial factors of CPU, GPU, NPU, accelerators and other supporting components. With IO and controllers and NICs all adding up. It seems like we're shifting from a processor centric world to a connect centric world on the hardware side. That first of all, do you buy that premise? And does hardware matter anymore with all the cloud? >> Hardware totally matters. I mean the cloud tried to convince us that hardware doesn't matter and it actually failed. And the reason I say that is because if you go to a cloud, you'll find 100s of different instance types that are all reflections of different types of assemblies of hardware. Faster IO, better storage, certain sizes of memory. All of that is a reflection of, applications need certain types of environments for acceleration, for performance, to do their job. Now I do think there's an element of, we're decomposing compute into all of these different sort of accelerators and the only way to bring that back together is connectivity through the network. But there's also SOCs when you get to the edge where you can integrate the entire system onto a pretty small device. I think the important part here is, we're leveraging hardware to do interesting work on behalf of applications that makes hardware exciting. And as an operating system geek, I couldn't be more thrilled, because that's what we do. We enable hardware, we get down into the bits and bytes and poke registers and bring things to life. There's a lot happening in the hardware world and applications can't always follow it directly. They need that level of indirection through a software abstraction and that's really what we're bringing to life here. >> We've seen now hardware specific AI, you know, AI chips and AI SOCs emerge. How do you make decisions about what you're going to support or do you try to support all of them? >> Well, we definitely have a breadth view of support and we're also just driven by customer demand. Where our customers are interested we work closely with our partners. We understand what their roadmaps are. We plan together ahead of time and we know where they're making investments and we work with our customers. What are the best chips that support their business needs and we focus there first but it ends up being a pretty broad list of hardware that we support. >> I could pick your brain for an hour. We didn't even get into super cloud, Chris. But, thanks so much for coming on theCUBE. It's great to have you. >> Absolutely, thanks for having me. >> All right. Thank you for watching. Keep it right there. Paul Gillin, Dave Vellante, theCUBE's live coverage of Red Hat Summit 2022 from Boston. We'll be right back. (mellow music)

(bright music) >> Well, welcome back to theCUBE's coverage of AWS re:Invent 2021. We're here joined by Ian Buck, general manager and vice president of Accelerated Computing at NVIDIA. I'm John Furrrier, host of theCUBE. Ian, thanks for coming on. >> Oh, thanks for having me. >> So NVIDIA, obviously, great brand. Congratulations on all your continued success. Everyone who does anything in graphics knows that GPU's are hot, and you guys have a great brand, great success in the company. But AI and machine learning, we're seeing the trend significantly being powered by the GPU's and other systems. So it's a key part of everything. So what's the trends that you're seeing in ML and AI that's accelerating computing to the cloud? >> Yeah. I mean, AI is kind of driving breakthroughs and innovations across so many segments, so many different use cases. We see it showing up with things like credit card fraud prevention, and product and content recommendations. Really, it's the new engine behind search engines, is AI. People are applying AI to things like meeting transcriptions, virtual calls like this, using AI to actually capture what was said. And that gets applied in person-to-person interactions. We also see it in intelligence assistance for contact center automation, or chat bots, medical imaging, and intelligence stores, and warehouses, and everywhere. It's really amazing what AI has been demonstrating, what it can do, and its new use cases are showing up all the time. >> You know, Ian, I'd love to get your thoughts on how the world's evolved, just in the past few years alone, with cloud. And certainly, the pandemic's proven it. You had this whole kind of fullstack mindset, initially, and now you're seeing more of a horizontal scale, but yet, enabling this vertical specialization in applications. I mean, you mentioned some of those apps. The new enablers, this kind of, the horizontal play with enablement for, you know, specialization with data, this is a huge shift that's going on. It's been happening. What's your reaction to that? >> Yeah. The innovation's on two fronts. There's a horizontal front, which is basically the different kinds of neural networks or AIs, as well as machine learning techniques, that are just being invented by researchers and the community at large, including Amazon. You know, it started with these convolutional neural networks, which are great for image processing, but has expanded more recently into recurrent neural networks, transformer models, which are great for language and language and understanding, and then the new hot topic, graph neural networks, where the actual graph now is trained as a neural network. You have this underpinning of great AI technologies that are being invented around the world. NVIDIA's role is to try to productize that and provide a platform for people to do that innovation. And then, take the next step and innovate vertically. Take it and apply it to a particular field, like medical, like healthcare and medical imaging, applying AI so that radiologists can have an AI assistant with them and highlight different parts of the scan that may be troublesome or worrying, or require some more investigation. Using it for robotics, building virtual worlds where robots can be trained in a virtual environment, their AI being constantly trained and reinforced, and learn how to do certain activities and techniques. So that the first time it's ever downloaded into a real robot, it works right out of the box. To activate that, we are creating different vertical solutions, vertical stacks, vertical products, that talk the languages of those businesses, of those users. In medical imaging, it's processing medical data, which is obviously a very complicated, large format data, often three-dimensional voxels. In robotics, it's building, combining both our graphics and simulation technologies, along with the AI training capabilities and difference capabilities, in order to run in real time. Those are just two simple- >> Yeah, no. I mean, it's just so cutting-edge, it's so relevant. I mean, I think one of the things you mentioned about the neural networks, specifically, the graph neural networks, I mean, we saw, I mean, just go back to the late 2000s, how unstructured data, or object storage created, a lot of people realized a lot of value out of that. Now you got graph value, you got network effect, you got all kinds of new patterns. You guys have this notion of graph neural networks that's out there. What is a graph neural network, and what does it actually mean from a deep learning and an AI perspective? >> Yeah. I mean, a graph is exactly what it sounds like. You have points that are connected to each other, that establish relationships. In the example of Amazon.com, you might have buyers, distributors, sellers, and all of them are buying, or recommending, or selling different products. And they're represented in a graph. If I buy something from you and from you, I'm connected to those endpoints, and likewise, more deeply across a supply chain, or warehouse, or other buyers and sellers across the network. What's new right now is, that those connections now can be treated and trained like a neural network, understanding the relationship, how strong is that connection between that buyer and seller, or the distributor and supplier, and then build up a network to figure out and understand patterns across them. For example, what products I may like, 'cause I have this connection in my graph, what other products may meet those requirements? Or, also, identifying things like fraud, When patterns and buying patterns don't match what a graph neural networks should say would be the typical kind of graph connectivity, the different kind of weights and connections between the two, captured by the frequency of how often I buy things, or how I rate them or give them stars, or other such use cases. This application, graph neural networks, which is basically capturing the connections of all things with all people, especially in the world of e-commerce, is very exciting to a new application of applying AI to optimizing business, to reducing fraud, and letting us, you know, get access to the products that we want. They have our recommendations be things that excite us and want us to buy things, and buy more. >> That's a great setup for the real conversation that's going on here at re:Invent, which is new kinds of workloads are changing the game, people are refactoring their business with, not just re-platforming, but actually using this to identify value. And also, your cloud scale allows you to have the compute power to, you know, look at a note in an arc and actually code that. It's all science, it's all computer science, all at scale. So with that, that brings up the whole AWS relationship. Can you tell us how you're working with AWS, specifically? >> Yeah, AWS have been a great partner, and one of the first cloud providers to ever provide GPUs to the cloud. More recently, we've announced two new instances, the G5 instance, which is based on our A10G GPU, which supports the NVIDIA RTX technology, our rendering technology, for real-time ray tracing in graphics and game streaming. This is our highest performance graphics enhanced application, allows for those high-performance graphics applications to be directly hosted in the cloud. And, of course, runs everything else as well. It has access to our AI technology and runs all of our AI stacks. We also announced, with AWS, the G5 G instance. This is exciting because it's the first Graviton or Arm-based processor connected to a GPU and successful in the cloud. The focus here is Android gaming and machine learning inference. And we're excited to see the advancements that Amazon is making and AWS is making, with Arm in the cloud. And we're glad to be part of that journey. >> Well, congratulations. I remember, I was just watching my interview with James Hamilton from AWS 2013 and 2014. He was teasing this out, that they're going to build their own, get in there, and build their own connections to take that latency down and do other things. This is kind of the harvest of all that. As you start looking at these new interfaces, and the new servers, new technology that you guys are doing, you're enabling applications. What do you see this enabling? As this new capability comes out, new speed, more performance, but also, now it's enabling more capabilities so that new workloads can be realized. What would you say to folks who want to ask that question? >> Well, so first off, I think Arm is here to stay. We can see the growth and explosion of Arm, led of course, by Graviton and AWS, but many others. And by bringing all of NVIDIA's rendering graphics, machine learning and AI technologies to Arm, we can help bring that innovation that Arm allows, that open innovation, because there's an open architecture, to the entire ecosystem. We can help bring it forward to the state of the art in AI machine learning and graphics. All of our software that we release is both supportive, both on x86 and on Arm equally, and including all of our AI stacks. So most notably, for inference, the deployment of AI models, we have the NVIDIA Triton inference server. This is our inference serving software, where after you've trained a model, you want to deploy it at scale on any CPU, or GPU instance, for that matter. So we support both CPUs and GPUs with Triton. It's natively integrated with SageMaker and provides the benefit of all those performance optimizations. Features like dynamic batching, it supports all the different AI frameworks, from PyTorch to TensorFlow, even a generalized Python code. We're activating, and help activating, the Arm ecosystem, as well as bringing all those new AI use cases, and all those different performance levels with our partnership with AWS and all the different cloud instances. >> And you guys are making it really easy for people to use use the technology. That brings up the next, kind of, question I wanted to ask you. I mean, a lot of people are really going in, jumping in big-time into this. They're adopting AI, either they're moving it from prototype to production. There's always some gaps, whether it's, you know, knowledge, skills gaps, or whatever. But people are accelerating into the AI and leaning into it hard. What advancements has NVIDIA made to make it more accessible for people to move faster through the system, through the process? >> Yeah. It's one of the biggest challenges. You know, the promise of AI, all the publications that are coming out, all the great research, you know, how can you make it more accessible or easier to use by more people? Rather than just being an AI researcher, which is obviously a very challenging and interesting field, but not one that's directly connected to the business. NVIDIA is trying to provide a fullstack approach to AI. So as we discover or see these AI technologies become available, we produce SDKs to help activate them or connect them with developers around the world. We have over 150 different SDKs at this point, serving industries from gaming, to design, to life sciences, to earth sciences. We even have stuff to help simulate quantum computing. And of course, all the work we're doing with AI, 5G, and robotics. So we actually just introduced about 65 new updates, just this past month, on all those SDKs. Some of the newer stuff that's really exciting is the large language models. People are building some amazing AI that's capable of understanding the corpus of, like, human understanding. These language models that are trained on literally the content of the internet to provide general purpose or open-domain chatbots, so the customer is going to have a new kind of experience with the computer or the cloud. We're offering those large language models, as well as AI frameworks, to help companies take advantage of this new kind of technology. >> You know, Ian, every time I do an interview with NVIDIA or talk about NVIDIA, my kids and friends, first thing they say is, "Can you get me a good graphics card?" They all want the best thing in their rig. Obviously the gaming market's hot and known for that. But there's a huge software team behind NVIDIA. This is well-known. Your CEO is always talking about it on his keynotes. You're in the software business. And you do have hardware, you are integrating with Graviton and other things. But it's a software practice. This is software. This is all about software. >> Right. >> Can you share, kind of, more about how NVIDIA culture and their cloud culture, and specifically around the scale, I mean, you hit every use case. So what's the software culture there at NVIDIA? >> Yeah, NVIDIA's actually a bigger, we have more software people than hardware people. But people don't often realize this. And in fact, that it's because of, it just starts with the chip, and obviously, building great silicon is necessary to provide that level of innovation. But it's expanded dramatically from there. Not just the silicon and the GPU, but the server designs themselves. We actually do entire server designs ourselves, to help build out this infrastructure. We consume it and use it ourselves, and build our own supercomputers to use AI to improve our products. And then, all that software that we build on top, we make it available, as I mentioned before, as containers on our NGC container store, container registry, which is accessible from AWS, to connect to those vertical markets. Instead of just opening up the hardware and letting the ecosystem develop on it, they can, with the low-level and programmatic stacks that we provide with CUDA. We believe that those vertical stacks are the ways we can help accelerate and advance AI. And that's why we make them so available. >> And programmable software is so much easier. I want to get that plug in for, I think it's worth noting that you guys are heavy hardcore, especially on the AI side, and it's worth calling out. Getting back to the customers who are bridging that gap and getting out there, what are the metrics they should consider as they're deploying AI? What are success metrics? What does success look like? Can you share any insight into what they should be thinking about, and looking at how they're doing? >> Yeah. For training, it's all about time-to-solution. It's not the hardware that's the cost, it's the opportunity that AI can provide to your business, and the productivity of those data scientists which are developing them, which are not easy to come by. So what we hear from customers is they need a fast time-to-solution to allow people to prototype very quickly, to train a model to convergence, to get into production quickly, and of course, move on to the next or continue to refine it. >> John Furrier: Often. >> So in training, it's time-to-solution. For inference, it's about your ability to deploy at scale. Often people need to have real-time requirements. They want to run in a certain amount of latency, in a certain amount of time. And typically, most companies don't have a single AI model. They have a collection of them they want to run for a single service or across multiple services. That's where you can aggregate some of your infrastructure. Leveraging the Triton inference server, I mentioned before, can actually run multiple models on a single GPU saving costs, optimizing for efficiency, yet still meeting the requirements for latency and the real-time experience, so that our customers have a good interaction with the AI. >> Awesome. Great. Let's get into the customer examples. You guys have, obviously, great customers. Can you share some of the use cases examples with customers, notable customers? >> Yeah. One great part about working at NVIDIA is, as technology company, you get to engage with such amazing customers across many verticals. Some of the ones that are pretty exciting right now, Netflix is using the G4 instances to do a video effects and animation content from anywhere in the world, in the cloud, as a cloud creation content platform. We work in the energy field. Siemens energy is actually using AI combined with simulation to do predictive maintenance on their energy plants, preventing, or optimizing, onsite inspection activities and eliminating downtime, which is saving a lot of money for the energy industry. We have worked with Oxford University. Oxford University actually has over 20 million artifacts and specimens and collections, across its gardens and museums and libraries. They're actually using NVIDIA GPU's and Amazon to do enhanced image recognition to classify all these things, which would take literally years going through manually, each of these artifacts. Using AI, we can quickly catalog all of them and connect them with their users. Great stories across graphics, across industries, across research, that it's just so exciting to see what people are doing with our technology, together with Amazon. >> Ian, thank you so much for coming on theCUBE. I really appreciate it. A lot of great content there. We probably could go another hour. All the great stuff going on at NVIDIA. Any closing remarks you want to share, as we wrap this last minute up? >> You know, really what NVIDIA's about, is accelerating cloud computing. Whether it be AI, machine learning, graphics, or high-performance computing and simulation. And AWS was one of the first with this, in the beginning, and they continue to bring out great instances to help connect the cloud and accelerated computing with all the different opportunities. The integrations with EC2, with SageMaker, with EKS, and ECS. The new instances with G5 and G5 G. Very excited to see all the work that we're doing together. >> Ian Buck, general manager and vice president of Accelerated Computing. I mean, how can you not love that title? We want more power, more faster, come on. More computing. No one's going to complain with more computing. Ian, thanks for coming on. >> Thank you. >> Appreciate it. I'm John Furrier, host of theCUBE. You're watching Amazon coverage re:Invent 2021. Thanks for watching. (bright music)

(upbeat music) >> Hello and welcome to theCUBE as we present AWS Startup Showcase, The Next Big Thing in AI, Security & Life Sciences, the hottest startups. And today's session is really the next big thing in AI Security & Life Sciences. As to the AI track is really a big one most important. And we have a feature in company, fiddler.ai. I'm your host, John Furrier with theCUBE. And we're joined by the founders, Krishna Gade, founder and CEO, and Amit Paka, founder and Chief Product Officer. Great to have the founders on. Gentlemen, thank you for coming on this Cube segment for the AWS Startup Showcase. >> Thanks, John... >> Good to be here. >> So the topic of this session is staying compliant and accelerating AI adoption and model performance monitoring. Basically, bottom line is how to be innovative with AI and stay (John laughs) within the rules of the road, if you will. So, super important topic. Everyone knows the benefits of what AI can do. Everyone sees machine learning being embedded in every single application, but the business drivers of compliance and all kinds of new kinds of regulations are popping up. So we don't. The question is how do you stay compliant? Which is essentially how do you not foreclose the future opportunities? That's really the question on everyone's mind these days. So let's get into it. But before we start let's take a minute to explain what you guys do. Krishna, we'll start with you first. What does fiddler.ai do? >> Absolutely, yeah. Fiddler is a model performance management platform company. We help, you know, enterprises, mid-market companies to build responsible AI by helping them continuously monitoring their AI, analyzing it, explaining it, so that they know what's going on with their AI solutions at any given point of time. And they can be like, ensuring that, you know businesses are intact and they're compliant with all the regulations that they have in their industry. >> Everyone thinks AI is a secret sauce. It's magic beans and automatically will just change over the company. (John laughs) So it's kind of like this almost like it's a hope. But the reality is there is some value there but there's something that has to be done first. So let's get into what this model performance management is because it's a concept that needs to be understood well but also you got to implement it properly. There's some foundational things you've got to you know, walk, crawl before you walk and walk before you run kind of thing. So let's get into it. What is model performance management? >> Yeah, that's a great question. So the core software artifact most an AI system is called an AI model. So it essentially represents the patterns inside data accessing manner so that it can actually predict the future. Now, for example, let's say I'm trying to build an AI based credit underwriting system. What I would do is I would look at the historical you know, loans data. You know, good loans and bad loans. And then, I will build it a model that can capture those patterns so that when a new customer comes in I can actually predict, you know, how likely they are going to default on the loan much more activity. And this helps me as a bank or center company to produce more good loans for my company and ensure that my customer is not, you know, getting the right customer service. Now, the problem though is this AI model is a black box. Unlike regular software code you cannot really open up and read its code and read its patterns and how it is doing. And so that's where the risks around the AI models come along. And so you need a ways to innovate to actually explain it. You need to understand it and you need to monitor it. And this is where the model performance management system like Fiddler can help you look into that black box. Understand how it's doing it, monitor its predictions continuously so that you know what these models are doing at any given point of time. >> I mean, I'd love to get your thoughts on this because on the product side I could, first of all, totally awesome concept. No one debates that. But now you've got more and more companies integrating with each other more data's being shared. And so the, you know, everyone knows what an app sec review is, right? But now they're thinking about this concept of how do you do review of models, right? So understanding what's inside the black box is a huge thing. How do you do this? What does it mean? >> Yeah, so typically what you would do is it's just like software where you would validate software code going through QA and like analysis. In case of models you would try to prove the model in like different granularities to really understand how the model is behaving. This could be at a model prediction like level in case of the loans example, Krishna just gave. Why is my model saying high-risk to in particular loan? Or it might be in case of explaining groups of loans. For example, why is my model making high-risk predictions to loans made in California or loans made to all men? Was it loans made to all women? And it could also be at the global level. What are the key data factors important to my model? So the ability to prove the model deeper and really opening up the black box and then using that knowledge to explain how the model is working to non-technical folks in compliance. Or to folks who are regulators, who just want to ensure that they know how the model works to make sure that it's keeping up with kind of lending regulations to ensure that it's not biased and so on. So that's typically the way you would do it with the machine learning model. >> Krishna, talk about the potential embarrassments that could happen. You just mentioned some of the use cases you heard from a mid-saying you know, female, male. I mean, machines, aren't that smart. (John laughs) >> Yeah. >> If they don't have the data. >> Yeah. >> And data is fragmented you've got silos with all kinds of challenges just on the data problem, right? >> Yeah. >> So nevermind the machine learning problems. So, this is huge. I mean, the embarrassment opportunities. >> Yeah. >> And the risk management on whether it's a hack or something else. So you've got public embarrassment by doing something really went wrong. And then, you've got the real business impact that could be damaging. >> Absolutely. You know, AI has come forward a lot, right? I mean, you know, you have lots of data these days. You have a lot of computing power an amazing algorithms that you can actually build really sophisticated models. Some of these models were known to beat humans in image recognition and whatnot. However, the problem is there are risks in using AI, you know, without properly testing it, without properly monitoring it. For example, a couple of years ago, Apple and Goldman Sachs launched a credit card, right? And for their users where they were using algorithms presumably AI or machine learning algorithms to set credit limits. What happened was within the same household husband and wife got 10 times difference in the credit limits being set for them. And some of these people had similar FICO scores, similar salary ranges. And some of them went online and complained about it and that included the likes of Steve Wozniak as well. >> Yeah. >> So this was, these kind of stories are usually embarrassing when you could lose customer trust overnight, right? And, you know, you have to do a lot of PR damage. Eventually, there was a regulatory probate with Goldman Sachs. So there are these problems if you're not properly monitoring area systems, properly validating and testing them before you launch to the users. And that is why tools like Fiddler are coming forward so that you know, enterprises can do this. So that they can ensure responsible AI for both their organization as well as their customers. >> That's a great point, I want to get into this. What it kind of means and the kind of the industry side of it? And then, how that impacts customers? If you guys don't mind, machine learning opposite a term MLOps has been coined in the industry as you know. Basically, operations around machine learning, which kind of gets into the workflows and development life cycles. But ultimately, as you mentioned, this black box and this model being made. There's a heavy reliance on data. So Amit, what does this mean? Because now is it becomes operational with MLOps. There is now internal workflows and activities and roles and responsibilities. How is this changing organizations, you know separate the embarrassment, which is totally true. Now I've got an internal operational aspect and there's dev involved. What's the issue? >> Yeah, so typically, so if you look at the whole life cycle of machine learning ops, in some ways mirrors the traditional life cycle of kind of DevOps but in some ways it introduces new complexities. Specifically, because the models can be a black box. That's one thing to kind of watch out for. And secondly, because these models are probabilistic artifact, which means they are trained on data to grab relationships for what kind of potentially making high accuracy predictions. But the data that they see in life might actually differ and that might hurt their performance especially because machine learning is applied towards these high ROI use cases. So this process of MLOps needs to change to incorporate the fact that machine learning models can be black boxes and machine learning models can decay. And so the second part I think that's also relevant is because machine learning models can decay. You don't just create one model you create multiple versions of these models. And so you have to constantly stay on top of how your model is deviating from your reality and actual reality and kind of bring it back to that representation of reality. >> So this is interesting, I like this. So now there's a model for the model. So this is interesting. You guys have innovated on this model performance management idea. Can you explain the framework and how you guys solve that regulatory compliance piece? Because if you can be a model of the model, if you will. >> Then. >> Then you can then have some stability around maintaining the code basis or the integrity of the model. >> Okay. >> How does that? What do you guys offer? Take us through the framework and how it works and then how it ties to that regulatory piece? >> So the MPM system or the model performance management system really sits at the heart of the machine learning workflow. Keeping track of the data that is flowing through your ML life cycle, keeping track of the models that are going, you know, we're getting created and getting deployed and how they're performing. Keeping track of the whole parts of the models. So it gives you a centralized way of managing all of these information in one place, right? It gives you an oversight from a compliance standpoint from an operational standpoint of what's going on with your models in production. Imagine you're a bank you're probably creating hundreds of these models, but a variety of use cases, credit risk, fraud, anti-money laundering. How are you going to know which models are actually working very well? Which models are stale? Which models are expired? How do you know which models are underperforming? You know, are you getting alerts? So this is what this kind of governance, this performance management is what the system offers. It's a visual interface, lots of dashboards, the developers, operations folks, compliance folks can go and look into. And then they would get alerts when things go wrong with respect to their models. In terms of how it can be helpful to meet in compliance regulations. For example, let's say I'm starting to create a new credit risk model in a bank. Now I'm innovating on different AI algorithms here immediately before I even deploy that model I have to validate it. I have to explain it and create a report so that I can submit to my internal risk management team which can then review it, you know, understand all kinds of risks around it. And then potentially share it with the audit team and then keep a log of these reports so that when a regulator comes visits them, you know they can share these reports. These are the model reports. Is that how the model was created? Fiddler helps them create these reports, keep all of these reports in one place. And then once the model is deployed, you know, it basically can help them monitor these models continuously. So that they don't just have one ad hoc report when it was created upfront, they can a continuous monitoring continuous dashboard in terms of what it was doing in the last one whatever number of months it was running for. >> You know what? >> Historically, if you were to regulate it like all AI applications in the U.S. the legacy regulations are the ones that today are applied as to the equal credit opportunity or the Fed guidelines of like SR 11-7 that kind of comment that's applicable to all banks. So there is no purpose-built AI regulation but the EU released a proposed regulation just about three weeks back. That classifies risk within applications, and specifically for high-risk applications. They propose new oversight and the ads mandating explainability helping teams understand how the models are working and monitoring to ensure that when a model is trained for high accuracy, it maintains that. So now those two mandatory needs of high risk application, those are the ones that are solved by Fiddler. >> Yeah, this is, you mentioned explainable AI. Could you just quickly define that for the audience? Because this is a trend we're seeing a lot more of. Take a minute to explain what is explainable AI? >> Yeah, as I said in the beginning, you know AI model is a new software artifact that is being created. It is the core of an AI system. It's what represents all the patterns in the data and coach them and then uses that knowledge to predict the future. Now how it encodes all of these patterns is black magic, right? >> Yeah. >> You really don't know how the model is working. And so explainable AI is a set of technologies that can help you unlock that black box. You know, quote-unquote debug that model, looking to the model is introspected inspected, probate, whatever you want to call it, to understand how it works. For example, let's say I created an AI model, that again, predicts, you know, loan risk. Now let's say some person, a person comes to my bank and applies for a $10,000 loan, and the bank rejects the loan or the model rejects the loan. Now, why did it do it, right? That's a question that can explain the way I can answer. They can answer, hey, you know, the person's, you know salary range, you know, is contributing to 20% of the loan risk or this person's previous debt is contributing to 30% of the loan risk. So you can get a detailed set of dashboards in terms of attribution of taking the loan risk, the composite loan risk, and then attributing it to all the inputs that the model is observing. And so therefore, you now know how the moral is treating each of these inputs. And so now you have an idea of like where the person is getting effected by this loaner's mark. So now as a human, as an underwriter or a loan officer lending officer, I have knowledge about how the model is working. I can then have my human intuition or lap on it. I can approve the model sometimes I can disapprove the model sometimes. I can use this feedback and deliver it to the data science team, the AI team, so they can actually make the model better over time. So this unlocking black box has several benefits throughout their life cycle. >> That's awesome. Great definition. Great call. I want to grab get that on the record for the audience. Also, we'll make a clip out of that too. One of the things that I meant you brought up I love and want to get into is this MLOps impact. So as we were just talking earlier debugging module models and production, totally cool, relevant, unpacked a black box. But model decay, that's an interesting concept. Can you explain more? Because this to me, I think is potentially a big blind spot for the industry, because, you know, I talked to Swami at Amazon, who runs their AI group and, you know, they want to make AI easier and ML easier with SageMaker and other tools. But you can fall into a trap of thinking everything's done at one and done. It's iterative is you've got leverage here. You got to keep track of the performance of the models, not just debugging them. Are they actually working? Is there new data? This is a whole another practice. Could you explain this concept of model decay? >> Yeah, so let's look at the lending example Krishna was just talking about. If you expect your customers to be your citizens, right? So you will have examples in your training set which might have historical loans made to people that the needs of 40, and let's say 70. And so you will train your model and your model will be trained our highest accuracy in making loans to these type of applicants. But now let's say introduced a new loan product that you're targeting, let's say younger college going folks. So that model is not trained to work well in those kinds of scenarios. Or it could also happen that you could get a lot more older people coming in to apply for these loans. So the data that the model can see in life might not represent the data that you train the model with. And the model has recognized relationships in this data and it might not recognize relationships in this new data. So this is a constant, I would say, it's an ongoing challenge that you would face when you have a live model in ensuring that the reality meets your representation of the reality when you train the model. And so this is something that's unique to machine learning models and it has not been a problem historically in the world of DevOps. But it is a very key problem in the DevOps. >> This is really great topic. And most people who are watching might want to might know of some of these problems when they see the main mainstream press talk about fairness in black versus white skin and bias and algorithms. I mean, that's kind of like the press state that talk about those kinds of like big high level topics. But what it really means is that the data (John laughs) of practiced fairness and bias and skewing and all kinds of new things that come up that the machines just can't handle. This is a big deal. So this is happening to every part of data in an organization. So, great problem statement. I guess the next segue would be, why Fiddler, why now? What are you guys doing? How are you solving these problems? Take us through some use cases. How people engage with you guys? How you solve the problem and how you guys see this evolving? >> Great, so Fiddler is a purpose-built platform to solve for model explainability of modern monitoring and moderate bias detection. This is the only thing that we do, right? So we are super focused on building this tool to be useful across a variety of, you know, AI problems, from financial services to retail, to advertising to human resources, healthcare and so on and so forth. And so we have found a lot of commonalities around how data scientists are solving these problems across these industries. And we've created a system that can be plugged into their workflows. For example, I could be a bank, you know, creating anti-money laundering models on a modern AI platform like TensorFlow. Or I could be like a retail company that is building a recommendation models in, you know, PyTorch, like library. You can bring all of those models into one under one sort of umbrella, like using Fiddler. We can support a variety of heterogeneous types of models. And that is a very very hard technical problem to solve. To be able to ingest and digest all these different types of monotypes and then provide a single pane of glass in terms of how the model is performing. How explaining the model, tracking the model life cycle throughout its existence, right? And so that is the value prop that Fiddler offers, the MLOps team, so they can get this oversight. And so this plugs in nicely with their MLOps so they don't have to change anything and give the additional benefit... >> So, you're basically creating faster outcomes because the teams can work on real problems. >> Right. >> And not have to deal with the maintenance of model management. >> Right. >> Whether it's debugging or decay evaluations, right? >> Right, we take care of all of their model operations from a monitoring standpoint, analysis standpoint, debugability, alerting. So that they can just build the right kind of models for their customers. And we give them all the insights and intelligence to know the problems with behind those models behind their datasets. So that they can actually build more accurate models more responsible models for their customers. >> Okay, Amit, give us the secret sauce. What's going on in the product? How does it all work? What's the secret sauce? >> So there are three key kind of pillars to Fiddler product. One is of course, we leverage the latest research, and we actually productize that in like amazing ways where when you explain models you get the explanation within a second. So this activates new use cases like, let's say counterfactual analysis. You can not only get explanations for your loan, you can also see hypothetically. What if this the loan applicant was, you know, had a higher income? What would the model do? So, that's one part productizing latest research. The second part is infrastructure at scale. So we are not just building something that would work for SMBs. We are building something that works on enterprise scale. So billions and billions of predictions, right? Flowing through the system. We want to make sure that we can handle as larger scale as seamlessly as kind of possible. So we are trying to activate that and making sure we are the best enterprise grade product on the market. And thirdly, user experience. What you'll see when you use Fiddler. Finally, when we do demos to kind of customers what they really see is the product. They don't see that the scale right, right, right then and there. They don't see the deep reason. What they see, what they see are these like beautiful experiences that are very intuitive to them. Where we've merged explainability and monitoring and bias detection in like seamless way. So you get the most intuitive experiences that are not just designed for the technical user, but also for the non-technical user. Who are also stakeholders within AI. >> So the scale thing is a huge point, by the way. I think that's something that you see successful companies. That's a differentiator and frankly, it's the new sustainability. So new lock-in, if you will, not to be in a bad way but in a good way. You do a good job. You get scale, you get leverage. I want to just point out and get your guys' thoughts on your approach on the frame. Where you guys are centralized. >> Right. >> So as decentralization continues to be a wave you guys are taking much more of a centralized approach. Why is that done? Take us through the decision on that. >> Yeah. So, I mean, in terms of, you know decentralization in terms of running models on different you know, containers and, you know, scoring them on multiple number of nodes, that's absolutely makes sense, right? When from a deployment standpoint from a inference standpoint. But when it comes to actually you know, understanding how the models are working. Visualizing them, monitoring them, knowing what's going on with the models. You need a centralized dashboard that a lapsed user can actually use or a head of AI governance inside a bank and use what are all the models that my team is shipping? You know, which models carry risk, you know? How are these models performing last week? This, you need a centralized repository. Otherwise, it'll be very very hard to track these models, right? Because the models are going to grow really really fast. You know, there are so many open source libraries, open source model architecture has been produced. And so many data scientists coming out of grad schools and whatnot. And the number of models in enterprise is just going to grow many many fold in the coming years. Now, how are you going to track all of these things without having a centralized platform? And that's what we envisaged a few years ago that every team will need an oversight tool like Fiddler. Which can keep track of all of their models in one place. And that's what we are finding from our customers. >> As long as you don't get in the way of them creating value, which is the goal, right? >> Right. >> And be frictionless take away the friction. >> Yeah. >> And enable it. Love the concept. I think you guys are on something big there, great products. Great vision. The question I have for you to kind of wrap things up here. Is that this is all new, right? And new, it's all goodness, right? If you've got scale in the Cloud, all these new benefits. Again, more techies coming out of grad school and Computer Science and Engineering, and just data analysis in general is changing. And there's more people to be democratized to be contributing. >> Right. >> How do you operationalize it? How do companies get this going? Because you've got a new thing happening. It's a new wave. >> Okay. >> But it's still the same game, make business run better. >> Right. >> So you've got to deploy something new. What's the operational playbook for companies to get started? >> Absolutely. First step is to, if a company is trying to install AI, incorporate AI into their workflow. You know, most companies I would say, they're in still early stages, right? There a lot of enterprises are still, you know, developing these models. Some of them may have been in labs. ML operationalization is starting to happen and it probably started in a year or two ago, right? So now when it comes to, you know, putting AI into practice, so far, you know, you can have AI models in labs. They're not going to hurt anyone. They're not going to hurt your business. They're not going to hurt your users. But once you operationalize them then you have to do it in a proper manner, in a responsible manner, in a trustworthy manner. And so we actually have a playbook in terms of how you would have to do this, right? How are you going to test these models? How are you going to analyze and validate them before they actually are deployed? How are you going to analyze, you know, look into data bias and training set bias, or test set bias. And once they are deployed to production are you tracking, you know, model performance or time? Are you tracking drifting models? You know, the decay part that we talked about. Do you have alerts in place when model performance goes all over the place? Now, all of a sudden, suddenly you get a lot of false positives in your fraud models. Are you able to track them? We have the personnel in place. You have the data scientists, the ML engineers, the MLOps engineers, the governance teams in place if it's in a regulated industry to use these tools. And then, the tools like Fiddler, will add value, will make them, you know, do their job, institutionalize this process of responsible AI. So that they're not only reaping the benefits of this great technology. There's no doubt about the AI, right? It's actually, it's going to be game changing but then they can also do it in a responsible and trustworthy manner. >> Yeah, it's really get some wins, get some momentum, see it. This is the Cloud way. It gets them some value immediately and grow from there. I was talking to a friend the other day, Amit, about IT the lecture. I don't worry about IT and all the Cloud. I go, there's no longer IT, IT is dead. It's an AI department now. (Amit laughs) So and this is kind of what you guys are getting at. This now it's data now it's AI. It's kind of like what IT used to be enabling organizations to be successful. You guys are looking at it from the perspective of the same way it's enabled success. You put it out that you provision (John laughs) algorithms instead of servers they're algorithms now. This is the new model. >> Yeah, we believe that all companies in the future as it happened to this wave of data are going to be AI companies, right? So it's really just a matter of time. And the companies that are first movers in this are going to have a significant advantage like we're seeing that in like banking already. Where the banks that have made the leap into AI battles are reaping benefits of enabling a lot more models at the same risk profile using deep learning models. As long as you're able to like validate these to ensure that they're meeting kind of like the regulations. But it's going to give significant advantages to a lot of companies as they move faster with respect to others in the same industry. >> Yeah, quickers too, saw a friend too on the compliance side. You mentioned trust and transparency with the whole EU thing. Some are saying that, you know, to be a public company, you're going to have to have AI disclosure soon. You're going to have to have on the disclosure in your public statements around how you're explaining your AI. Again, fantasy today. But pretty plausible. >> Right, absolutely. I mean, the real reality today is, you know less than 10% of the CEOs care about ethical AI, right? And that has to change. And I think, you know, and I think that has to change for the better, because at the end of the day, if you are using AI, if you're not using in a responsible and trustworthy manner then there is like regulation. There is compliance risk, there's operational business risk. You know, customer trust. Losing customers trust can be huge. So I think, you know, we want to provide that you know, insurance, or like, you know like a preventative mechanism. So that, you know, if you have these tools in place then you're less likely to get into those situations. >> Awesome. Great, great conversation, Krishna, Amit. Thank you for sharing both the founders of Fiddler.ai. Great company. On the right side of history in my opinion, the next big thing in AI. AI departments, AI compliance, AI reporting. (John laughs) Explainable AI, ethical AI, all part of this next revolution. Gentlemen, thank you for joining us on theCUBE Amazon Startup Showcase. >> Thanks for having us, John. >> Okay, it's theCUBE coverage. Thank you for watching. (upbeat music)

>> Narrator: From around the globe, it's the CUBE with digital coverage of IBM Think 2021, brought to you by IBM. >> Welcome back to IBM Think 2021, the virtual edition. This is the CUBEs, continuous, deep dive coverage of the people, processes and technologies that are really changing our world. Right now, we're going to talk about modernization and what's beyond with Jamie Thomas, general manager, strategy and development, IBM Enterprise Security. Jamie, always a pleasure. Great to see you again. Thanks for coming on. >> It's great to see you, Dave. And thanks for having me on the CUBE is always a pleasure. >> Yeah, it is our pleasure. And listen, we've been hearing a lot about IBM is focused on hybrid cloud, Arvind Krishna says we must win the architectural battle for hybrid cloud. I love that. We've been hearing a lot about AI. And I wonder if you could talk about IBM Systems and how it plays into that strategy? >> Sure, well, it's a great time to have this discussion Dave. As you all know, IBM Systems Technology is used widely around the world, by many, many 1000s of clients in the context of our IBM System Z, our power systems and storage. And what we have seen is really an uptake of monetization around those workloads, if you will, driven by hybrid cloud, the hybrid cloud agenda, as well as an uptake of Red Hat OpenShift, as a vehicle for this modernization. So it's pretty exciting stuff, what we see as many clients taking advantage of OpenShift on Linux, to really modernize these environments, and then stay close, if you will, to that systems of record database and the transactions associated with it. So they're seeing a definite performance advantage to taking advantage of OpenShift. And it's really fascinating to see the things that they're doing. So if you look at financial services, for instance, there's a lot of focus on risk analytics. So things like fraud, anti money laundering, mortgage risk, types of applications being done in this context, when you look at our retail industry clients, you see also a lot of customer centricity solutions, if you will, being deployed on OpenShift. And once again, having Linux close to those traditional LPARs of AIX, I-Series, or in the context of z/OS. So those are some of the things we see happening. And it's quite real. >> Now, you didn't mention power, but I want to come back and ask you about power. Because a few weeks ago, we were prompted to dig in a little bit with the when Arvind was on with Pat Kessinger at Intel and talking about the relationship you guys have. And so we dug in a little bit, we thought originally, we said, oh, it's about quantum. But we dug in. And we realized that the POWER10 is actually the best out there and the highest performance in terms of disaggregating memory. And we see that as a future architecture for systems and actually really quite excited about it about the potential that brings not only to build beyond system on a chip and system on a package, but to start doing interesting things at the Edge. You know, what do you what's going on with power? >> Well, of course, when I talked about OpenShift, we're doing OpenShift on power Linux, as well as Z Linux, but you're exactly right in the context for a POWER10 processor. We couldn't be more we're so excited about this processor. First of all, it's our first delivery with our partner Samsung with a seven nanometer form factor. The processor itself has only 18 billion transistors. So it's got a few transistors there. But one of the cool inventions, if you will, that we have created is this expansive memory region as part of this design point, which we call memory inception, it gives us the ability to reach memory across servers, up to two petabytes of memory. Aside from that, this processor has generational improvements and core and thread performance, improved energy efficiency. And all of this, Dave is going to give us a lot of opportunity with new workloads, particularly around artificial intelligence and inferencing around artificial intelligence. I mean, that's going to be that's another critical innovation that we see here in this POWER10 processor. >> Yeah, processor performance is just exploding. We're blowing away the historical norms. I think many people don't realize that. Let's talk about some of the key announcements that you've made in quantum last time we spoke on the qubit for last year, I think we did a deeper dive on quantum. You've made some announcements around hardware and software roadmaps. Give us the update on quantum please. >> Well, there is so much that has happened since we last spoke on the quantum landscape. And the key thing that we focused on in the last six months is really an articulation of our roadmaps, so the roadmap around hardware, the roadmap around software, and we've also done quite a bit of ecosystem development. So in terms of the roadmap around hardware, we put ourselves out there we've said we were going to get to over 1000 qubit machine and in 2023, so that's our milestone. And we've got a number of steps we've outlined along that way, of course, we have to make progress, frankly, every six months in terms of innovating around the processor, the electronics and the fridge associated with these machines. So lots of exciting innovation across the board. We've also published a software roadmap, where we're articulating how we improve a circuit execution speeds. So we hope, our plan to show shortly a 100 times improvement in circuit execution speeds. And as we go forward in the future, we're modifying our Qiskit programming model to not only allow a easily easy use by all types of developers, but to improve the fidelity of the entire machine, if you will. So all of our innovations go hand in hand, our hardware roadmap, our software roadmap, are all very critical in driving the technical outcomes that we think are so important for quantum to become a reality. We've deployed, I would say, in our quantum cloud over, you know, over 20 machines over time, we never quite identify the precise number because frankly, as we put up a new generation machine, we often retire when it's older. So we're constantly updating them out there, and every machine that comes on online, and that cloud, in fact, represents a sea change and hardware and a sea change in software. So they're all the latest and greatest that our clients can have access to. >> That's key, the developer angle you got redshift running on quantum yet? >> Okay, I mean, that's a really good question, you know, as part of that software roadmap in terms of the evolution and the speed of that circuit execution is really this interesting marriage between classical processing and quantum processing and bring those closer together. And in the context of our classical operations that are interfacing with that quantum processor, we're taking advantage of OpenShift, running on that classical machine to achieve that. And once again, if, as you can imagine, that'll give us a lot of flexibility in terms of where that classical machine resides and how we continue the evolution the great marriage, I think that's going to that will exist that does exist and will exist between classical computing and quantum computing. >> I'm glad I asked it was kind of tongue in cheek. But that's a key thread to the ecosystem, which is critical to obviously, you know, such a new technology. How are you thinking about the ecosystem evolution? >> Well, the ecosystem here for quantum is infinitely important. We started day one, on this journey with free access to our systems for that reason, because we wanted to create easy entry for anyone that really wanted to participate in this quantum journey. And I can tell you, it really fascinates everyone, from high school students, to college students, to those that are PhDs. But during this journey, we have reached over 300,000 unique users, we have now over 500,000 unique downloads of our Qiskit programming model. But to really achieve that is his back plane by this ongoing educational thrust that we have. So we've created an open source textbook, around Qiskit that allows organizations around the world to take advantage of it from a curriculum perspective. We have over 200 organizations that are using our open source textbook. Last year, when we realized we couldn't do our in person programming camps, which were so exciting around the world, you can imagine doing an in person programming camp and South Africa and Asia and all those things we did in 2019. Well, we had just like you all, we had to go completely virtual, right. And we thought that we would have a few 100 people sign up for our summer school, we had over 4000 people sign up for our summer school. And so one of the things we had to do is really pedal fast to be able to support that many students in this summer school that kind of grew out of our proportions. The neat thing was once again, seeing all the kids and students around the world taking advantage of this and learning about quantum computing. And then I guess that the end of last year, Dave, to really top this off, we did something really fundamentally important. And we set up a quantum center for historically black colleges and universities, with Howard University being the anchor of this quantum center. And we're serving 23 HBCUs now, to be able to reach a new set of students, if you will, with STEM technologies, and most importantly, with quantum. And I find, you know, the neat thing about quantum is is very interdisciplinary. So we have quantum physicist, we have electrical engineers, we have engineers on the team, we have computer scientists, we have people with biology and chemistry and financial services backgrounds. So I'm pretty excited about the reach that we have with quantum into HBCUs and even beyond right I think we can do some we can have some phenomenal results and help a lot of people on this journey to quantum and you know, obviously help ourselves but help these students as well. >> What do you see in people do with quantum and maybe some of the use cases. I mean you mentioned there's sort of a connection to traditional workloads, but obviously some new territory what's exciting out there? >> Well, there's been a really a number of use cases that I think are top of mind right now. So one of the most interesting to me has been one that showed us a few months ago that we talked about in the press actually a few months ago, which is with Exxon Mobil. And they really started looking at logistics in the context of Maritime shipping, using quantum. And if you think of logistics, logistics are really, really complicated. Logistics in the face of a pandemic are even more complicated and logistics when things like the Suez Canal shuts down, are even more complicated. So think about, you know, when the Suez Canal shut down, it's kind of like the equivalent of several major airports around the world shutting down and then you have to reroute all the traffic, and that traffic and maritime shipping is has to be very precise, has to be planned the stops are plan, the routes are plan. And the interest that ExxonMobil has had in this journey is not just more effective logistics, but how do they get natural gas shipped around the world more effectively, because their goal is to bring energy to organizations into countries while reducing CO2 emissions. So they have a very grand vision that they're trying to accomplish. And this logistics operation is just one of many, then we can think of logistics, though being a being applicable to anyone that has a supply chain. So to other shipping organizations, not just Maritime shipping. And a lot of the optimization logic that we're learning from that set of work also applies to financial services. So if we look at optimization, around portfolio pricing, and everything, a lot of the similar characteristics will also go be applicable to the financial services industry. So that's one big example. And I guess our latest partnership that we announced with some fanfare, about two weeks ago, was with the Cleveland Clinic, and we're doing a special discovery acceleration activity with the Cleveland Clinic, which starts prominently with artificial intelligence, looking at chemistry and genomics, and improve speed around machine learning for all of the the critical healthcare operations that the Cleveland Clinic has embarked on but as part of that journey, they like many clients are evolving from artificial intelligence, and then learning how they can apply quantum as an accelerator in the future. And so they also indicated that they will buy the first commercial on premise quantum computer for their operations and place that in Ohio, in the the the years to come. So it's a pretty exciting relationship. These relationships show the power of the combination, once again, of classical computing, using that intelligently to solve very difficult problems. And then taking advantage of quantum for what it can uniquely do in a lot of these use cases. >> That's great description, because it is a strong connection to things that we do today. It's just going to do them better, but then it's going to open up a whole new set of opportunities. Everybody wants to know, when, you know, it's all over the place. Because some people say, oh, not for decades, other people say I think it's going to be sooner than you think. What are you guys saying about timeframe? >> We're certainly determined to make it sooner than later. Our roadmaps if you note go through 2023. And we think the 2023 is going to will be a pivotal year for us in terms of delivery around those roadmaps. But it's these kind of use cases and this intense working with these clients, 'cause when they work with us, they're giving us feedback on everything that we've done, how does this programming model really help me solve these problems? What do we need to do differently? In the case of Exxon Mobil, they've given us a lot of really great feedback on how we can better fine tune all elements of the system to improve that system. It's really allowed us to chart a course for how we think about the programming model in particular in the context of users. Just last week, in fact, we announced some new machine learning applications, which these applications are really to allow artificial intelligence users and programmers to get take advantage of quantum without being a quantum physicist or expert, right. So it's really an encapsulation of a composable elements so that they can start to use, using an interface allows them to access through PyTorch into the quantum computer, take advantage of some of the things we're doing around neural networks and things like that, once again, without having to be experts in quantum. So I think those are the kind of things we're learning how to do better, fundamentally through this co-creation and development with our quantum network. And our quantum network now is over 140 unique organizations and those are commercial, academic, national laboratories and startups that we're working with. >> The picture started become more clear, we're seeing emerging AI applications, a lot of work today in AI is in modeling. Over time, it's going to shift toward inference and real time and practical applications. Everybody talks about Moore's law being dead. Well, in fact, the yes, I guess, technically speaking, but the premise or the outcome of Moore's law is actually accelerating, we're seeing processor performance, quadrupling every two years now, when you include the GPU along with the CPU, the DSPs, the accelerators. And so that's going to take us through this decade, and then then quantum is going to power us, you know, well beyond who can even predict that. It's a very, very exciting time. Jamie, I always love talking to you. Thank you so much for coming back on the CUBE. >> Well, I appreciate the time. And I think you're exactly right, Dave, you know, we talked about POWER10, just for a few minutes there. But one of the things we've done in POWER10, as well as we've embedded AI into every core that processor, so you reduce that latency, we've got a 10 to 20 times improvement over the last generation in terms of artificial intelligence, you think about the evolution of a classical machine like that state of the art, and then combine that with quantum and what we can do in the future, I think is a really exciting time to be in computing. And I really appreciate your time today to have this dialogue with you. >> Yeah, it's always fun and it's of national importance as well. Jamie Thomas, thanks so much. This is Dave Vellante with the CUBE keep it right there our continuous coverage of IBM Think 2021 will be right back. (gentle music) (bright music)

(bright music) >> Narrator: From around the globe, it's the CUBE with digital coverage of IBM Think 2021, brought to you by IBM. >> Welcome back to IBM Think 2021, the virtual edition. This is the CUBEs, continuous, deep dive coverage of the people, processes and technologies that are really changing our world. Right now, we're going to talk about modernization and what's beyond with Jamie Thomas, general manager, strategy and development, IBM Enterprise Security. Jamie, always a pleasure. Great to see you again. Thanks for coming on. >> It's great to see you, Dave. And thanks for having me on the CUBE is always a pleasure. >> Yeah, it is our pleasure. And listen, we've been hearing a lot about IBM is focused on hybrid cloud, Arvind Krishna says we must win the architectural battle for hybrid cloud. I love that. We've been hearing a lot about AI. And I wonder if you could talk about IBM Systems and how it plays into that strategy? >> Sure, well, it's a great time to have this discussion Dave. As you all know, IBM Systems Technology is used widely around the world, by many, many 1000s of clients in the context of our IBM System Z, our power systems and storage. And what we have seen is really an uptake of monetization around those workloads, if you will, driven by hybrid cloud, the hybrid cloud agenda, as well as an uptake of Red Hat OpenShift, as a vehicle for this modernization. So it's pretty exciting stuff, what we see as many clients taking advantage of OpenShift on Linux, to really modernize these environments, and then stay close, if you will, to that systems of record database and the transactions associated with it. So they're seeing a definite performance advantage to taking advantage of OpenShift. And it's really fascinating to see the things that they're doing. So if you look at financial services, for instance, there's a lot of focus on risk analytics. So things like fraud, anti money laundering, mortgage risk, types of applications being done in this context, when you look at our retail industry clients, you see also a lot of customer centricity solutions, if you will, being deployed on OpenShift. And once again, having Linux close to those traditional LPARs of AIX, I-Series, or in the context of z/OS. So those are some of the things we see happening. And it's quite real. >> Now, you didn't mention power, but I want to come back and ask you about power. Because a few weeks ago, we were prompted to dig in a little bit with the when Arvind was on with Pat Kessinger at Intel and talking about the relationship you guys have. And so we dug in a little bit, we thought originally, we said, oh, it's about quantum. But we dug in. And we realized that the POWER10 is actually the best out there and the highest performance in terms of disaggregating memory. And we see that as a future architecture for systems and actually really quite excited about it about the potential that brings not only to build beyond system on a chip and system on a package, but to start doing interesting things at the Edge. You know, what do you what's going on with power? >> Well, of course, when I talked about OpenShift, we're doing OpenShift on power Linux, as well as Z Linux, but you're exactly right in the context for a POWER10 processor. We couldn't be more we're so excited about this processor. First of all, it's our first delivery with our partner Samsung with a seven nanometer form factor. The processor itself has only 18 billion transistors. So it's got a few transistors there. But one of the cool inventions, if you will, that we have created is this expansive memory region as part of this design point, which we call memory inception, it gives us the ability to reach memory across servers, up to two petabytes of memory. Aside from that, this processor has generational improvements and core and thread performance, improved energy efficiency. And all of this, Dave is going to give us a lot of opportunity with new workloads, particularly around artificial intelligence and inferencing around artificial intelligence. I mean, that's going to be that's another critical innovation that we see here in this POWER10 processor. >> Yeah, processor performance is just exploding. We're blowing away the historical norms. I think many people don't realize that. Let's talk about some of the key announcements that you've made in quantum last time we spoke on the qubit for last year, I think we did a deeper dive on quantum. You've made some announcements around hardware and software roadmaps. Give us the update on quantum please. >> Well, there is so much that has happened since we last spoke on the quantum landscape. And the key thing that we focused on in the last six months is really an articulation of our roadmaps, so the roadmap around hardware, the roadmap around software, and we've also done quite a bit of ecosystem development. So in terms of the roadmap around hardware, we put ourselves out there we've said we were going to get to over 1000 qubit machine and in 2023, so that's our milestone. And we've got a number of steps we've outlined along that way, of course, we have to make progress, frankly, every six months in terms of innovating around the processor, the electronics and the fridge associated with these machines. So lots of exciting innovation across the board. We've also published a software roadmap, where we're articulating how we improve a circuit execution speeds. So we hope, our plan to show shortly a 100 times improvement in circuit execution speeds. And as we go forward in the future, we're modifying our Qiskit programming model to not only allow a easily easy use by all types of developers, but to improve the fidelity of the entire machine, if you will. So all of our innovations go hand in hand, our hardware roadmap, our software roadmap, are all very critical in driving the technical outcomes that we think are so important for quantum to become a reality. We've deployed, I would say, in our quantum cloud over, you know, over 20 machines over time, we never quite identify the precise number because frankly, as we put up a new generation machine, we often retire when it's older. So we're constantly updating them out there, and every machine that comes on online, and that cloud, in fact, represents a sea change and hardware and a sea change in software. So they're all the latest and greatest that our clients can have access to. >> That's key, the developer angle you got redshift running on quantum yet? >> Okay, I mean, that's a really good question, you know, as part of that software roadmap in terms of the evolution and the speed of that circuit execution is really this interesting marriage between classical processing and quantum processing and bring those closer together. And in the context of our classical operations that are interfacing with that quantum processor, we're taking advantage of OpenShift, running on that classical machine to achieve that. And once again, if, as you can imagine, that'll give us a lot of flexibility in terms of where that classical machine resides and how we continue the evolution the great marriage, I think that's going to that will exist that does exist and will exist between classical computing and quantum computing. >> I'm glad I asked it was kind of tongue in cheek. But that's a key thread to the ecosystem, which is critical to obviously, you know, such a new technology. How are you thinking about the ecosystem evolution? >> Well, the ecosystem here for quantum is infinitely important. We started day one, on this journey with free access to our systems for that reason, because we wanted to create easy entry for anyone that really wanted to participate in this quantum journey. And I can tell you, it really fascinates everyone, from high school students, to college students, to those that are PhDs. But during this journey, we have reached over 300,000 unique users, we have now over 500,000 unique downloads of our Qiskit programming model. But to really achieve that is his back plane by this ongoing educational thrust that we have. So we've created an open source textbook, around Qiskit that allows organizations around the world to take advantage of it from a curriculum perspective. We have over 200 organizations that are using our open source textbook. Last year, when we realized we couldn't do our in person programming camps, which were so exciting around the world, you can imagine doing an in person programming camp and South Africa and Asia and all those things we did in 2019. Well, we had just like you all, we had to go completely virtual, right. And we thought that we would have a few 100 people sign up for our summer school, we had over 4000 people sign up for our summer school. And so one of the things we had to do is really pedal fast to be able to support that many students in this summer school that kind of grew out of our proportions. The neat thing was once again, seeing all the kids and students around the world taking advantage of this and learning about quantum computing. And then I guess that the end of last year, Dave, to really top this off, we did something really fundamentally important. And we set up a quantum center for historically black colleges and universities, with Howard University being the anchor of this quantum center. And we're serving 23 HBCUs now, to be able to reach a new set of students, if you will, with STEM technologies, and most importantly, with quantum. And I find, you know, the neat thing about quantum is is very interdisciplinary. So we have quantum physicist, we have electrical engineers, we have engineers on the team, we have computer scientists, we have people with biology and chemistry and financial services backgrounds. So I'm pretty excited about the reach that we have with quantum into HBCUs and even beyond right I think we can do some we can have some phenomenal results and help a lot of people on this journey to quantum and you know, obviously help ourselves but help these students as well. >> What do you see in people do with quantum and maybe some of the use cases. I mean you mentioned there's sort of a connection to traditional workloads, but obviously some new territory what's exciting out there? >> Well, there's been a really a number of use cases that I think are top of mind right now. So one of the most interesting to me has been one that showed us a few months ago that we talked about in the press actually a few months ago, which is with Exxon Mobil. And they really started looking at logistics in the context of Maritime shipping, using quantum. And if you think of logistics, logistics are really, really complicated. Logistics in the face of a pandemic are even more complicated and logistics when things like the Suez Canal shuts down, are even more complicated. So think about, you know, when the Suez Canal shut down, it's kind of like the equivalent of several major airports around the world shutting down and then you have to reroute all the traffic, and that traffic and maritime shipping is has to be very precise, has to be planned the stops are plan, the routes are plan. And the interest that ExxonMobil has had in this journey is not just more effective logistics, but how do they get natural gas shipped around the world more effectively, because their goal is to bring energy to organizations into countries while reducing CO2 emissions. So they have a very grand vision that they're trying to accomplish. And this logistics operation is just one of many, then we can think of logistics, though being a being applicable to anyone that has a supply chain. So to other shipping organizations, not just Maritime shipping. And a lot of the optimization logic that we're learning from that set of work also applies to financial services. So if we look at optimization, around portfolio pricing, and everything, a lot of the similar characteristics will also go be applicable to the financial services industry. So that's one big example. And I guess our latest partnership that we announced with some fanfare, about two weeks ago, was with the Cleveland Clinic, and we're doing a special discovery acceleration activity with the Cleveland Clinic, which starts prominently with artificial intelligence, looking at chemistry and genomics, and improve speed around machine learning for all of the the critical healthcare operations that the Cleveland Clinic has embarked on but as part of that journey, they like many clients are evolving from artificial intelligence, and then learning how they can apply quantum as an accelerator in the future. And so they also indicated that they will buy the first commercial on premise quantum computer for their operations and place that in Ohio, in the the the years to come. So it's a pretty exciting relationship. These relationships show the power of the combination, once again, of classical computing, using that intelligently to solve very difficult problems. And then taking advantage of quantum for what it can uniquely do in a lot of these use cases. >> That's great description, because it is a strong connection to things that we do today. It's just going to do them better, but then it's going to open up a whole new set of opportunities. Everybody wants to know, when, you know, it's all over the place. Because some people say, oh, not for decades, other people say I think it's going to be sooner than you think. What are you guys saying about timeframe? >> We're certainly determined to make it sooner than later. Our roadmaps if you note go through 2023. And we think the 2023 is going to will be a pivotal year for us in terms of delivery around those roadmaps. But it's these kind of use cases and this intense working with these clients, 'cause when they work with us, they're giving us feedback on everything that we've done, how does this programming model really help me solve these problems? What do we need to do differently? In the case of Exxon Mobil, they've given us a lot of really great feedback on how we can better fine tune all elements of the system to improve that system. It's really allowed us to chart a course for how we think about the programming model in particular in the context of users. Just last week, in fact, we announced some new machine learning applications, which these applications are really to allow artificial intelligence users and programmers to get take advantage of quantum without being a quantum physicist or expert, right. So it's really an encapsulation of a composable elements so that they can start to use, using an interface allows them to access through PyTorch into the quantum computer, take advantage of some of the things we're doing around neural networks and things like that, once again, without having to be experts in quantum. So I think those are the kind of things we're learning how to do better, fundamentally through this co-creation and development with our quantum network. And our quantum network now is over 140 unique organizations and those are commercial, academic, national laboratories and startups that we're working with. >> The picture started become more clear, we're seeing emerging AI applications, a lot of work today in AI is in modeling. Over time, it's going to shift toward inference and real time and practical applications. Everybody talks about Moore's law being dead. Well, in fact, the yes, I guess, technically speaking, but the premise or the outcome of Moore's law is actually accelerating, we're seeing processor performance, quadrupling every two years now, when you include the GPU along with the CPU, the DSPs, the accelerators. And so that's going to take us through this decade, and then then quantum is going to power us, you know, well beyond who can even predict that. It's a very, very exciting time. Jamie, I always love talking to you. Thank you so much for coming back on the CUBE. >> Well, I appreciate the time. And I think you're exactly right, Dave, you know, we talked about POWER10, just for a few minutes there. But one of the things we've done in POWER10, as well as we've embedded AI into every core that processor, so you reduce that latency, we've got a 10 to 20 times improvement over the last generation in terms of artificial intelligence, you think about the evolution of a classical machine like that state of the art, and then combine that with quantum and what we can do in the future, I think is a really exciting time to be in computing. And I really appreciate your time today to have this dialogue with you. >> Yeah, it's always fun and it's of national importance as well. Jamie Thomas, thanks so much. This is Dave Vellante with the CUBE keep it right there our continuous coverage of IBM Think 2021 will be right back. (gentle music) (bright music)

>> Narrator: From around the globe it's theCUBE with digital coverage of DockerCon Live 2020. Brought to you by Docker and its ecosystem partners. >> Hi, I'm Stu Miniman and welcome to theCUBE's coverage of DockerCon Live 2020. Really excited to be part of this online event. We've been involved with DockerCon for a long time, of course one of my favorite things is always to be able to talk to the practitioners. Of course we remember for years, Docker exploded onto the marketplace, millions of people downloaded it, using it. So joining me is Hui Xue, who is a Principal Deputy Director of Medical Signal Processing at the National Heart, Lung, and Blood Institute, which is part of the National Institute of Health. Hui, thank you so much for joining us. >> Thank you for inviting me. >> So let's start. Of course, the name of your institute, very specific. I think anyone in the United States knows the NIH. Tell us a little bit about your role there and kind of the scope of what your team covers. >> So I'm basically a researcher and developer of the medical imaging technology. We are the heart, lung and the blood, so we work and focus on imaging the heart. So what we exactly do is to develop the new and novel imaging technology and deploy them to the front of our clinical library, which Docker played an essential role in the process. So, yeah, that's what we do at NHLBI. >> Okay, excellent. So research, you know, of course in the medical field with the global pandemic gets a lot of attention. So you keyed it up there. Let's understand, where does containerization and Docker specifically play into the work that your team is doing? >> So, maybe I'd like to give an example which will suffice. So for example, we're working on the magnetic resonance imaging, MRI. Many of us may may already have been scanned. So we're using MRI to image the heart. What Docker plays, is Docker allow us to deploy our imaging technology to the clinical hospital. So we have a global deployment around 40 hospitals, a bit more, around the world. If we are for example develop a new AI-based image analysis for the heart image, what we do with Docker is we can put our model and software into the Docker so that our collaboration sites, they will pull the software that contains the latest technology, then use them for the patients, of course under the research agreement at NIH. Because Docker is so efficient, available globally, we can actually implement a continuous integration and testing, update the framework based on Docker. Then our collaborators would have the latest technology instead of, you know, in the traditional medical imaging in general, the iteration of technology is pretty slow. But with all this latest technology, and such like container Docker come into the field. It's actually relatively new. In the past two to three years, all these paradigm is, it's changing, certainly very exciting to us. It give us the flexibility we never had before to reach our customers, to reach other people in the world to help them. They also help us so that's a very good experience to have. >> Yeah that's pretty powerful what you're talking about there rather than you know, we install some equipment, who knows how often things get updated, how do you make sure to synchronize between different locations. Obviously the medical field highly regulated and being a government agency, talk a little bit about how you make sure you have the right version control, security is in place, how do all of those things sort out? >> Yes, that's an essential question. So firstly I want to clarify one thing. So it's not NIH who endorse Docker, it's us as researchers. We practiced Docker too and we trust its performance. This container technology is efficient, it's globally available and it's very secure. So all the communication between the container and the imaging equipment is encrypted. We also have all the paperwork it saved to set up to allow us to provide technology to our clinician. When they post the latest software, every version they put up into the Docker went through an automated integration test system. So every time they make a change, the newer version of software runs through a rigorous test, something like 200 gigabytes of data runs through and checked everything is still working. So the basic principle is we don't allow any version of the software to be delivered to customer without testing Docker. Let's say this container technology in general actually is 100% automating all this process, which actually give us a lot of freedom so we have a rather very small team here at NIH. Many people are actually very impressed by how many customer we support within this so small team. So the key reason is because we have a strongly utilized container technology, so its automation is unparalleled, certainly much better than anything I had before using this container technology. So that's actually the key to maintain the quality and the continuous service to our customers. >> Yeah, absolutely. Automation is something we've been talking about in the industry for a long time but if we implement it properly it can have a huge impact. Can you bring us inside a little bit, you know, what tools are you doing? How is that automation set up and managed? And how that fits into the Docker environment. >> So I kind of describe to be more specific. So we are using a continuous testing framework. There are several apps to be using a specific one to build on, which is an open source Python tool, rather small actually. What it can do is, this tool will set up at the service, then this service will watch for example our GitHub repo. Whenever I make a change or someone in the team makes a change for example, fix a bug, add a new feature, or maybe update a new AI model, we push the edge of the GitHub then there's a continuous building system that will notice, it will trigger the integration test run all inside Docker environment. So this is the key. What container technology offers is that we can have 100% reproducible runtime environment for our customers as the software provider, because in our particular use case we don't set up customer with the uniform hardware so they bought their own server around the world, so everyone may have slightly different hardware. We don't want that to get into our software experience. So Docker actually offers us the 100% control of the runtime environment which is very essential if we want to deliver a consistent medical imaging experience because most applications actually it's rather computational intensive, so they don't want something to run for like one minute in one site and maybe three minutes at another site. So what Docker place is that Docker will run all the integration tests. If everything pass then they pack the Docker image then send to the Docker Hub. Then all our collaborators around the world have new image then they will coordinate with them so they will find a proper time to update then they have the newer technology in time. So that's why Docker is such a useful tool for us. >> Yeah, absolutely. Okay, containerization in Docker really transformed the way a lot of those computational solutions happen. I'm wondering if you can explain a little bit more the stack that you're using if people that might not have looked at solutions for a couple of years think oh it's containers, it's dateless architectures, I'm not sure how it fits into my other network environment. Can you tell us what are you doing for the storage in the network? >> So we actually have a rather vertical integration in this medical imaging application, so we build our own service as the software, its backbone is C++ for the higher computational efficiency. There's lots of Python because these days AI model essential. What Docker provides, as I mentioned, uniform always this runtime environment so we have a fixed GCC version then if we want to go into that detail. Specific version of numerical library, certain versions of Python, will be using PyTorch a lot. So that's our AI backbone. Another way of using Docker is actually we deploy the same container into the Microsoft Azure cloud. That's another ability I found out about Docker, so we never need to change anything in our software development process, but the same container I give you must work everywhere on the cloud, on site, for our customers. This actually reduces the development cost, also improve our efficiency a lot. Another important aspect is this actually will improve customers', how do they say it, customer acceptance a lot because they go to one customer, tell them the software you are running is actually running on 30 other sites exactly the same up to the let's say heights there, so it's bit by bit consistent. This actually help us convince many people. Every time when I describe this process I think most people accept the idea. They actually appreciate the way how we deliver software to them because we always can falling back. So yes, here is another aspect. So we have many Docker images that's in the Docker Hub, so if one deployment fails, they can easily falling back. That's actually very important for medical imaging applications that fail because hospitals need to maintain their continuous level of service. So even we want to avoid this completely but yes occasionally, very occasionally, there will be some function not working or some new test case never covered before, then we give them an magnet then, falling back, that's actually also our policy and offered by the container technology. >> Yeah, absolutely. You brought up, many have said that the container is that atomic unit of building block and that portability around any platform environment. What about container orchestration? How are you managing these environments you talked about in the public cloud or in different environments? What are you doing for container orchestration? >> Actually our set-up might be the simplest case. So we basically have a private Docker repo which we paid, actually the Institute has paid. We have something like 50 or 100 private repos, then for every repo we have one specific Docker setup with different software versions of different, for example some image is for PyTorch another for TensorFlow depending on our application. Maybe some customer has the requirement to have rather small Docker image size then they have some trimmed down version of image. In this process, because it's still in a small number like 20, 30 active repo, we are actually managing it semi-automatically so we have the service running to push and pull, and loading back images but we actually configured this process here at the Institute whenever we feel we have something new to offer to the customer. Regarding managing this Docker image, it's actually another aspect for the medical image. So at the customer side, we had a lot of discussion with them for whether we want to set up a continuous automated app, but in the end they decided, they said they'd better have customers involved. Better have some people. So we were finally stopped there by, we noticed customer, there are something new to update then they will decide when to update, how to test. So this is another aspect. Even we have a very high level of confirmation using the container technology, we found it's not 100%. In some site, it's still better have human supervision to help because if the goal is to maintain 100% continuous service then in the end they need some experts on the field to test and verify. So that's how they are in the current stage of deployment of this Docker image. We found it's rather light-weight so even with a few people at NIH in our team, they can manage a rather large network globally, so it's really exciting for us. >> Excellent. Great. I guess final question, give us a little bit of a road map as to, you've already talked about leveraging AI in there, the various pieces, what are you looking for from Docker in the ecosystem, and your solution for the rest of the year? >> I would say the future definitely is on the cloud. One major direction we are trying to push is to go the clinical hospital, linking and use the cloud in building as a routine. So in current status, some of sites, hospital may be very conservative, they are afraid of the security, the connection, all kinds of issues related to cloud. But this scenario is changing rapidly, especially container technology contributes a lot on the cloud. So it makes the whole thing so easy, so reliable. So our next push is to move in lots of the application into the cloud only. So the model will be, for example, we have new AI applications. It may be only available on the cloud. If some customer is waiting to use them they will have to be willing to connect to the cloud and maybe sending data there and receive, for example, the AI apps from our running Docker image in the cloud, but what we need to do is to make the Docker building even more efficiency. Make the computation 100% stable so we can utilize the huge computational power in the cloud. Also the price, so the key here is the price. So if we have one setup in the cloud, a data center for example, we currently maintain two data centers one across Europe, another is in United States. So if we have one data center and 50 hospitals using it every day, then we need the numbers. The average price for one patient comes to a few dollars per patient. So if we consider this medical health care system the costs, the ideal costs of using cloud computing can be truly trivial, but what we can offer to patients and doctor has never happened. The computation you can bring to us is something they never saw before and they never experienced. So I believe that's the future, it's not, the old model is everyone has his own computational server, then maintaining that, it costs a lot of work. Even doctor make the software aspects much easier, but the hardware, someone still need to set-up them. But using cloud will change all of. So I think the next future is definitely to wholly utilize the cloud with the container technology. >> Excellent. Well, we thank you so much. I know everyone appreciates the work your team's doing and absolutely if things can be done to allow scalability and lower cost per patient that would be a huge benefit. Thank you so much for joining us. >> Thank you. >> All right, stay tuned for lots more coverage from theCUBE at DockerCon Live 2020. I'm Stu Miniman and thank you for watching theCUBE. (gentle music)

>> Narrator: From theCUBE Studios in Palo Alto and Boston, connecting with thought leaders all around the world, this is a CUBE conversation. >> Hello everyone, welcome to this special CUBE interview. We are here at theCUBE Virtual covering AWS Summit Virtual Online. This is Amazon's Summits that they normally do all around the world. They're doing them now virtually. We are here in the Palo Alto COVID-19 quarantine crew getting all the interviews here with a special guest, Vice President of Machine Learning, we have Swami, CUBE Alumni, who's been involved in not only the machine learning, but all of the major activity around AWS around how machine learning's evolved, and all the services around machine learning workflows from transcribe, recognition, you name it. Swami, you've been at the helm for many years, and we've also chatted about that before. Welcome to the virtual CUBE covering AWS Summit. >> Hey, pleasure to be here, John. >> Great to see you. I know times are tough. Everything okay at Amazon? You guys are certainly cloud scaled, not too unfamiliar of working remotely. You do a lot of travel, but what's it like now for you guys right now? >> We're actually doing well. We have been I mean, this many of, we are working hard to make sure we continue to serve our customers. Even from their site, we have done, yeah, we had taken measures to prepare, and we are confident that we will be able to meet customer demands per capacity during this time. So we're also helping customers to react quickly and nimbly, current challenges, yeah. Various examples from amazing startups working in this area to reorganize themselves to serve customer. We can talk about that common layer. >> Large scale, you guys have done a great job and fun watching and chronicling the journey of AWS, as it now goes to a whole 'nother level with the post pandemic were expecting even more surge in everything from VPNs, workspaces, you name it, and all these workloads are going to be under a lot of pressure to do more and more value. You've been at the heart of one of the key areas, which is the tooling, and the scale around machine learning workflows. And this is where customers are really trying to figure out what are the adequate tools? How do my teams effectively deploy machine learning? Because now, more than ever, the data is going to start flowing in as virtualization, if you will, of life, is happening. We're going to be in a hybrid world with life. We're going to be online most of the time. And I think COVID-19 has proven that this new trajectory of virtualization, virtual work, applications are going to have to flex, and adjust, and scale, and be reinvented. This is a key thing. What's going on with machine learning, what's new? Tell us what are you guys doing right now. >> Yeah, I see now, in AWS, we offer broadest-- (poor audio capture obscures speech) All the way from like expert practitioners, we offer our frameworks and infrastructure layer support for all popular frameworks from like TensorFlow, Apache MXNet, and PyTorch, PowerShell, (poor audio capture obscures speech) custom chips like inference share. And then, for aspiring ML developers, who want to build their own custom machine learning models, we're actually building, we offer SageMaker, which is our end-to-end machine learning service that makes it easy for customers to be able to build, train, tune, and debug machine learning models, and it is one of our fastest growing machine learning services, and many startups and enterprises are starting to standardize their machine learning building on it. And then, the final tier is geared towards actually application developers, who did not want to go into model-building, just want an easy API to build capabilities to transcribe, run voice recognition, and so forth. And I wanted to talk about one of the new capabilities we are about to launch, enterprise search called Kendra, and-- >> So actually, so just from a news standpoint, that's GA now, that's being announced at the Summit. >> Yeah. >> That was a big hit at re:Invent, Kendra. >> Yeah. >> A lot of buzz! It's available. >> Yep, so I'm excited to say that Kendra is our new machine learning powered, highly accurate enterprise search service that has been made generally available. And if you look at what Kendra is, we have actually reimagined the traditional enterprise search service, which has historically been an underserved market segment, so to speak. If you look at it, on the public search, on the web search front, it is a relatively well-served area, whereas the enterprise search has been an area where data in enterprise, there are a huge amount of data silos, that is spread in file systems, SharePoint, or Salesforce, or various other areas. And deploying a traditional search index has always that even simple persons like when there's an ID desk open or when what is the security policy, or so forth. These kind of things have been historically, people have to find within an enterprise, let alone if I'm actually in a material science company or so forth like what 3M was trying to do. Enable collaboration of researchers spread across the world, to search their experiment archives and so forth. It has been super hard for them to be able to things, and this is one of those areas where Kendra has enabled the new, of course, where Kendra is a deep learning powered search service for enterprises, which breaks down data silos, and collects actually data across various things all the way from S3, or file system, or SharePoint, and various other data sources, and uses state-of-art NLP techniques to be able to actually index them, and then, you can query using natural language queries such as like when there's my ID desk-scoping, and the answer, it won't just give you a bunch of random, right? It'll tell you it opens at 8:30 a.m. in the morning. >> Yeah. >> Or what is the credit card cashback returns for my corporate credit card? It won't give you like a long list of links related to it. Instead it'll give you answer to be 2%. So it's that much highly accurate. (poor audio capture obscures speech) >> People who have been in the enterprise search or data business know how hard this is. And it is super, it's been a super hard problem, the old in the old guard models because databases were limiting to schemas and whatnot. Now, you have a data-driven world, and this becomes interesting. I think the big takeaway I took away from Kendra was not only the new kind of discovery navigation that's possible, in terms of low latency, getting relevant content, but it's really the under-the-covers impact, and I think I'd like to get your perspective on this because this has been an active conversation inside the community, in cloud scale, which is data silos have been a problem. People have had built these data silos, and they really talk about breaking them down but it's really again hard, there's legacy problems, and well, applications that are tied to them. How do I break my silos down? Or how do I leverage either silos? So I think you guys really solve a problem here around data silos and scale. >> Yeah. >> So talk about the data silos. And then, I'm going to follow up and get your take on the kind of size of of data, megabytes, petabytes, I mean, talk about data silos, and the scale behind it. >> Perfect, so if you look at actually how to set up something like a Kendra search cluster, even as simple as from your Management Console in the AWS, you'll be able to point Kendra to various data sources, such as Amazon S3, or SharePoint, and Salesforce, and various others. And say, these are kind of data I want to index. And Kendra automatically pulls in this data, index these using its deep learning and NLP models, and then, automatically builds a corpus. Then, I, as in user of the search index, can actually start querying it using natural language, and don't have to worry where it comes from, and Kendra takes care of things like access control, and it uses finely-tuned machine learning algorithms under the hood to understand the context of natural language query and return the most relevant. I'll give a real-world example of some of the field customers who are using Kendra. For instance, if you take a look at 3M, 3M is using Kendra to support search, support its material science R&D by enabling natural language search of their expansive repositories of past research documents that may be relevant to a new product. Imagine what this does to a company like 3M. Instead of researchers who are spread around the world, repeating the same experiments on material research over and over again, now, their engineers and researchers will allow everybody to quickly search through documents. And they can innovate faster instead of trying to literally reinvent the wheel all the time. So it is better acceleration to the market. Even we are in this situation, one of the interesting work that you might be interested in is the Semantic Scholar team at Allen Institute for AI, recently opened up what is a repository of scientific research called COVID-19 Open Research Dataset. These are expert research articles. (poor audio capture obscures speech) And now, the index is using Kendra, and it helps scientists, academics, and technologists to quickly find information in a sea of scientific literature. So you can even ask questions like, "Hey, how different is convalescent plasma "treatment compared to a vaccine?" And various in that question and Kendra automatically understand the context, and gets the summary answer to these questions for the customers, so. And this is one of the things where when we talk about breaking the data silos, it takes care of getting back the data, and putting it in a central location. Understanding the context behind each of these documents, and then, being able to also then, quickly answer the queries of customers using simple query natural language as well. >> So what's the scale? Talk about the scale behind this. What's the scale numbers? What are you guys seeing? I see you guys always do a good job, I've run a great announcement, and then following up with general availability, which means I know you've got some customers using it. What are we talking about in terms of scales? Petabytes, can you give some insight into the kind of data scale you're talking about here? >> So the nice thing about Kendra is it is easily linearly scalable. So I, as a developer, I can keep adding more and more data, and that is it linearly scales to whatever scale our customers want. So and that is one of the underpinnings of Kendra search engine. So this is where even if you see like customers like PricewaterhouseCoopers is using Kendra to power its regulatory application to help customers search through regulatory information quickly and easily. So instead of sifting through hundreds of pages of documents manually to answer certain questions, now, Kendra allows them to answer natural language question. I'll give another example, which is speaks to the scale. One is Baker Tilly, a leading advisory, tax, and assurance firm, is using Kendra to index documents. Compared to a traditional SharePoint-based full-text search, now, they are using Kendra to quickly search product manuals and so forth. And they're able to get answers up to 10x faster. Look at that kind of impact what Kendra has, being able to index vast amount of data, with in a linearly scalable fashion, keep adding in the order of terabytes, and keep going, and being able to search 10x faster than traditional, I mean traditional keyword search based algorithm is actually a big deal for these customers. They're very excited. >> So what is the main problem that you're solving with Kendra? What's the use case? If I'm the customer, what's my problem that you're solving? Is it just response to data, whether it's a call center, or support, or is it an app? I mean, what's the main focus that you guys came out? What was the vector of problem that you're solving here? >> So when we talked to customers before we started building Kendra, one of the things that constantly came back for us was that they wanted the same ease of use and the ability to search the world wide web, and customers like us to search within an enterprise. So it can be in the form of like an internal search to search within like the HR documents or internal wiki pages and so forth, or it can be to search like internal technical documentation or the public documentation to help the contact centers or is it the external search in terms of customer support and so forth, or to enable collaboration by sharing knowledge base and so forth. So each of these is really dissected. Why is this a problem? Why is it not being solved by traditional search techniques? One of the things that became obvious was that unlike the external world where the web pages are linked that easily with very well-defined structure, internal world is very messy within an enterprise. The documents are put in a SharePoint, or in a file system, or in a storage service like S3, or on naturally, tell-stores or Box, or various other things. And what really customers wanted was a system which knows how to actually pull the data from various these data silos, still understand the access control behind this, and enforce them in the search. And then, understand the real data behind it, and not just do simple keyword search, so that we can build remarkable search service that really answers queries in a natural language. And this has been the theme, premise of Kendra, and this is what had started to resonate with our customers. I talked with some of the other examples even in areas like contact centers. For instance, Magellan Health is using Kendra for its contact centers. So they are able to seamlessly tie like member, provider, or client specific information with other inside information about health care to its agents so that they can quickly resolve the call. Or it can be on internally to do things like external search as well. So very satisfied client. >> So you guys took the basic concept of discovery navigation, which is the consumer web, find what you're looking for as fast as possible, but also took advantage of building intelligence around understanding all the nuances and configuration, schemas, access, under the covers and allowing things to be discovered in a new way. So you basically makes data be discoverable, and then, provide an interface. >> Yeah. >> For discovery and navigation. So it's a broad use cat, then. >> Right, yeah that's sounds somewhat right except we did one thing more. We actually understood not just, we didn't just do discovery and also made it easy for people to find the information but they are sifting through like terabytes or hundreds of terabytes of internal documentation. Sometimes, one other things that happens is throwing a bunch of hundreds of links to these documents is not good enough. For instance, if I'm actually trying to find out for instance, what is the ALS marker in an health care setting, and for a particular research project, then, I don't want to actually sift through like thousands of links. Instead, I want to be able to correctly pinpoint which document contains answer to it. So that is the final element, which is to really understand the context behind each and every document using natural language processing techniques so that you not only find discover the information that is relevant but you also get like highly accurate possible precise answers to some of your questions. >> Well, that's great stuff, big fan. I was really liking the announcement of Kendra. Congratulations on the GA of that. We'll make some room on our CUBE Virtual site for your team to put more Kendra information up. I think it's fascinating. I think that's going to be the beginning of how the world changes, where this, this certainly with the voice activation and API-based applications integrating this in. I just see a ton of activity that this is going to have a lot of headroom. So appreciate that. The other thing I want to get to while I have you here is the news around the augmented artificial intelligence has been brought out as well. >> Yeah. >> So the GA of that is out. You guys are GA-ing everything, which is right on track with your cadence of AWS laws, I'd say. What is this about? Give us the headline story. What's the main thing to pay attention to of the GA? What have you learned? What's the learning curve, what's the results? >> So augmented artificial intelligence service, I called it A2I but Amazon A2I service, we made it generally available. And it is a very unique service that makes it easy for developers to augment human intelligence with machine learning predictions. And this is historically, has been a very challenging problem. We look at, so let me take a step back and explain the general idea behind it. You look at any developer building a machine learning application, there are use cases where even actually in 99% accuracy in machine learning is not going to be good enough to directly use that result as the response to back to the customer. Instead, you want to be able to augment that with human intelligence to make sure, hey, if my machine learning model is returning, saying hey, my confidence interval for this prediction is less than 70%, I would like it to be augmented with human intelligence. Then, A2I makes it super easy for customers to be, developers to use actually, a human reviewer workflow that comes in between. So then, I can actually send it either to the public pool using Mechanical Turk, where we have more than 500,000 Turkers, or I can use a private workflow as a vendor workflow. So now, A2I seamlessly integrates with our Textract, Rekognition, or SageMaker custom models. So now, for instance, NHS is integrated A2I with Textract, so that, and they are building these document processing workflows. The areas where the machine learning model confidence load is not as high, they will be able augment that with their human reviewer workflows so that they can actually build in highly accurate document processing workflow as well. So this, we think is a powerful capability. >> So this really kind of gets to what I've been feeling in some of the stuff we worked with you guys on our machine learning piece. It's hard for companies to hire machine learning people. This has been a real challenge. So I like this idea of human augmentation because humans and machines have to have that relationship, and if you build good abstraction layers, and you abstract away the complexity, which is what you guys do, and that's the vision of cloud, then, you're going to need to have that relationship solidified. So at what point do you think we're going to be ready for theCUBE team, or any customer that doesn't have the or can't find a machine learning person? Or may not want to pay the wages that's required? I mean it's hard to find a machine learning engineer, and when does the data science piece come in with visualization, the spectrum of pure computer science, math, machine learning guru to full end user productivity? Machine learning is where you guys are doing a lot of work. Can you just share your opinion on that evolution of where we are on that? Because people want to get to the point where they don't have to hire machine learning folks. >> Yeah. >> And have that kind support too. >> If you look at the history of technology, I actually always believe that many of these highly disruptive technology started as a way that it is available only to experts, and then, they quickly go through the cycles, where it becomes almost common place. I'll give an example with something totally outside the IT space. Let's take photography. I think more than probably 150 years ago, the first professional camera was invented, and built like three to four years still actually take a really good picture. And there were only very few expert photographers in the world. And then, fast forward to time where we are now, now, even my five-year-old daughter takes actually very good portraits, and actually gives it as a gift to her mom for Mother's Day. So now, if you look at Instagram, everyone is a professional photographer. I kind of think the same thing is about to, it will happen in machine learning too. Compared to 2012, where there were very few deep learning experts, who can really build these amazing applications, now, we are starting to see like tens of thousands of actually customers using machine learning in production in AWS, not just proof of concepts but in production. And this number is rapidly growing. I'll give one example. Internally, if you see Amazon, to aid our entire company to transform and make machine learning as a natural part of the business, six years ago, we started a Machine Learning University. And since then, we have been training all our engineers to take machine learning courses in this ML University, and a year ago, we actually made these coursework available through our Training and Certification platform in AWS, and within 48 hours, more than 100,000 people registered. Think about it, that's like a big all-time record. That's why I always like to believe that developers are always eager to learn, they're very hungry to pick up new technology, and I wouldn't be surprised if four or five years from now, machine learning is kind of becomes a normal feature of the app, the same with databases are, and that becomes less special. If that day happens, then, I would see it as my job is done, so. >> Well, you've got a lot more work to do because I know from the conversations I've been having around this COVID-19 pandemic is it's that there's general consensus and validation that the future got pulled forward, and what used to be an inside industry conversation that we used to have around machine learning and some of the visions that you're talking about has been accelerated on the pace of the new cloud scale, but now that people now recognize that virtual and experiencing it firsthand globally, everyone, there are now going to be an acceleration of applications. So we believe there's going to be a Cambrian explosion of new applications that got to reimagine and reinvent some of the plumbing or abstractions in cloud to deliver new experiences, because the expectations have changed. And I think one of the things we're seeing is that machine learning combined with cloud scale will create a whole new trajectory of a Cambrian explosion of applications. So this has kind of been validated. What's your reaction to that? I mean do you see something similar? What are some of the things that you're seeing as we come into this world, this virtualization of our lives, it's every vertical, it's not one vertical anymore that's maybe moving faster. I think everyone sees the impact. They see where the gaps are in this new reality here. What's your thoughts? >> Yeah, if you see the history from machine learning specifically around deep learning, while the technology is really not new, especially because the early deep learning paper was probably written like almost 30 years ago. And why didn't we see deep learning take us sooner? It is because historically, deep learning technologies have been hungry for computer resources, and hungry for like huge amount of data. And then, the abstractions were not easy enough. As you rightfully pointed out that cloud has come in made it super easy to get like access to huge amount of compute and huge amount of data, and you can literally pay by the hour or by the minute. And with new tools being made available to developers like SageMaker and all the AI services, we are talking about now, there is an explosion of options available that are easy to use for developers that we are starting to see, almost like a huge amount of like innovations starting to pop up. And unlike traditional disruptive technologies, which you usually see crashing in like one or two industry segments, and then, it crosses the chasm, and then goes mainstream, but machine learning, we are starting to see traction almost in like every industry segment, all the way from like in financial sector, where fintech companies like Intuit is using it to forecast its call center volume and then, personalization. In the health care sector, companies like Aidoc are using computer vision to assist radiologists. And then, we are seeing in areas like public sector. NASA has partnered with AWS to use machine learning to do anomaly detection, algorithms to detect solar flares in the space. And yeah, examples are plenty. It is because now, machine learning has become such common place that and almost every industry segment and every CIO is actually already looking at how can they reimagine, and reinvent, and make their customer experience better covered by machine learning. In the same way, Amazon actually asked itself, like eight or 10 years ago, so very exciting. >> Well, you guys continue to do the work, and I agree it's not just machine learning by itself, it's the integration and the perfect storm of elements that have come together at this time. Although pretty disastrous, but I think ultimately, it's going to come out, we're going to come out of this on a whole 'nother trajectory. It's going to be creativity will be emerged. You're going to start seeing really those builders thinking, "Okay hey, I got to get out there. "I can deliver, solve the gaps we are exposed. "Solve the problems, "pre-create new expectations, new experience." I think it's going to be great for software developers. I think it's going to change the computer science field, and it's really bringing the lifestyle aspect of things. Applications have to have a recognition of this convergence, this virtualization of life. >> Yeah. >> The applications are going to have to have that. So and remember virtualization helped Amazon formed the cloud. Maybe, we'll get some new kinds of virtualization, Swami. (laughs) Thanks for coming on, really appreciate it. Always great to see you. Thanks for taking the time. >> Okay, great to see you, John, also. Thank you, thanks again. >> We're with Swami, the Vice President of Machine Learning at AWS. Been on before theCUBE Alumni. Really sharing his insights around what we see around this virtualization, this online event at the Amazon Summit, we're covering with the Virtual CUBE. But as we go forward, more important than ever, the data is going to be important, searching it, finding it, and more importantly, having the humans use it building an application. So theCUBE coverage continues, for AWS Summit Virtual Online, I'm John Furrier, thanks for watching. (enlightening music)

(lively music) >> Narrator: Live, from San Francisco, celebrating 10 years of high tech coverage, it's theCUBE, covering VMworld 2019! Brought to you by VMware and its ecosystem partners. >> Okay, welcome back to theCUBE's live coverage in VMworld 2019. We're in San Francisco. We're in Moscone North Lobby. I'm John Frer, my co Stu Miniman, here covering all the action of VMworld, two sets for theCUBE, our tenth year, Stu. Keeping it going. Two great guests, John Fanelli, CUBE Alumni, Vice President of Product, Virtual GPUs at NVIDIA Kevin Gray, Director of Product Marketing, Dell EMC. Thanks for coming back on. Good to see you. >> Awesome. >> Good to see you guys, too. >> NVIDIA, big news, we saw your CEO up on the keynote videoing in. Two big announcements. You got some stats on some Windows stats to talk about. Let's talk about the news first, get the news out of the way. >> Sure, at this show, NVIDIA announced our new product called NVIDIA Virtual Compute Server. So for the very first time anywhere, we're able to virtualize artificial intelligence, deep learning, machine learning, and data analytics. Of course, we did that in conjunction with our partner, VMware. This runs on top of vSphere and also in conjunction with our partner at Dell. All of this Virtual Compute Server runs on Dell VxRail, as well. >> What's the impact going to be for that? What does that mean for the customers? >> For customers, it's really going to be the on-ramp for Enterprise AI. A lot of customers, let's say they have a team of maybe eight data scientists are doing data analytics, if they want to move through GPU today, they have to buy eight GPUs. However, with our new solution, maybe they start with two GPUs and put four users on a GPU. Then as their models get bigger and their data gets bigger, they move to one user per GPU. Then ultimately, because we support multiple GPUs now as part of this, they move to a VM that has maybe four GPUs in it. We allow the enterprise to start to move on to AI and deep learning, in particular, machine learning for data analytics very easily. >> GPUs are in high demand. My son always wants the next NVIDIA, in part told me to get some GPUs from you when you came on. Ask the NVIDIA guy to get some for his gaming rig. Kidding aside, now in the enterprise, really important around some of the data crunching, this has really been a great use case. Talk about how that's changed, how people think about it, and how it's impacted traditional enterprise. >> From a data analytics perspective, the data scientists will ingest data, they'll run some machine learning on it, they'll create an inference model that they run to drive predictive business decisions. What we've done is we've GPU-accelerated the key libraries, the technologies, like PyTorch, XGBoost to use a GPU. The first announcement is about how they can now use Virtual Compute Server to do that. The second announcement is that workflow is, as I mentioned, they'll start small, and then they'll do bigger models, and eventually they want to train that scale. So what they want to do is they want to move to the cloud so they can have hundreds or thousands of GPUs. The second announcement is that NVIDIA and VMware are bringing Virtual Compute Server to VMware Cloud running on AWS with our T4 GPUs. So now I can scale virtually starting with fractional GPU to single GPU to multi GPU, and push a button with HCX and move it directly into AWS T4 accelerated cloud. >> That's the roadmap so you can get in, get the work done, scale up, that's the benefit of that. Availability, timing, when all of this is going to hit in-- >> So Virtual Compute Server is available on Friday, the 29th. We're looking at mid next year for the full suite of VMware Cloud on top of Aws T4. >> Kevin, you guys are supplier here at Dell EMC. What's the positioning there with you guys? >> We're working very closely with NVIDIA in general on all of their efforts around both AI as well as VDI too. We'll work quite a bit, most recently on the VDI front as well. We look to drive things like qualifying the devices. There's both VDI or analytics applications. >> Kevin, bring us up-to-date 'cause it's funny we were talking about this is our 10th year here at the show. I remember sitting across Howard Street here in 2010 and Dell, and HP, and IBM all claiming who had the lowest dollar per desktop as to what they were doing in VDI. It's a way different discussion here in 2019. >> Absolutely. Go ahead. >> One of the things that we've learned with NVIDIA is that it's really about the user experience. It's funny we're at a transition point now from Windows 7 to Windows 10. The last transition was Windows XP to Windows 7. What we did then is we took Windows 7, we tore everything out of it we possibly could, we made it look like XP, and we shoved it out. 10 years later, that doesn't work. Everyone's got their iPhones, their iOS devices, their Android devices. Microsoft's done a great job on Windows 10 being immersive. Now we're focused on user experience. When the VDI environment, as you move to Windows 10, you may not be aware of this, but from Windows 7 to Windows 10, it uses 50% more CPU, and you don't even get that great of a user experience. You pop a GPU in there, and you're good. Most of our customers together are working on a five-year life cycle. That means over the next five years, they're going to get 10 updates of Windows 10, and they're going to get like 60 updates of their Office applications. That means that they want to be future-proof now by putting the GPUs in to guarantee a great user experience. >> On the performance side too, obviously. In auto updates, this is the push notification world we live in. This has to built in from day one. >> Absolutely, and if you look at what Dell's doing, we really built this into both our VxRails and our VxBlocks. GPUs are just now part of it. We do these fully qualified. It stacks specifically for VDI environments as well. We're working a lot with the n-vector tools from VM which makes sure we're-- >> VDI finally made it! >> qualifying user experience. >> All these years. >> Yes, yes. In fact, we have this user experience tool called n-vector, which actually, without getting super technical for the audience, it allows you to look at the user experience based on frame-rate, latency, and image quality. We put this tool together, but Dell has really been taking a lead on testing it and promoting it to the users to really drive the cost-effectiveness. It still is about the dollar per desktop, but it's the dollar per dazzling desktop. (laughing) >> Kevin, I hear the frame-rate in there, and I've got all the remote workers, and you're saying how do I make sure that's not the gaming platform they're using because I know how important that is. >> Absolutely. There's a ton of customers that are out there that we're using. We look at folks like Guillevin as like the example of a company that's worked with us and NVIDIA to truly drive types of applications that are essential to VDI. These types of power workers doing applications like Autodesk, that user experience and that ability to support multiple users. If you look at Pat, he talked a little bit about any cloud, any application, any device. In VDI, that's really what it's about, allowing those workers to come together. >> I think the thing that the two of you mentioned, and Stu you pointed out brilliantly was that VDI is not just an IT thing anymore. It really is the expectation now that my rig, if I'm a gamer, or a young person, the younger kids, if you're under 25, if you don't have a kick-ass rig, (laughs) that's what they call it. Multiple monitors, that's the expectation, again, mobility. Work experience, workspace. >> Exactly, along those same lines, by the way. >> This is the whole category. It's not just like a VDI, this thing over here that used to be talked about as an IT thing. >> It's about the workflow. So it's how do I get my job done. We used to use words like "business worker" and "knowledge worker." It's just I'm a worker. Everybody today uses their phone that's mobile. They use their computer at home, they use their computer at work. They're all running with dual monitors. Dual monitors, sometimes dual 4K monitors. That really benefits as well from having a GPU. I know we're on TV so hopefully some of you guys are watching VDI on your GPU-accelerated. It's things like Skype, WebEX, Zoom, all the collaboration to 'em, Microsoft Teams, they all benefit from our joint solution, like the GPU. >> These new subsystems like GPUs become so critical. They're not just subsystem, they are the main part because the offload is now part of the new operating environment. >> We optimized together jointly using the n-vector tool. We optimized the server and operating environment, so that if you run into GPU, you can right-size your CPU in terms of cores, speed, etc., so that you get the best user experience at a most cost effective way. >> Also, the gaming world helps bring in the new kind of cool visualization. That's going to move into just the workflow of workers. You start to see this immersive experience, VR, ARs obviously around the corner. It's only going to get more complex, more needs for GPUs. >> Yes, in fact, we're seeing more, I think, requirements for AR and VR from business than we are actually for gaming. Don't you want to go into your auto showroom at your house and feel the fine Corinthian leather? >> We got to upgrade our CUBE game, get more GPU focused and get some tracing in there. >> Kevin, I know I've seen things from the Dell family on levering VR in the enterprise space. >> Oh, absolutely. If you look at a lot of the things that we're doing with some of the telcos around 5G. They're very interested in VR and AR. Those are areas that'll continue to use things like GPUs to help accelerate those types of applications. It really does come down to having that scalable infrastructure that's easy to manage and easy to operate. That's where I think the partnership with NVIDIA really comes together. >> Deep learning and all this stuff around data. Michael Dell always comes on theCUBE, talks about it. He sees data as the biggest opportunity and challenge. In whatever applications coming in, you got to be able to pound into that data. That's where AI's really shown... Machine learning has kind of shown that that's helping heavy lifting a lot of things that were either manual. >> Exactly. The one thing that's really great about data analytics that are GPU-accelerated is we can take a job that used to take days and bring it down to hours. Obviously, doing something faster is great, but if I take a job that used to take a week and I can do it in one day, that means I have four more days to do other things. It's almost like I'm hiring people for free because I get four more extra work days. The other thing that's really interesting as our joint solution is you can leverage that same virtual GPU technology. You can do VDI by day and at night, you run Compute. So when your users aren't at work, you migrate them off, you spin up your VMs that are doing your data analytics using our RAPIDS technology, and then you're able to get that platform running 24 by seven. >> Productivity gains just from an infrastructure. Even the user too, up and down, the productivity gains are significant. So I'll get three monitors now. I'm going to get one of those Alienware curved monitors. >> Just the difference we had, we have a suite here at the show, and just the difference, you can see such a difference when you insert the GPUs into the platform. It's just makes all the difference. >> John, I got to ask you a personal question. How many times have people asked you for a GPU? You must get that all the time? >> We do. I have a NVIDIA backpack. When I walk around, there's a lot of people that only know NVIDIA for games. So random people will always ask for that. >> I've got two sons and two daughters and they just nerd out on the GPUs. >> I think he's trying to get me to commit on camera on giving him a GPU. (laughing) I think I'm in trouble here. >> Yeah, they get the latest and greatest. Any new stuff, they're going to be happy to be the first on the block to get the GPU. It's certainly impacted on the infrastructure side, the components, the operating environment, Windows 10. Any other data you guys have to share that you think is notable around how all this is coming together working from user experience around Windows and VDI? >> I think one piece of data, again, going back to your first comment about cost per desktop. We're seeing a lot of migration to Windows 10. Customers are buying our joint solution from Dell which includes our hardware and software. They're buying that five-year life cycle, so we actually put a program in place to really drive down the cost. It's literally like $3 per month to have a GPU-accelerated virtual desktop. It's really great Value for the customers besides the great productivity. >> If you look at doing some of these workloads on premises, some of the costs can come down. We had a recent study around the VxBlock as an example. We showed that running GPUs and VDI can be up as much as 45% less on a VxBlock at scale. When you talk about the whole hybrid cloud, multi-cloud strategy, there's pluses and minuses to both. Certainly, if we look at some of the ability to start small and scale out, whether you're going HCI or you're going CI, I think there's a VDI solution there that can really drive the economics. >> The intense workloads. Is there any industries that are key for you guys in terms of verticals? >> Absolutely. So we're definitely looking at a lot of the CAD/CAM industries. We just did a certification on our platforms with Dassault's CATIA system. That's an area that we'll continue to explore as we move forward. >> I think in the workstation side of things, it's all the standard, it's automotive, it's manufacturing. Architecture is interesting. Architecture is one of those companies that has kind of an S and B profile. They have lots of offices, but they have enterprise requirements for all the hard work that they do. Then with VDI, we're very strong in financial services as well as healthcare. In fact, if you haven't seen, you should come by. We have a Bloomberg demo for financial services about the impact for traders. I have a virtualized GPU desktop. >> The speed is critical for them. Final question. Take-aways from the show this year, 2019 VMworld, Stu, we got 10 years to look back, but guys, take-aways from the show that you're going to take back from this week. >> I think there's still a lot of interest and enthusiasm. Surprisingly, there's still a lot of customers that haven't finished there migration to Windows 10 and they're coming to us saying, Oh my gosh, I only have until January, what can you do to help me? (laughing) >> Get some GPUs. Thoughts from the show. >> The multi-cloud world continues to evolve, the continued partnerships that emerge as part of this is just pretty amazing in how that's changing in things like virtual GPUs and accelerators. That experience that people have come to expect from the cloud is something, for me is a take-away. >> John Fanelli, NVIDIA, thanks for coming on. Congratulations on all the success. Kevin, Dell EMC, thanks for coming on. >> Thank you. >> Thanks for the insights. Here on theCUBE, Vmworld 2019. John Furrier, Stu Miniman, stay with us for more live coverage after this short break. (lively music)

>> From the Silicon Angle Media office in Boston, Massachusetts, It's theCUBE. (upbeat music) Now, here's your host, Stu Miniman. >> Hi. I'm Stu Miniman, and this is theCUBE's Boston area studio; our actually brand-new studio, and I'm really excited to have I believe is a first-time guest, a long-time caller, you know, a long time listener >> Yeah, yep. first time caller, good buddy of mine Dave Graham, who is the director, is a director of emerging technologies: messaging at Dell Technologies. Disclaimer, Dave and I worked together at a company some of you might have heard on the past, it was EMC Corporation, which was a local company. Dave and I both left EMC, and Dave went back, after Dell had bought EMC. So Dave, thanks so much for joining, it is your first time on theCUBE, yes? >> It is the first time on theCUBE. >> Yeah, so. >> Lets do some, Some of the first times that I actually interacted with, with this team here, you and I were bloggers and doing lots of stuff back in the industry, so it's great to be able to talk to you on-camera. >> Yeah, same here. >> All right, so Dave, I mentioned you were a returning former EMC-er, now Dell tech person, and you spent some time at Juniper, at some startups, but give our audience a little bit about your background and your passions. >> Oh, so background-wise, yep, so started my career in technology, if you will, at EMC, worked, started in inside sales of all places. Worked my way into a consulting/engineer type position within ECS, which was, obviously a pretty hard-core product inside of EMC now, or Dell Technologies now. Left, went to a startup, everybody's got to do a start up at some point in their life, right? Take the risk, make the leap, that was awesome, was actually one of those Cloud brokers that's out there, like Nasuni, company called Sertis. Had a little bit of trouble about eight months in, so it kind of fell apart. >> Yeah, the company did, not you. >> The company did! (men laughing) I was fine, you know, but the, yeah, the company had some problems, but ended up leaving there, going to Symantec of all places, so I worked on the Veritas side, kind of the enterprise side, which just recently got bought out by Avago, evidently just. >> Broadcom >> Broadcom, Broadcom, art of the grand whole Avago. >> Dave, Dave, you know we're getting up there in years and our tech, when we keep talking about something 'cause I was just reading about, right, Broadcom, which was of course Avago bought Broadcom in the second largest tech acquisition in history, but when they acquired Broadcom, they took on the name because most people know Broadcom, not as many people know Avago, even those of us with backgrounds in the chip semiconductor and all those pieces. I mean you got Brocade in there, you've got some of the software companies that they've bought over the time, so some of those go together. But yeah, Veritas and Symantec, those of us especially with some storage and networking background know those brands well. >> Absolutely, PLX's being the PCI switched as well, it's actually Broadcom, those things. So yeah, went from Symantec after a short period of time there, went to Juniper Networks, ran part of their Center of Excellence, kind of a data center overlay team, the only non-networking guy in a networking company, it felt like. Can't say that I learned a ton about the networking side, but definitely saw a huge expansion in the data center space with Juniper, which was awesome to see. And then the opportunity came to come back to Dell Technologies. Kind of a everything old becoming new again, right? Going and revisiting a whole bunch of folks that I had worked with 13, you know, 10 years ago. >> Dave, it's interesting, you know, I think about, talk about somebody like Broadcom, and Avago, and things like that. I remember reading blog posts of yours, that you'd get down to some of that nitty-level, you and I would be ones that would be the talk about the product, all right now pull the board out, let me look at all the components, let me understand, you know, the spacing, and the cooling, and all the things there, but you know here it's 2019, Dave. Don't you know software is eating the world? So, tell us a little bit about what you're working on these days, because the high-level things definitely don't bring to mind the low-level board pieces that we used to talk about many years ago. >> Exactly, yeah, it's no longer, you know, thermals and processing power as much, right? Still aspects of that, but a lot of what we're focused on now, or what I'm focused on now is within what we call the emerging technology space. Or horizon 2, horizon 3, I guess. >> Sounds like something some analyst firm came up with, Dave. (Dave laughing) >> Yeah, like Industry 4.0, 5.0 type stuff. It's all exciting stuff, but you know when you look at technologies like five, 5G, fifth generation wireless, you know both millimeter waves, sub six gigahertz, AI, you know, everything old becoming new again, right? Stuff from the fifties, and sixties that's now starting to permeate everything that we do, you're not opening your mouth and breathing unless you're talking about AI at some point, >> Yeah, and you bring up a great point. So, we've spent some time with the Dell team understanding AI, but help connect for our audience that when you talk high AI we're talking about, we're talking about data at the center of everything, and it's those applications, are you working on some of those solutions, or is it the infrastructure that's going to enable that, and what needs to be done at that level for things to work right? >> I think it's all of the above. The beauty of kind of Dell Technologies that you sit across, both infrastructure and software. You look at the efforts and the energies, stuff like VMware buying, BitFusion, right, as a mechanism trying to assuage some of that low-level hardware stuff. Start to tap into what the infrastructure guys have always been doing. When you bring that kind of capability up the stack, now you can start to develop within the software mindset, how, how you're going to access this. Infrastructure still plays a huge part of it, you got to run it on something, right? You can't really do serverless AI at this point, am I allowed to say that? (man laughing) >> Well, you could say that, I might disagree with you, because absolutely >> Eh, that's fine. there's AI that's running on it. Don't you know, Dave, I actually did my serverless 101 article that I had, I actually had Ashley Gorakhpurwalla, who is the General Manager of Dell servers, holding the t-shirt that "there is no serverless, it's just, you know, a function that you only pay the piece that you need when you need and everything there." But the point of the humor that I was having there is even the largest server manufacturer in the world knows that underneath that serverless discussion, absolutely, there is still infrastructure that plays there, just today it tends to primarily be in AWS with all of their services, but that proliferation, serverless, we're just letting the developers be developers and not have to think about that stuff, and I mean, Dave, the stuff we've had background, you know, we want to get rid of silos and make things simpler, I mean, it's the things we've been talking about for decades, it's just, for me it was interesting to look at, it is very much a developer application driven piece, top-down as opposed to so many of the virtualization and infrastructure as a service is more of a bottom-up, let me try to change this construct so that we can then provide what you need above it, it's just a slightly different way of looking at things. >> Yeah, and I think we're really trying to push for that stuff, so you know you can bundle together hardware that makes it, makes the development platform easy to do, right? But the efforts and energy of our partnerships, Dell has engaged in a lot of partnerships within the industry, NVIDIA, Intel, AMD, Graphcore, you name it, right? We're out in that space working along with those folks, but a lot of that is driven by software. It's, you write to a library, like Kudu, or, you know pyEight, you know, PyTorch, you're using these type of elements and you're moving towards that, but then it has to run on something, right? So we want to be in that both-end space, right? We want to enable that kind of flexibility capability, and obviously not prevent it, but we want to also expose that platform to as many people within the industry as possible so they can kind of start to develop on it. You're becoming a platform company, really, when it comes down to it. >> I don't want to get down the semantical arguments of AI, if you will, but what are you hearing from customers, and what's some kind of driving some of the discussions lately that's the reality of AI as opposed to some of just the buzzy hype that everybody talks about? >> Well I still think there's some ambiguity in market around AI versus automation even, so what people that come and ask us are well, "you know, I believe in this thing called artificial intelligence, and I want to do X, Y, and Z." And these particular workloads could be better handled by a simple, not to distill it down to the barest minimum, but like cron jobs, something that's, go back in the history, look at the things that matter, that you could do very very simply that don't require a large amount of library, or sort of an understanding of more advanced-type algorithms or developments that way. In the reverse, you still have that capability now, where everything that we're doing within industry, you use chat-bots. Some of the intelligence that goes into those, people are starting to recognize, this is a better way that I could serve my customers. Really, it's that business out kind of viewpoint. How do I access these customers, where they may not have the knowledge set here, but they're coming to us and saying, "it's more than just, you know, a call, an IVR system," you know, like an electronic IVR system, right? Like I come in and it's just quick response stuff. I need some context, I need to be able to do this, and transform my data into something that's useful for my customers. >> Yeah, no, this is such a great point, Dave. The thing I've asked many times, is, my entire career we've talked about intelligence and we've talked about automation, what's different about it today? And the reality is, is it used to be all right. I was scripting things, or I would have some Bash processes, or I would put these things together. The order of magnitude and scale of what we're talking about today, I couldn't do it manually if I wanted to. And that automation is really, can be really cool these days, and it's not as, to set all of those up, there is more intelligence built into it, so whether it's AI or just machine learning kind of underneath it, that spectrum that we talk about it, there's some real-use cases, a real lot of things that are happening there, and it definitely is, order of magnitudes more improved than what we were talking about say, back when we were both at EMC and the latest generation of Symmetrix was much more intelligent than the last generation, but if you look at that 10 years later, boy, it's, it is night and day, and how could we ever have used those terms before, compared to where we are today. >> Yeah it's, it's, somebody probably at some point coined the term, "exponential". Like, things become exponential as you start to look at it. Yeah, the development in the last 10 years, both in computing horsepower, and GPU/GPGPU horsepower, you know, the innovation around, you know FPGAs are back in a big way now, right? All that brainpower that used to be in these systems now, you now can benefit even more from the flexibility of the systems in order to get specific workloads done. It's not for everybody, we all know that, but it's there. >> I'm glad you brought up FPGAs because those of us that are hardware geeks, I mean, some reason I studied mechanical engineering, not realizing that software would be a software world that we live in. I did a video with Amy Lewis and she's like, "what was your software-defined moments?" I'm like, "gosh, I'm the frog sitting in the pot, and, would love to, if I can't network-diagram it, or put these things together, networking guy, it's my background! So, the software world, but it is a real renaissance in hardware these days. Everything from the FPGAs you mentioned, you look at NVIDIA and all of their partners, and the competitors there. Anything you geeking out on the hardware side? >> I, yeah, a lot of the stuff, I mean, the era of GPU showed up in a big way, all right? We have NVIDIA to thank for that whole, I mean, the kudos to them for developing a software ecosystem alongside a hardware. I think that's really what sold that and made that work. >> Well, you know, you have to be able to solve that Bitcoin mining problem, so. >> Well, you know, depending on which cryptocurrency you did, EMD kind of snuck in there with their stuff and they did some of that stuff better. But you have that kind of competing architecture stuff, which is always good, competition you want. I think now that what we're seeing is that specific workloads now benefit from different styles of compute. And so you have the companies like Graphcore, or the chip that was just launched out of China this past week that's configurable to any type of network, enteral network underneath the covers. You see that kind of evolution in capability now, where general purpose is good, but now you start to go into reconfigurable elements so, I'll, FPGAs are some of these more advanced chips. The neuromorphic hardware, which is always, given my background in psychology, is always interesting to me, so anything that is biomorphic or neuromorphic to me is pinging around up here like, "oh, you're going to emulate the brain?" And Intel's done stuff, BraincChip's done stuff, Netspace, it's amazing. I just, the workloads that are coming along the way, I think are starting to demand different types or more effectiveness within that hardware now, so you're starting to see a lot of interesting developments, IPUs, TPUs, Teslas getting into the inferencing bit now, with their own hardware, so you see a lot of effort and energy being poured in there. Again, there's not going to be one ring to rule them all, to cop Tolkien there for a moment, but there's going to be, I think you're going to start to see the disparation of workloads into those specific hardware platforms. Again, software, it's going to start to drive the applications for how you see these things going, and it's going to be the people that can service the most amount of platforms, or the most amount of capability from a single platform even, I think are the people who are going to come out ahead. And whether it'll be us or any of our August competitors, it remains to be seen, but we want to be in that space we want to be playing hard in that space as well. >> All right Dave, last thing I want to ask you about is just career. So, it's interesting, at Vmworld, I kind of look at it in like, "wow, I'm actually, I'm sitting at a panel for Opening Acts, which is done by the VMunderground people the Sunday, day before VMworld really starts, talking about jobs and there's actually three panels, you know, careers, and financial, and some of those things, >> I'm going to be there, so come on by, >> Maybe I should join startin' at 1 o'clock Monday evening, I'm actually participating in a career cafe, talking about people and everything like that, so all that stuff's online if you want to check it out, but you know, right, you said psychology is what you studied but you worked in engineering, you were a systems engineer, and now you do messaging. The hardcore techies, there's always that boundary between the techies and the marketings, but I think it's obvious to our audience when they hear you geeking out on the TPUs and all the things there that you are not just, you're quite knowledgeable when it comes about the technology, and the good technical marketers I find tend to come from that kind of background, but give us a little bit, looking back at where you've been and where you're going, and some of those dynamics. >> Yeah, I was blessed from a really young age with a father who really loved technology. We were building PCs, like back in the eighties, right, when that was a thing, you know, "I built my AMD 386 DX box" >> Have you watched the AMC show, "Halt and Catch Fire," when that was on? >> Yeah, yeah, yeah, so there was that kind of, always interesting to me, and I, with the way my mind works, I can't code to save my life, that's my brother's gift, not mine. But being able to kind of assemble things in my head was kind of always something that stuck in the back. So going through college, I worked as a lab resident as well, working in computer labs and doing that stuff. It's just been, it's been a passion, right? I had the education, was very, you know, that was my family, was very hard on the education stuff. You're going to do this. But being able to follow that passion, a lot of things fell into place with that, it's been a huge blessing. But even in grad school when I was getting my Masters in clinical counseling, I ran my own consulting business as well, just buying and selling hardware. And a lot of what I've done is just I read and ask a ton of questions. I'm out on Twitter, I'm not the brightest bulb in the, of the bunch, but I've learned to ask a lot of questions and the amount of community support in that has gotten me a lot of where I am as well. But yeah, being able to come out on this side, marketing is, like you're saying, it's kind of an anathema to the technical guys, "oh those are the guys that kind of shine the, shine the turd, so to speak," right? But being able to come in and being able to kind of influence the way and make sure that we're technically sound in what we're saying, but you have to translate some of the harder stuff, the more hardcore engineering terms into layman's terms, because not everybody's going to approach that. A CIO with a double E, or an MS in electrical engineering are going on down that road are very few and far between. A lot of these folks have grown up or developed their careers in understanding things, but being able to kind of go in and translate through that, it's been a huge blessing, it's nice. But always following the areas where, networking for me was never a strong point, but jumping in, going, "hey, I'm here to learn," and being willing to learn has been one of the biggest, biggest things I think that's kind of reinforced that career process. >> Yeah, definitely Dave, that intellectual curiosity is something that serves anyone in the tech industry quite well, 'cause, you know, nobody is going to be an expert on everything, and I've spoken to some of the brightest people in the industry, and even they realize nobody can keep up with all of it, so that being able to ask questions, participate, and Dave, thank you so much for helping me, come have this conversation, great as always to have a chat. >> Ah, great to be here Stu, thanks. >> Alright, so be sure to check out the theCUBE.net, which is where all of our content always is, what shows we will be at, all the history of where we've been. This studio is actually in Marlborough, Massachusetts, so not too far outside of Boston, right on the 495 loop, we're going to be doing lot more videos here, myself and Dave Vellante are located here, we have a good team here, so look for more content out of here, and of course our big studio out of Palo Alto, California. So if we can be of help, please feel free to reach out, I'm Stu Miniman, and as always, thanks for watching theCUBE. (upbeat electronic music)

(upbeat music) >> Live from Las Vegas, it's theCUBE covering AWS re:Invent 2018. Brought to you by Amazon Web Services, Intel, and their ecosystem partners. >> Okay, welcome back everyone. Live coverage here in Las Vegas for Amazon re:Invent 2018. Day three, we're winding down over 150 videos. We'll have over 500 clips. Losing the voice. Dave Vellante, my co-host. Suzi analyst tech that we're going to extract theCUBE insights, James Kobielus. David Floyer from Wikibon. Jim you've been prolific on the blogs, Siliconangle.com, great stories. David you've got some research. What's your take? Jim, you're all over what's going on in the news. What's the impact? >> Well I think what this years re:Invent shows is that AWS is doubling down on A.I. If you look at the sheer range of innovative A.I. capabilities they've introduced into their portfolio, in terms of their announcements, it's really significant. A. They have optimized tense or flow for their cloud. B. They now have an automated labeling, called Ground Truth, labeling capability that leverages mechanical turf, which has been an Amazon capability for a while. They've also got now the industries first, what's called reinforcement learning plug-in to their data science tool chain, in this case Sage Maker, reinforcement learning is becoming so important for robotics, and gaming, and lots of other applications of A.I., and I'm just scratching the surface. So they've announced a lot of things, and David can discuss other things, but I'm seeing the depth of A.I. Their investment in it shows that they've really got their fingers on what enterprises are doing, and will be doing to differentiate themselves with this technology over the next five to ten years. >> What's an area that you see that people are getting? Clearly A.I. What areas are people missing that's compelling that you've observed here? >> When you say people are missing, you mean the general...? >> Journalists. >> Oh. >> Audience. There's so much news. >> Yeah. Yeah. >> Where are the nuggets that are hidden in the news? (laughing) What are you seeing that people might not see that's different? >> Getting back to the point I was raising, which is that robotics is becoming a predominant application realm for A.I. Robotics, outside the laboratory, or outside of the industrial I.O.T., robots are coming into everything, and there's a special type of A.I. you build into robots, re-enforcement learning is a big part of it. So I think the general, if you look at the journalists, they've missed the fact that I've seen in the past couple of years, robotics and re-enforcement learning are almost on the verge of being mainstream in the space, and AWS gets it. Just the depth of their investments. Like Deep Racer, that cute little autonomous vehicle that they rolled out here at this event, that just shows that they totally get it. That will be a huge growth sector. >> David Floyer, outpost is their on premises cloud. You've been calling this for I don't know how many years, >> (laughing) Three years. >> Three years? >> Yeah. What's the impact? >> And people said, no way Foyer's wrong (laughing). >> So you get vindication but... >> And people, in particular in AWS. (laughing) >> So you're right. So you're right, but is it going to be out in a year? >> Yeah, next in 2019. >> Will this thing actually make it to the market? And if it does what is the impact? Who wins and who loses? >> Well let's start with will it get to the market? Absolutely. It is outposts, AWS Outposts, is the name. It is taking AWS in the cloud and putting it on premise. The same API's. The same services. It'll be eventually identical between the two. And that has enormous increase in the range, and the reach that AWS and the time that AWS can go after. It is a major, major impact on the marketplace, puts pressure on a whole number of people, the traditional vendors who are supplying that marketplace of the moment, and in my opinion it's going to be wildly successful. People have been waiting that, wanting that, particularly in the enterprise market. They reasons for it are simple. Latency, low latency, you've got to have the data and the compute very close together. Moving data is very, very expensive over long distances, and the third one is many people want, or need to have the data in certain places. So the combination is meeting the requirements, they've taken a long time to get there. I think it's going to be, however wildly successful. It's going to be coming out in 2019. They'll have their alpha, their betas in the beginning of it. They'll have some announcements, probably about mid 2019. >> Who's threatened by this? Everybody? Cisco? HP? Dell? >> The integration of everything, storage, networking, compute, all in the same box is obviously a threat to all suppliers within that. And their going to have to adapt to that pretty strongly. It's going to be a declining market. Declining markets are good if you adapt properly. A lot of people make a lot of money from, like IBM, from mainframe. >> It's a huge threat to IBM. >> You're playing it safe. You're not naming names. (laughing) Okay, I'll rephrase. What's your prediction? >> What's my prediction on? >> Of the landscape after this is wildly successful. >> The landscape is that the alternatives is going to be a much, much smaller pie, and only those that have volume, and only those that can adapt to that environment are going to survive. >> Well, and let's name names. So who's threatened by this? Clearly Dell, EMC, is threatened by this. >> HP. >> HP, New Tanix, the VX rat guys, Lenovo is in there. Are they wiped out? No, but they have to respond. How do they respond? >> They have to respond, yeah. They have to have self service. They have to have utility pricing. They have to connect to the cloud. So either they go hard after AWS, connecting AWS, or they belly up to Microsoft >> With Azure Stack, >> Microsoft Azure. that's clearly going to be their fallback place, so in a way, Microsoft with Azure Stack is also threatened by this, but in a way it's goodness for them because the ecosystem is going to evolve to that. So listen, these guys don't just give up. >> No, no I know. >> They're hard competitors, they're fighters. It's also to me a confirmation of Oracle's same same strategy. On paper Oracle's got that down, they're executing on that, even though it's in a narrow Oracle world. So I think it does sort of indicate that that iPhone for the enterprise strategy is actually quite viable. If I may jump in here, four things stood out to me. The satellite as a service, was to me amazing. What's next? Amazon with scale, there's just so many opportunities for them. The Edge, if we have time. >> I was going to talk about the Edge. >> Love to talk about the Edge. The hybrid evolution, and Open Source. Amazon use to make it easy for the enterprise players to complete. They had limited sales and service capabilities, they had no Open Source give back, they were hybrid deniers. Everything's going to go into the public cloud. That's all changed. They're making it much, much more difficult, for what they call the old guard, to compete. >> So that same way the objection? >> Yeah, they're removing those barriers, those objections. >> Awesome. Edge. >> Yeah, and to comment on one of the things you were talking about, which is the Edge, they have completely changed their approach to the Edge. They have put in Neo as part of Sage Maker, which allows them to push out inference code, and they themselves are pointing out that inference code is 90% of all the compute, into... >> Not the training. >> Not the training, but the inference code after that, that's 90% of the compute. They're pushing that into the devices at the Edge, all sorts of architectures. That's a major shift in mindset about that. >> Yeah, and in fact I was really impressed by Elastic Inference for the same reasons, because it very much is a validation of a trend I've been seeing in the A.I. space for the last several years, which is, you can increasingly build A.I. in your preferred visual, declarative environment with Python code, and then the abstraction layers of the A.I. Ecosystem have developed to a point where, the ecosystem increasingly will auto-compile to TensorFlow, or MXNet, or PyTorch, and then from there further auto-compile your deployed trained model to the most efficient format for the Edge device, for the GP, or whatever. Where ever it's going to be executed, that's already a well established trend. The fact that AWS has productized that, with this Elastic Inference in their cloud, shows that not only do they get that trend, they're just going to push really hard. I'm making sure that AWS, it becomes in many ways, the hub of efficient inferencing for everybody. >> One more quick point on the Edge, if I may. What's going on on the Edge reminds me of the days when Microsoft was trying to take Windows and stick it on mobile. Right, the windows phone. Top down, I.T. guys coming at it, >> Oh that's right. >> and that's what a lot of people are doing today in IT. It's not going to work. What Amazon is doing see, we're going to build an environment that you can build applications on, that are secure, you can manage them from a bottoms up approach. >> Yeah. Absolutely. >> Identifying what the operations technology developers want. Giving them the tools to do that. That's a winning strategy. >> And focusing on them producing the devices, not themselves. >> Right. >> And not declaring where the boundaries are. >> Spot on. >> Very very important. >> Yep. >> And they're obviously inferencing, you get most value out of the data if you put that inferencing as close as you possibly can to that data, within a camera, is in the camera itself. >> And I eluded to it earlier, another key announcement from AWS here is, first of all the investment in Sage Maker itself is super impressive. In the year since they've introduced it, look at they've already added, they have that slide with all the feature enhancements, and new modules. Sage Maker Ground Truth, really important, the fully managed service for automating labeling of training datasets, using Mechanical Turk . The vast majority of the costs in a lot of A.I. initiatives involves human annotators of training data, and without human annotated training data you can't do supervised learning, which is the magic on a lot of A.I, AWS gets the fact that their customers want to automate that to the nth degree. Now they got that. >> We sound like Fam boys (laughing). >> That's going to be wildly popular. >> As we say, clean data makes good M.L., and good M.L. makes great A.I. >> Yeah. (laughing) >> So you don't want any dirty data out there. Cube, more coverage here. Cube insights panel, here in theCUBE at re:Invent. Stay with us for more after this short break. (upbeat music)

>> Live from Las Vegas, it's theCUBE, covering AWS re:Invent 2018. Brought to you by Amazon Web Services, Intel, and their Ecosystem partners. >> Oh welcome back, to theCUBE. Continuing coverage here from AWS re:Invent, as we start to wind down our coverage here on the second day. We'll be here tomorrow as well, live on theCUBE, bringing you interviews from Hall D at the Sands Expo. Along with Justin Warren, I'm John Walls, and we're joined by Jonathan Ballon, who's the Vice President of the internet of things at Intel. Jonathan, thank you for being with us today. Good to see you, >> Thanks for having me guys. >> All right, interesting announcement today, and last year it was all about DeepLens. This year it's about DeepRacer. Tell us about that. >> What we're really trying to do is make AI accessible to developers and democratize various AI tools. Last year it was about computer vision. The DeepLens camera was a way for developers to very inexpensively get a hold of a camera, the first camera that was a deep-learning enabled, cloud connected camera, so that they could start experimenting and see what they could do with that type of device. This year we took the camera and we put it in a car, and we thought what could they do if we add mobility to the equation, and specifically, wanted to introduce a relatively obscure form of AI called reinforcement learning. Historically this has been an area of AI that hasn't really been accessible to most developers, because they haven't had the compute resources at their disposal, or the scale to do it. And so now, what we've done is we've built a car, and a set of tools that help the car run. >> And it's a little miniature car, right? I mean it's a scale. >> It's 1/118th scale, it's an RC car. It's four-wheel drive, four-wheel steering. It's got GPS, it's got two batteries. One that runs the car itself, one that runs the compute platform and the camera. It's got expansion capabilities. We've got plans for next year of how we can turbo-charge the car. >> I love it. >> Right now it's baby steps, so to speak, and basically giving the developer the chance to write a reinforcement learning model, an algorithm that helps them to determine what is the optimum way that this car can move around a track, but you're not telling the car what the optimum way is, you're letting the car figure it out on their own. And that's really the key to reinforcement learning is you don't need a large dataset to begin with, it's pre-trained. You're actually letting, in this case, a device figure it out for themselves, and this becomes very powerful as a tool, when you think about it being applied to various industries, or various use-cases, where we don't know the answer today, but we can allow vast amounts of computing resources to run a reinforcement model over and over, perhaps millions of times, until they find the optimum solution. >> So how do you, I mean that's a lot of input right? That's a lot, that's a crazy number of variables. So, how do you do that? So, how do you, like in this case, provide a car with all the multiple variables that will come into play. How fast it goes, and which direction it goes, and all that, and on different axes and all those things, to make these own determinations, and how will that then translate to a real specific case in the workplace? >> Well, I mean the obvious parallel is of course autonomous driving. AWS had Formula One on stage today during Andy Jassy's keynote, that's also an Intel customer, and what Formula One does is they have the fastest cars in the world, and they have over 120 sensors on that car that are bringing in over a million pieces of data per second. Being able to process that vast amount of data that quickly, which includes a variety of data, like it's not just, it's also audio data, it's visual data, and being able to use that to inform decisions in close to real time, requires very powerful compute resources, and those resources exist both in the cloud as well as close to the source of the data itself at the edge, in the physical environment. >> So, tell us a bit about the software that's involved here, 'cause people think of Intel, you know that some people don't know about the software heritage that Intel has. It's not just about, the Intel inside isn't just the hardware chips that's there, there's a lot of software that goes into this. So, what's the Intel angle here on the software that powers this kind of distributed learning. >> Absolutely, software is a very important part of any AI architecture, and for us we've a tremendous amount of investment. It's almost perhaps, equal investment in software as we do in hardware. In the case of what we announced today with DeepRacer and AWS, there's some toolkits that allow developers to better harness the compute resources on the car itself. Two things specifically, one is we have a tool called, RL Coach or Reinforcement Learning Coach, that is integrated into SageMaker, AWS' machine learning toolkit, that allows them to access better performance in the cloud of that data that's coming into the, off their model and into their cloud. And then we also have a toolkit called OpenVINO. It's not about drinking wine. >> Oh darn. >> Alright. >> Open means it's an opensource contribution that we made to the industry. Vino, V-I-N-O is Visual Inference and Neural Network Optimization, and this is a powerful tool, because so much of AI is about harnessing compute resources efficiently, and as more and more of the data that we bring into our compute environments is actually taking place in the physical world, it's really important to be able to do that in a cost-effective and power-efficient way. OpenVINO allows developers to actually isolate individual cores or an integrated GPU on a CPU without knowing anything about hardware architecture, and it allows them then to apply different applications, or different algorithms, or inference workloads very efficiently onto that compute architecture, but it's abstracted away from any knowledge of that. So, it's really designed for an application developer, who maybe is working with a data scientist that's built a neural network in a framework like TensorFlow, or Onyx, or Pytorch, any tool that they're already comfortable with, abstract away from the silicon and optimize their model onto this hardware platform, so it performs at orders of magnitude better performance then what you would get from a more traditional GPU approach. >> Yeah, and that kind of decision making about understanding chip architectures to be able to optimize how that works, that's some deep magic really. The amount of understanding that you would need to have to do that as a human is enormous, but as a developer, I don't know anything about chip architectures, so it sounds like the, and it's a thing that we've been hearing over the last couple of days, is these tools allow developers to have essentially superpowers, so you become an augmented intelligence yourself. Rather than just giving everything to an artificial intelligence, these tools actually augment the human intelligence and allow you to do things that you wouldn't otherwise be able to do. >> And that's I think the key to getting mass market adoption of some of these AI implementations. So, for the last four or five years since ImageNet solved the image recognition problem, and now we have greater accuracy from computer models then we do from our own human eyes, really AI was limited to academia, or large IT tech companies, or proof-of-concepts. It didn't really scale into these production environments, but what we've seen over the couple of years is really a democratization of AI by companies like AWS and Intel that are making tools available to developers, so they don't need to know how to code in Python to optimize a compute module, or they don't need to, in many cases, understand the fundamental underlying architectures. They can focus on whatever business problem they're tryin' to solve, or whatever AI use-case it is that they're working on. >> I know you talked about DeepLens last year, and now we've got DeepRacer this year, and you've got the contest going on throughout this coming year with DeepRacer, and we're going to have a big race at the AWS re:Invent 2019. So what's next? I mean, or what are you thinking about conceptually to, I guess build on what you've already started there? >> Well, I can't reveal what next years, >> Well that I understand >> Project will be. >> But generally speaking. >> But what I can tell you, what I can tell you is what's available today in these DeepRacer cars is a level playing field. Everyone's getting the same car and they have essentially the same tool sets, but I've got a couple of pro-tips for your viewers if they want to win some of these AWS Summits that are going to be around the world in 2019. Two pro-tips, one is they can leverage the OpenVINO toolkit to get much higher inference performance from what's already on that car. So, I encourage them to work with OpenVINO. It's integrated into SageMaker, so that they have easy access to it if they're an AWS developer, but also we're going to allow an expansion of, almost an accelerator of the car itself, by being able to plug in an Intel Neural Compute Stick. We just released the second version of this stick. It's a USB form factor. It's got a Movidius Myriad X Vision processing unit inside. This years version is eight times more powerful than last years version, and when they plug it into the car, all of that inference workload, all of those images, and information that's coming off those sensors will be put onto the VPU, allowing all the CPU, and GPU resources to be used for other activities. It's going to allow that car to go at turbo speed. >> To really cook. >> Yeah. (laughing) >> Alright, so now you know, you have no excuse, right? I mean Jonathan has shared the secret sauce, although I still think when you said OpenVINO you got Justin really excited. >> It is vino time. >> It is five o'clock actually. >> Alright, thank you for being with us. >> Thanks for having me guys. >> And good luck with DeepRacer for the coming year. >> Thank you. >> It looks like a really, really fun project. We're back with more, here at AWS re:Invent on theCUBE, live in Las Vegas. (rhythmic digital music)