(mellow music) >> Hello everyone, welcome to theCUBE's coverage of AWS Startup Showcase. This is the closing panel session on AI machine learning, the top startups generating generative AI on AWS. It's a great panel. This is going to be the experts talking about riding the wave in generative AI. We got Ankur Mehrotra, who's the director and general manager of AI and machine learning at AWS, and Clem Delangue, co-founder and CEO of Hugging Face, and Ori Goshen, who's the co-founder and CEO of AI21 Labs. Ori from Tel Aviv dialing in, and rest coming in here on theCUBE. Appreciate you coming on for this closing session for the Startup Showcase. >> Thanks for having us. >> Thank you for having us. >> Thank you. >> I'm super excited to have you all on. Hugging Face was recently in the news with the AWS relationship, so congratulations. Open source, open science, really driving the machine learning. And we got the AI21 Labs access to the LLMs, generating huge scale live applications, commercial applications, coming to the market, all powered by AWS. So everyone, congratulations on all your success, and thank you for headlining this panel. Let's get right into it. AWS is powering this wave here. We're seeing a lot of push here from applications. Ankur, set the table for us on the AI machine learning. It's not new, it's been goin' on for a while. Past three years have been significant advancements, but there's been a lot of work done in AI machine learning. Now it's released to the public. Everybody's super excited and now says, "Oh, the future's here!" It's kind of been going on for a while and baking. Now it's kind of coming out. What's your view here? Let's get it started. >> Yes, thank you. So, yeah, as you may be aware, Amazon has been in investing in machine learning research and development since quite some time now. And we've used machine learning to innovate and improve user experiences across different Amazon products, whether it's Alexa or Amazon.com. But we've also brought in our expertise to extend what we are doing in the space and add more generative AI technology to our AWS products and services, starting with CodeWhisperer, which is an AWS service that we announced a few months ago, which is, you can think of it as a coding companion as a service, which uses generative AI models underneath. And so this is a service that customers who have no machine learning expertise can just use. And we also are talking to customers, and we see a lot of excitement about generative AI, and customers who want to build these models themselves, who have the talent and the expertise and resources. For them, AWS has a number of different options and capabilities they can leverage, such as our custom silicon, such as Trainium and Inferentia, as well as distributed machine learning capabilities that we offer as part of SageMaker, which is an end-to-end machine learning development service. At the same time, many of our customers tell us that they're interested in not training and building these generative AI models from scratch, given they can be expensive and can require specialized talent and skills to build. And so for those customers, we are also making it super easy to bring in existing generative AI models into their machine learning development environment within SageMaker for them to use. So we recently announced our partnership with Hugging Face, where we are making it super easy for customers to bring in those models into their SageMaker development environment for fine tuning and deployment. And then we are also partnering with other proprietary model providers such as AI21 and others, where we making these generative AI models available within SageMaker for our customers to use. So our approach here is to really provide customers options and choices and help them accelerate their generative AI journey. >> Ankur, thank you for setting the table there. Clem and Ori, I want to get your take, because the riding the waves, the theme of this session, and to me being in California, I imagine the big surf, the big waves, the big talent out there. This is like alpha geeks, alpha coders, developers are really leaning into this. You're seeing massive uptake from the smartest people. Whether they're young or around, they're coming in with their kind of surfboards, (chuckles) if you will. These early adopters, they've been on this for a while; Now the waves are hitting. This is a big wave, everyone sees it. What are some of those early adopter devs doing? What are some of the use cases you're seeing right out of the gate? And what does this mean for the folks that are going to come in and get on this wave? Can you guys share your perspective on this? Because you're seeing the best talent now leaning into this. >> Yeah, absolutely. I mean, from Hugging Face vantage points, it's not even a a wave, it's a tidal wave, or maybe even the tide itself. Because actually what we are seeing is that AI and machine learning is not something that you add to your products. It's very much a new paradigm to do all technology. It's this idea that we had in the past 15, 20 years, one way to build software and to build technology, which was writing a million lines of code, very rule-based, and then you get your product. Now what we are seeing is that every single product, every single feature, every single company is starting to adopt AI to build the next generation of technology. And that works both to make the existing use cases better, if you think of search, if you think of social network, if you think of SaaS, but also it's creating completely new capabilities that weren't possible with the previous paradigm. Now AI can generate text, it can generate image, it can describe your image, it can do so many new things that weren't possible before. >> It's going to really make the developers really productive, right? I mean, you're seeing the developer uptake strong, right? >> Yes, we have over 15,000 companies using Hugging Face now, and it keeps accelerating. I really think that maybe in like three, five years, there's not going to be any company not using AI. It's going to be really kind of the default to build all technology. >> Ori, weigh in on this. APIs, the cloud. Now I'm a developer, I want to have live applications, I want the commercial applications on this. What's your take? Weigh in here. >> Yeah, first, I absolutely agree. I mean, we're in the midst of a technology shift here. I think not a lot of people realize how big this is going to be. Just the number of possibilities is endless, and I think hard to imagine. And I don't think it's just the use cases. I think we can think of it as two separate categories. We'll see companies and products enhancing their offerings with these new AI capabilities, but we'll also see new companies that are AI first, that kind of reimagine certain experiences. They build something that wasn't possible before. And that's why I think it's actually extremely exciting times. And maybe more philosophically, I think now these large language models and large transformer based models are helping us people to express our thoughts and kind of making the bridge from our thinking to a creative digital asset in a speed we've never imagined before. I can write something down and get a piece of text, or an image, or a code. So I'll start by saying it's hard to imagine all the possibilities right now, but it's certainly big. And if I had to bet, I would say it's probably at least as big as the mobile revolution we've seen in the last 20 years. >> Yeah, this is the biggest. I mean, it's been compared to the Enlightenment Age. I saw the Wall Street Journal had a recent story on this. We've been saying that this is probably going to be bigger than all inflection points combined in the tech industry, given what transformation is coming. I guess I want to ask you guys, on the early adopters, we've been hearing on these interviews and throughout the industry that there's already a set of big companies, a set of companies out there that have a lot of data and they're already there, they're kind of tinkering. Kind of reminds me of the old hyper scaler days where they were building their own scale, and they're eatin' glass, spittin' nails out, you know, they're hardcore. Then you got everybody else kind of saying board level, "Hey team, how do I leverage this?" How do you see those two things coming together? You got the fast followers coming in behind the early adopters. What's it like for the second wave coming in? What are those conversations for those developers like? >> I mean, I think for me, the important switch for companies is to change their mindset from being kind of like a traditional software company to being an AI or machine learning company. And that means investing, hiring machine learning engineers, machine learning scientists, infrastructure in members who are working on how to put these models in production, team members who are able to optimize models, specialized models, customized models for the company's specific use cases. So it's really changing this mindset of how you build technology and optimize your company building around that. Things are moving so fast that I think now it's kind of like too late for low hanging fruits or small, small adjustments. I think it's important to realize that if you want to be good at that, and if you really want to surf this wave, you need massive investments. If there are like some surfers listening with this analogy of the wave, right, when there are waves, it's not enough just to stand and make a little bit of adjustments. You need to position yourself aggressively, paddle like crazy, and that's how you get into the waves. So that's what companies, in my opinion, need to do right now. >> Ori, what's your take on the generative models out there? We hear a lot about foundation models. What's your experience running end-to-end applications for large foundation models? Any insights you can share with the app developers out there who are looking to get in? >> Yeah, I think first of all, it's start create an economy, where it probably doesn't make sense for every company to create their own foundation models. You can basically start by using an existing foundation model, either open source or a proprietary one, and start deploying it for your needs. And then comes the second round when you are starting the optimization process. You bootstrap, whether it's a demo, or a small feature, or introducing new capability within your product, and then start collecting data. That data, and particularly the human feedback data, helps you to constantly improve the model, so you create this data flywheel. And I think we're now entering an era where customers have a lot of different choice of how they want to start their generative AI endeavor. And it's a good thing that there's a variety of choices. And the really amazing thing here is that every industry, any company you speak with, it could be something very traditional like industrial or financial, medical, really any company. I think peoples now start to imagine what are the possibilities, and seriously think what's their strategy for adopting this generative AI technology. And I think in that sense, the foundation model actually enabled this to become scalable. So the barrier to entry became lower; Now the adoption could actually accelerate. >> There's a lot of integration aspects here in this new wave that's a little bit different. Before it was like very monolithic, hardcore, very brittle. A lot more integration, you see a lot more data coming together. I have to ask you guys, as developers come in and grow, I mean, when I went to college and you were a software engineer, I mean, I got a degree in computer science, and software engineering, that's all you did was code, (chuckles) you coded. Now, isn't it like everyone's a machine learning engineer at this point? Because that will be ultimately the science. So, (chuckles) you got open source, you got open software, you got the communities. Swami called you guys the GitHub of machine learning, Hugging Face is the GitHub of machine learning, mainly because that's where people are going to code. So this is essentially, machine learning is computer science. What's your reaction to that? >> Yes, my co-founder Julien at Hugging Face have been having this thing for quite a while now, for over three years, which was saying that actually software engineering as we know it today is a subset of machine learning, instead of the other way around. People would call us crazy a few years ago when we're seeing that. But now we are realizing that you can actually code with machine learning. So machine learning is generating code. And we are starting to see that every software engineer can leverage machine learning through open models, through APIs, through different technology stack. So yeah, it's not crazy anymore to think that maybe in a few years, there's going to be more people doing AI and machine learning. However you call it, right? Maybe you'll still call them software engineers, maybe you'll call them machine learning engineers. But there might be more of these people in a couple of years than there is software engineers today. >> I bring this up as more tongue in cheek as well, because Ankur, infrastructure's co is what made Cloud great, right? That's kind of the DevOps movement. But here the shift is so massive, there will be a game-changing philosophy around coding. Machine learning as code, you're starting to see CodeWhisperer, you guys have had coding companions for a while on AWS. So this is a paradigm shift. How is the cloud playing into this for you guys? Because to me, I've been riffing on some interviews where it's like, okay, you got the cloud going next level. This is an example of that, where there is a DevOps-like moment happening with machine learning, whether you call it coding or whatever. It's writing code on its own. Can you guys comment on what this means on top of the cloud? What comes out of the scale? What comes out of the benefit here? >> Absolutely, so- >> Well first- >> Oh, go ahead. >> Yeah, so I think as far as scale is concerned, I think customers are really relying on cloud to make sure that the applications that they build can scale along with the needs of their business. But there's another aspect to it, which is that until a few years ago, John, what we saw was that machine learning was a data scientist heavy activity. They were data scientists who were taking the data and training models. And then as machine learning found its way more and more into production and actual usage, we saw the MLOps become a thing, and MLOps engineers become more involved into the process. And then we now are seeing, as machine learning is being used to solve more business critical problems, we're seeing even legal and compliance teams get involved. We are seeing business stakeholders more engaged. So, more and more machine learning is becoming an activity that's not just performed by data scientists, but is performed by a team and a group of people with different skills. And for them, we as AWS are focused on providing the best tools and services for these different personas to be able to do their job and really complete that end-to-end machine learning story. So that's where, whether it's tools related to MLOps or even for folks who cannot code or don't know any machine learning. For example, we launched SageMaker Canvas as a tool last year, which is a UI-based tool which data analysts and business analysts can use to build machine learning models. So overall, the spectrum in terms of persona and who can get involved in the machine learning process is expanding, and the cloud is playing a big role in that process. >> Ori, Clem, can you guys weigh in too? 'Cause this is just another abstraction layer of scale. What's it mean for you guys as you look forward to your customers and the use cases that you're enabling? >> Yes, I think what's important is that the AI companies and providers and the cloud kind of work together. That's how you make a seamless experience and you actually reduce the barrier to entry for this technology. So that's what we've been super happy to do with AWS for the past few years. We actually announced not too long ago that we are doubling down on our partnership with AWS. We're excited to have many, many customers on our shared product, the Hugging Face deep learning container on SageMaker. And we are working really closely with the Inferentia team and the Trainium team to release some more exciting stuff in the coming weeks and coming months. So I think when you have an ecosystem and a system where the AWS and the AI providers, AI startups can work hand in hand, it's to the benefit of the customers and the companies, because it makes it orders of magnitude easier for them to adopt this new paradigm to build technology AI. >> Ori, this is a scale on reasoning too. The data's out there and making sense out of it, making it reason, getting comprehension, having it make decisions is next, isn't it? And you need scale for that. >> Yes. Just a comment about the infrastructure side. So I think really the purpose is to streamline and make these technologies much more accessible. And I think we'll see, I predict that we'll see in the next few years more and more tooling that make this technology much more simple to consume. And I think it plays a very important role. There's so many aspects, like the monitoring the models and their kind of outputs they produce, and kind of containing and running them in a production environment. There's so much there to build on, the infrastructure side will play a very significant role. >> All right, that's awesome stuff. I'd love to change gears a little bit and get a little philosophy here around AI and how it's going to transform, if you guys don't mind. There's been a lot of conversations around, on theCUBE here as well as in some industry areas, where it's like, okay, all the heavy lifting is automated away with machine learning and AI, the complexity, there's some efficiencies, it's horizontal and scalable across all industries. Ankur, good point there. Everyone's going to use it for something. And a lot of stuff gets brought to the table with large language models and other things. But the key ingredient will be proprietary data or human input, or some sort of AI whisperer kind of role, or prompt engineering, people are saying. So with that being said, some are saying it's automating intelligence. And that creativity will be unleashed from this. If the heavy lifting goes away and AI can fill the void, that shifts the value to the intellect or the input. And so that means data's got to come together, interact, fuse, and understand each other. This is kind of new. I mean, old school AI was, okay, got a big model, I provisioned it long time, very expensive. Now it's all free flowing. Can you guys comment on where you see this going with this freeform, data flowing everywhere, heavy lifting, and then specialization? >> Yeah, I think- >> Go ahead. >> Yeah, I think, so what we are seeing with these large language models or generative models is that they're really good at creating stuff. But I think it's also important to recognize their limitations. They're not as good at reasoning and logic. And I think now we're seeing great enthusiasm, I think, which is justified. And the next phase would be how to make these systems more reliable. How to inject more reasoning capabilities into these models, or augment with other mechanisms that actually perform more reasoning so we can achieve more reliable results. And we can count on these models to perform for critical tasks, whether it's medical tasks, legal tasks. We really want to kind of offload a lot of the intelligence to these systems. And then we'll have to get back, we'll have to make sure these are reliable, we'll have to make sure we get some sort of explainability that we can understand the process behind the generated results that we received. So I think this is kind of the next phase of systems that are based on these generated models. >> Clem, what's your view on this? Obviously you're at open community, open source has been around, it's been a great track record, proven model. I'm assuming creativity's going to come out of the woodwork, and if we can automate open source contribution, and relationships, and onboarding more developers, there's going to be unleashing of creativity. >> Yes, it's been so exciting on the open source front. We all know Bert, Bloom, GPT-J, T5, Stable Diffusion, that work up. The previous or the current generation of open source models that are on Hugging Face. It has been accelerating in the past few months. So I'm super excited about ControlNet right now that is really having a lot of impact, which is kind of like a way to control the generation of images. Super excited about Flan UL2, which is like a new model that has been recently released and is open source. So yeah, it's really fun to see the ecosystem coming together. Open source has been the basis for traditional software, with like open source programming languages, of course, but also all the great open source that we've gotten over the years. So we're happy to see that the same thing is happening for machine learning and AI, and hopefully can help a lot of companies reduce a little bit the barrier to entry. So yeah, it's going to be exciting to see how it evolves in the next few years in that respect. >> I think the developer productivity angle that's been talked about a lot in the industry will be accelerated significantly. I think security will be enhanced by this. I think in general, applications are going to transform at a radical rate, accelerated, incredible rate. So I think it's not a big wave, it's the water, right? I mean, (chuckles) it's the new thing. My final question for you guys, if you don't mind, I'd love to get each of you to answer the question I'm going to ask you, which is, a lot of conversations around data. Data infrastructure's obviously involved in this. And the common thread that I'm hearing is that every company that looks at this is asking themselves, if we don't rebuild our company, start thinking about rebuilding our business model around AI, we might be dinosaurs, we might be extinct. And it reminds me that scene in Moneyball when, at the end, it's like, if we're not building the model around your model, every company will be out of business. What's your advice to companies out there that are having those kind of moments where it's like, okay, this is real, this is next gen, this is happening. I better start thinking and putting into motion plans to refactor my business, 'cause it's happening, business transformation is happening on the cloud. This kind of puts an exclamation point on, with the AI, as a next step function. Big increase in value. So it's an opportunity for leaders. Ankur, we'll start with you. What's your advice for folks out there thinking about this? Do they put their toe in the water? Do they jump right into the deep end? What's your advice? >> Yeah, John, so we talk to a lot of customers, and customers are excited about what's happening in the space, but they often ask us like, "Hey, where do we start?" So we always advise our customers to do a lot of proof of concepts, understand where they can drive the biggest ROI. And then also leverage existing tools and services to move fast and scale, and try and not reinvent the wheel where it doesn't need to be. That's basically our advice to customers. >> Get it. Ori, what's your advice to folks who are scratching their head going, "I better jump in here. "How do I get started?" What's your advice? >> So I actually think that need to think about it really economically. Both on the opportunity side and the challenges. So there's a lot of opportunities for many companies to actually gain revenue upside by building these new generative features and capabilities. On the other hand, of course, this would probably affect the cogs, and incorporating these capabilities could probably affect the cogs. So I think we really need to think carefully about both of these sides, and also understand clearly if this is a project or an F word towards cost reduction, then the ROI is pretty clear, or revenue amplifier, where there's, again, a lot of different opportunities. So I think once you think about this in a structured way, I think, and map the different initiatives, then it's probably a good way to start and a good way to start thinking about these endeavors. >> Awesome. Clem, what's your take on this? What's your advice, folks out there? >> Yes, all of these are very good advice already. Something that you said before, John, that I disagreed a little bit, a lot of people are talking about the data mode and proprietary data. Actually, when you look at some of the organizations that have been building the best models, they don't have specialized or unique access to data. So I'm not sure that's so important today. I think what's important for companies, and it's been the same for the previous generation of technology, is their ability to build better technology faster than others. And in this new paradigm, that means being able to build machine learning faster than others, and better. So that's how, in my opinion, you should approach this. And kind of like how can you evolve your company, your teams, your products, so that you are able in the long run to build machine learning better and faster than your competitors. And if you manage to put yourself in that situation, then that's when you'll be able to differentiate yourself to really kind of be impactful and get results. That's really hard to do. It's something really different, because machine learning and AI is a different paradigm than traditional software. So this is going to be challenging, but I think if you manage to nail that, then the future is going to be very interesting for your company. >> That's a great point. Thanks for calling that out. I think this all reminds me of the cloud days early on. If you went to the cloud early, you took advantage of it when the pandemic hit. If you weren't native in the cloud, you got hamstrung by that, you were flatfooted. So just get in there. (laughs) Get in the cloud, get into AI, you're going to be good. Thanks for for calling that. Final parting comments, what's your most exciting thing going on right now for you guys? Ori, Clem, what's the most exciting thing on your plate right now that you'd like to share with folks? >> I mean, for me it's just the diversity of use cases and really creative ways of companies leveraging this technology. Every day I speak with about two, three customers, and I'm continuously being surprised by the creative ideas. And the future is really exciting of what can be achieved here. And also I'm amazed by the pace that things move in this industry. It's just, there's not at dull moment. So, definitely exciting times. >> Clem, what are you most excited about right now? >> For me, it's all the new open source models that have been released in the past few weeks, and that they'll keep being released in the next few weeks. I'm also super excited about more and more companies getting into this capability of chaining different models and different APIs. I think that's a very, very interesting development, because it creates new capabilities, new possibilities, new functionalities that weren't possible before. You can plug an API with an open source embedding model, with like a no-geo transcription model. So that's also very exciting. This capability of having more interoperable machine learning will also, I think, open a lot of interesting things in the future. >> Clem, congratulations on your success at Hugging Face. Please pass that on to your team. Ori, congratulations on your success, and continue to, just day one. I mean, it's just the beginning. It's not even scratching the service. Ankur, I'll give you the last word. What are you excited for at AWS? More cloud goodness coming here with AI. Give you the final word. >> Yeah, so as both Clem and Ori said, I think the research in the space is moving really, really fast, so we are excited about that. But we are also excited to see the speed at which enterprises and other AWS customers are applying machine learning to solve real business problems, and the kind of results they're seeing. So when they come back to us and tell us the kind of improvement in their business metrics and overall customer experience that they're driving and they're seeing real business results, that's what keeps us going and inspires us to continue inventing on their behalf. >> Gentlemen, thank you so much for this awesome high impact panel. Ankur, Clem, Ori, congratulations on all your success. We'll see you around. Thanks for coming on. Generative AI, riding the wave, it's a tidal wave, it's the water, it's all happening. All great stuff. This is season three, episode one of AWS Startup Showcase closing panel. This is the AI ML episode, the top startups building generative AI on AWS. I'm John Furrier, your host. Thanks for watching. (mellow music)

(chill electronic music) >> Hello everyone, welcome to theCUBE's presentation of the AWS Startups Showcase, AI and machine learning, the top startups building generative AI on AWS. This is the season three, episode one of the ongoing series covering the exciting startups from the AWS ecosystem, talk about AI and machine learning. Can't believe it's three years and season one. I'm your host, John Furrier. Got a great guest today, we're joined by Joseph Nelson, the co-founder and CEO of Roboflow, doing some cutting edge stuff around computer vision and really at the front end of this massive wave coming around, large language models, computer vision. The next gen AI is here, and it's just getting started. We haven't even scratched a service. Thanks for joining us today. >> Thanks for having me. >> So you got to love the large language model, foundation models, really educating the mainstream world. ChatGPT has got everyone in the frenzy. This is educating the world around this next gen AI capabilities, enterprise, image and video data, all a big part of it. I mean the edge of the network, Mobile World Conference is happening right now, this month, and it's just ending up, it's just continue to explode. Video is huge. So take us through the company, do a quick explanation of what you guys are doing, when you were founded. Talk about what the company's mission is, and what's your North Star, why do you exist? >> Yeah, Roboflow exists to really kind of make the world programmable. I like to say make the world be read and write access. And our North Star is enabling developers, predominantly, to build that future. If you look around, anything that you see will have software related to it, and can kind of be turned into software. The limiting reactant though, is how to enable computers and machines to understand things as well as people can. And in a lot of ways, computer vision is that missing element that enables anything that you see to become software. So in the virtue of, if software is eating the world, computer vision kind of makes the aperture infinitely wide. It's something that I kind of like, the way I like to frame it. And the capabilities are there, the open source models are there, the amount of data is there, the computer capabilities are only improving annually, but there's a pretty big dearth of tooling, and an early but promising sign of the explosion of use cases, models, and data sets that companies, developers, hobbyists alike will need to bring these capabilities to bear. So Roboflow is in the game of building the community around that capability, building the use cases that allow developers and enterprises to use computer vision, and providing the tooling for companies and developers to be able to add computer vision, create better data sets, and deploy to production, quickly, easily, safely, invaluably. >> You know, Joseph, the word in production is actually real now. You're seeing a lot more people doing in production activities. That's a real hot one and usually it's slower, but it's gone faster, and I think that's going to be more the same. And I think the parallel between what we're seeing on the large language models coming into computer vision, and as you mentioned, video's data, right? I mean we're doing video right now, we're transcribing it into a transcript, linking up to your linguistics, times and the timestamp, I mean everything's data and that really kind of feeds. So this connection between what we're seeing, the large language and computer vision are coming together kind of cousins, brothers. I mean, how would you compare, how would you explain to someone, because everyone's like on this wave of watching people bang out their homework assignments, and you know, write some hacks on code with some of the open AI technologies, there is a corollary directly related to to the vision side. Can you explain? >> Yeah, the rise of large language models are showing what's possible, especially with text, and I think increasingly will get multimodal as the images and video become ingested. Though there's kind of this still core missing element of basically like understanding. So the rise of large language models kind of create this new area of generative AI, and generative AI in the context of computer vision is a lot of, you know, creating video and image assets and content. There's also this whole surface area to understanding what's already created. Basically digitizing physical, real world things. I mean the Metaverse can't be built if we don't know how to mirror or create or identify the objects that we want to interact with in our everyday lives. And where computer vision comes to play in, especially what we've seen at Roboflow is, you know, a little over a hundred thousand developers now have built with our tools. That's to the tune of a hundred million labeled open source images, over 10,000 pre-trained models. And they've kind of showcased to us all of the ways that computer vision is impacting and bringing the world to life. And these are things that, you know, even before large language models and generative AI, you had pretty impressive capabilities, and when you add the two together, it actually unlocks these kind of new capabilities. So for example, you know, one of our users actually powers the broadcast feeds at Wimbledon. So here we're talking about video, we're streaming, we're doing things live, we've got folks that are cropping and making sure we look good, and audio/visual all plugged in correctly. When you broadcast Wimbledon, you'll notice that the camera controllers need to do things like track the ball, which is moving at extremely high speeds and zoom crop, pan tilt, as well as determine if the ball bounced in or out. The very controversial but critical key to a lot of tennis matches. And a lot of that has been historically done with the trained, but fallible human eye and computer vision is, you know, well suited for this task to say, how do we track, pan, tilt, zoom, and see, track the tennis ball in real time, run at 30 plus frames per second, and do it all on the edge. And those are capabilities that, you know, were kind of like science fiction, maybe even a decade ago, and certainly five years ago. Now the interesting thing, is that with the advent of of generative AI, you can start to do things like create your own training data sets, or kind of create logic around once you have this visual input. And teams at Tesla have actually been speaking about, of course the autopilot team's focused on doing vision tasks, but they've combined large language models to add reasoning and logic. So given that you see, let's say the tennis ball, what do you want to do? And being able to combine the capabilities of what LLM's represent, which is really a lot of basically, core human reasoning and logic, with computer vision for the inputs of what's possible, creates these new capabilities, let alone multimodality, which I'm sure we'll talk more about. >> Yeah, and it's really, I mean it's almost intoxicating. It's amazing that this is so capable because the cloud scales here, you got the edge developing, you can decouple compute power, and let Moore's law and all the new silicone and the processors and the GPUs do their thing, and you got open source booming. You're kind of getting at this next segment I wanted to get into, which is the, how people should be thinking about these advances of the computer vision. So this is now a next wave, it's here. I mean I'd love to have that for baseball because I'm always like, "Oh, it should have been a strike." I'm sure that's going to be coming soon, but what is the computer vision capable of doing today? I guess that's my first question. You hit some of it, unpack that a little bit. What does general AI mean in computer vision? What's the new thing? Because there are old technology's been around, proprietary, bolted onto hardware, but hardware advances at a different pace, but now you got new capabilities, generative AI for vision, what does that mean? >> Yeah, so computer vision, you know, at its core is basically enabling machines, computers, to understand, process, and act on visual data as effective or more effective than people can. Traditionally this has been, you know, task types like classification, which you know, identifying if a given image belongs in a certain category of goods on maybe a retail site, is the shoes or is it clothing? Or object detection, which is, you know, creating bounding boxes, which allows you to do things like count how many things are present, or maybe measure the speed of something, or trigger an alert when something becomes visible in frame that wasn't previously visible in frame, or instant segmentation where you're creating pixel wise segmentations for both instance and semantic segmentation, where you often see these kind of beautiful visuals of the polygon surrounding objects that you see. Then you have key point detection, which is where you see, you know, athletes, and each of their joints are kind of outlined is another more traditional type problem in signal processing and computer vision. With generative AI, you kind of get a whole new class of problem types that are opened up. So in a lot of ways I think about generative AI in computer vision as some of the, you know, problems that you aimed to tackle, might still be better suited for one of the previous task types we were discussing. Some of those problem types may be better suited for using a generative technique, and some are problem types that just previously wouldn't have been possible absent generative AI. And so if you make that kind of Venn diagram in your head, you can think about, okay, you know, visual question answering is a task type where if I give you an image and I say, you know, "How many people are in this image?" We could either build an object detection model that might count all those people, or maybe a visual question answering system would sufficiently answer this type of problem. Let alone generative AI being able to create new training data for old systems. And that's something that we've seen be an increasingly prominent use case for our users, as much as things that we advise our customers and the community writ large to take advantage of. So ultimately those are kind of the traditional task types. I can give you some insight, maybe, into how I think about what's possible today, or five years or ten years as you sort go back. >> Yes, definitely. Let's get into that vision. >> So I kind of think about the types of use cases in terms of what's possible. If you just imagine a very simple bell curve, your normal distribution, for the longest time, the types of things that are in the center of that bell curve are identifying objects that are very common or common objects in context. Microsoft published the COCO Dataset in 2014 of common objects and contexts, of hundreds of thousands of images of chairs, forks, food, person, these sorts of things. And you know, the challenge of the day had always been, how do you identify just those 80 objects? So if we think about the bell curve, that'd be maybe the like dead center of the curve, where there's a lot of those objects present, and it's a very common thing that needs to be identified. But it's a very, very, very small sliver of the distribution. Now if you go out to the way long tail, let's go like deep into the tail of this imagined visual normal distribution, you're going to have a problem like one of our customers, Rivian, in tandem with AWS, is tackling, to do visual quality assurance and manufacturing in production processes. Now only Rivian knows what a Rivian is supposed to look like. Only they know the imagery of what their goods that are going to be produced are. And then between those long tails of proprietary data of highly specific things that need to be understood, in the center of the curve, you have a whole kind of messy middle, type of problems I like to say. The way I think about computer vision advancing, is it's basically you have larger and larger and more capable models that eat from the center out, right? So if you have a model that, you know, understands the 80 classes in COCO, well, pretty soon you have advances like Clip, which was trained on 400 million image text pairs, and has a greater understanding of a wider array of objects than just 80 classes in context. And over time you'll get more and more of these larger models that kind of eat outwards from that center of the distribution. And so the question becomes for companies, when can you rely on maybe a model that just already exists? How do you use your data to get what may be capable off the shelf, so to speak, into something that is usable for you? Or, if you're in those long tails and you have proprietary data, how do you take advantage of the greatest asset you have, which is observed visual information that you want to put to work for your customers, and you're kind of living in the long tails, and you need to adapt state of the art for your capabilities. So my mental model for like how computer vision advances is you have that bell curve, and you have increasingly powerful models that eat outward. And multimodality has a role to play in that, larger models have a role to play in that, more compute, more data generally has a role to play in that. But it will be a messy and I think long condition. >> Well, the thing I want to get, first of all, it's great, great mental model, I appreciate that, 'cause I think that makes a lot of sense. The question is, it seems now more than ever, with the scale and compute that's available, that not only can you eat out to the middle in your example, but there's other models you can integrate with. In the past there was siloed, static, almost bespoke. Now you're looking at larger models eating into the bell curve, as you said, but also integrating in with other stuff. So this seems to be part of that interaction. How does, first of all, is that really happening? Is that true? And then two, what does that mean for companies who want to take advantage of this? Because the old model was operational, you know? I have my cameras, they're watching stuff, whatever, and like now you're in this more of a, distributed computing, computer science mindset, not, you know, put the camera on the wall kind of- I'm oversimplifying, but you know what I'm saying. What's your take on that? >> Well, to the first point of, how are these advances happening? What I was kind of describing was, you know, almost uni-dimensional in that you have like, you're only thinking about vision, but the rise of generative techniques and multi-modality, like Clip is a multi-modal model, it has 400 million image text pairs. That will advance the generalizability at a faster rate than just treating everything as only vision. And that's kind of where LLMs and vision will intersect in a really nice and powerful way. Now in terms of like companies, how should they be thinking about taking advantage of these trends? The biggest thing that, and I think it's different, obviously, on the size of business, if you're an enterprise versus a startup. The biggest thing that I think if you're an enterprise, and you have an established scaled business model that is working for your customers, the question becomes, how do you take advantage of that established data moat, potentially, resource moats, and certainly, of course, establish a way of providing value to an end user. So for example, one of our customers, Walmart, has the advantage of one of the largest inventory and stock of any company in the world. And they also of course have substantial visual data, both from like their online catalogs, or understanding what's in stock or out of stock, or understanding, you know, the quality of things that they're going from the start of their supply chain to making it inside stores, for delivery of fulfillments. All these are are visual challenges. Now they already have a substantial trove of useful imagery to understand and teach and train large models to understand each of the individual SKUs and products that are in their stores. And so if I'm a Walmart, what I'm thinking is, how do I make sure that my petabytes of visual information is utilized in a way where I capture the proprietary benefit of the models that I can train to do tasks like, what item was this? Or maybe I'm going to create AmazonGo-like technology, or maybe I'm going to build like delivery robots, or I want to automatically know what's in and out of stock from visual input fees that I have across my in-store traffic. And that becomes the question and flavor of the day for enterprises. I've got this large amount of data, I've got an established way that I can provide more value to my own customers. How do I ensure I take advantage of the data advantage I'm already sitting on? If you're a startup, I think it's a pretty different question, and I'm happy to talk about. >> Yeah, what's startup angle on this? Because you know, they're going to want to take advantage. It's like cloud startups, cloud native startups, they were born in the cloud, they never had an IT department. So if you're a startup, is there a similar role here? And if I'm a computer vision startup, what's that mean? So can you share your your take on that, because there'll be a lot of people starting up from this. >> So the startup on the opposite advantage and disadvantage, right? Like a startup doesn't have an proven way of delivering repeatable value in the same way that a scaled enterprise does. But it does have the nimbleness to identify and take advantage of techniques that you can start from a blank slate. And I think the thing that startups need to be wary of in the generative AI enlarged language model, in multimodal world, is building what I like to call, kind of like sandcastles. A sandcastle is maybe a business model or a capability that's built on top of an assumption that is going to be pretty quickly wiped away by improving underlying model technology. So almost like if you imagine like the ocean, the waves are coming in, and they're going to wipe away your progress. You don't want to be in the position of building sandcastle business where, you don't want to bet on the fact that models aren't going to get good enough to solve the task type that you might be solving. In other words, don't take a screenshot of what's capable today. Assume that what's capable today is only going to continue to become possible. And so for a startup, what you can do, that like enterprises are quite comparatively less good at, is embedding these capabilities deeply within your products and delivering maybe a vertical based experience, where AI kind of exists in the background. >> Yeah. >> And we might not think of companies as, you know, even AI companies, it's just so embedded in the experience they provide, but that's like the vertical application example of taking AI and making it be immediately usable. Or, of course there's tons of picks and shovels businesses to be built like Roboflow, where you're enabling these enterprises to take advantage of something that they have, whether that's their data sets, their computes, or their intellect. >> Okay, so if I hear that right, by the way, I love, that's horizontally scalable, that's the large language models, go up and build them the apps, hence your developer focus. I'm sure that's probably the reason that the tsunami of developer's action. So you're saying picks and shovels tools, don't try to replicate the platform of what could be the platform. Oh, go to a VC, I'm going to build a platform. No, no, no, no, those are going to get wiped away by the large language models. Is there one large language model that will rule the world, or do you see many coming? >> Yeah, so to be clear, I think there will be useful platforms. I just think a lot of people think that they're building, let's say, you know, if we put this in the cloud context, you're building a specific type of EC2 instance. Well, it turns out that Amazon can offer that type of EC2 instance, and immediately distribute it to all of their customers. So you don't want to be in the position of just providing something that actually ends up looking like a feature, which in the context of AI, might be like a small incremental improvement on the model. If that's all you're doing, you're a sandcastle business. Now there's a lot of platform businesses that need to be built that enable businesses to get to value and do things like, how do I monitor my models? How do I create better models with my given data sets? How do I ensure that my models are doing what I want them to do? How do I find the right models to use? There's all these sorts of platform wide problems that certainly exist for businesses. I just think a lot of startups that I'm seeing right now are making the mistake of assuming the advances we're seeing are not going to accelerate or even get better. >> So if I'm a customer, if I'm a company, say I'm a startup or an enterprise, either one, same question. And I want to stand up, and I have developers working on stuff, I want to start standing up an environment to start doing stuff. Is that a service provider? Is that a managed service? Is that you guys? So how do you guys fit into your customers leaning in? Is it just for developers? Are you targeting with a specific like managed service? What's the product consumption? How do you talk to customers when they come to you? >> The thing that we do is enable, we give developers superpowers to build automated inventory tracking, self-checkout systems, identify if this image is malignant cancer or benign cancer, ensure that these products that I've produced are correct. Make sure that that the defect that might exist on this electric vehicle makes its way back for review. All these sorts of problems are immediately able to be solved and tackled. In terms of the managed services element, we have solutions as integrators that will often build on top of our tools, or we'll have companies that look to us for guidance, but ultimately the company is in control of developing and building and creating these capabilities in house. I really think the distinction is maybe less around managed service and tool, and more around ownership in the era of AI. So for example, if I'm using a managed service, in that managed service, part of their benefit is that they are learning across their customer sets, then it's a very different relationship than using a managed service where I'm developing some amount of proprietary advantages for my data sets. And I think that's a really important thing that companies are becoming attuned to, just the value of the data that they have. And so that's what we do. We tell companies that you have this proprietary, immense treasure trove of data, use that to your advantage, and think about us more like a set of tools that enable you to get value from that capability. You know, the HashiCorp's and GitLab's of the world have proven like what these businesses look like at scale. >> And you're targeting developers. When you go into a company, do you target developers with freemium, is there a paid service? Talk about the business model real quick. >> Sure, yeah. The tools are free to use and get started. When someone signs up for Roboflow, they may elect to make their work open source, in which case we're able to provide even more generous usage limits to basically move the computer vision community forward. If you elect to make your data private, you can use our hosted data set managing, data set training, model deployment, annotation tooling up to some limits. And then usually when someone validates that what they're doing gets them value, they purchase a subscription license to be able to scale up those capabilities. So like most developer centric products, it's free to get started, free to prove, free to poke around, develop what you think is possible. And then once you're getting to value, then we're able to capture the commercial upside in the value that's being provided. >> Love the business model. It's right in line with where the market is. There's kind of no standards bodies these days. The developers are the ones who are deciding kind of what the standards are by their adoption. I think making that easy for developers to get value as the model open sources continuing to grow, you can see more of that. Great perspective Joseph, thanks for sharing that. Put a plug in for the company. What are you guys doing right now? Where are you in your growth? What are you looking for? How should people engage? Give the quick commercial for the company. >> So as I mentioned, Roboflow is I think one of the largest, if not the largest collections of computer vision models and data sets that are open source, available on the web today, and have a private set of tools that over half the Fortune 100 now rely on those tools. So we're at the stage now where we know people want what we're working on, and we're continuing to drive that type of adoption. So companies that are looking to make better models, improve their data sets, train and deploy, often will get a lot of value from our tools, and certainly reach out to talk. I'm sure there's a lot of talented engineers that are tuning in too, we're aggressively hiring. So if you are interested in being a part of making the world programmable, and being at the ground floor of the company that's creating these capabilities to be writ large, we'd love to hear from you. >> Amazing, Joseph, thanks so much for coming on and being part of the AWS Startup Showcase. Man, if I was in my twenties, I'd be knocking on your door, because it's the hottest trend right now, it's super exciting. Generative AI is just the beginning of massive sea change. Congratulations on all your success, and we'll be following you guys. Thanks for spending the time, really appreciate it. >> Thanks for having me. >> Okay, this is season three, episode one of the ongoing series covering the exciting startups from the AWS ecosystem, talking about the hottest things in tech. I'm John Furrier, your host. Thanks for watching. (chill electronic music)

(upbeat music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase AI Machine Learning Top Startups Building Generative AI on AWS. This is season 3, episode 1 of the ongoing series covering the exciting startup from the AWS ecosystem to talk about AI and machine learning. I'm your host, John Furrier. I'm joined by two great guests here, Adam Wenchel, who's the CEO of Arthur, and Chief Scientist of Arthur, John Dickerson. Talk about how they help people build better LLM AI systems to get them into the market faster. Gentlemen, thank you for coming on. >> Yeah, thanks for having us, John. >> Well, I got to say I got to temper my enthusiasm because the last few months explosion of interest in LLMs with ChatGPT, has opened the eyes to everybody around the reality of that this is going next gen, this is it, this is the moment, this is the the point we're going to look back and say, this is the time where AI really hit the scene for real applications. So, a lot of Large Language Models, also known as LLMs, foundational models, and generative AI is all booming. This is where all the alpha developers are going. This is where everyone's focusing their business model transformations on. This is where developers are seeing action. So it's all happening, the wave is here. So I got to ask you guys, what are you guys seeing right now? You're in the middle of it, it's hitting you guys right on. You're in the front end of this massive wave. >> Yeah, John, I don't think you have to temper your enthusiasm at all. I mean, what we're seeing every single day is, everything from existing enterprise customers coming in with new ways that they're rethinking, like business things that they've been doing for many years that they can now do an entirely different way, as well as all manner of new companies popping up, applying LLMs to everything from generating code and SQL statements to generating health transcripts and just legal briefs. Everything you can imagine. And when you actually sit down and look at these systems and the demos we get of them, the hype is definitely justified. It's pretty amazing what they're going to do. And even just internally, we built, about a month ago in January, we built an Arthur chatbot so customers could ask questions, technical questions from our, rather than read our product documentation, they could just ask this LLM a particular question and get an answer. And at the time it was like state of the art, but then just last week we decided to rebuild it because the tooling has changed so much that we, last week, we've completely rebuilt it. It's now way better, built on an entirely different stack. And the tooling has undergone a full generation worth of change in six weeks, which is crazy. So it just tells you how much energy is going into this and how fast it's evolving right now. >> John, weigh in as a chief scientist. I mean, you must be blown away. Talk about kid in the candy store. I mean, you must be looking like this saying, I mean, she must be super busy to begin with, but the change, the acceleration, can you scope the kind of change you're seeing and be specific around the areas you're seeing movement and highly accelerated change? >> Yeah, definitely. And it is very, very exciting actually, thinking back to when ChatGPT was announced, that was a night our company was throwing an event at NeurIPS, which is maybe the biggest machine learning conference out there. And the hype when that happened was palatable and it was just shocking to see how well that performed. And then obviously over the last few months since then, as LLMs have continued to enter the market, we've seen use cases for them, like Adam mentioned all over the place. And so, some things I'm excited about in this space are the use of LLMs and more generally, foundation models to redesign traditional operations, research style problems, logistics problems, like auctions, decisioning problems. So moving beyond the already amazing news cases, like creating marketing content into more core integration and a lot of the bread and butter companies and tasks that drive the American ecosystem. And I think we're just starting to see some of that. And in the next 12 months, I think we're going to see a lot more. If I had to make other predictions, I think we're going to continue seeing a lot of work being done on managing like inference time costs via shrinking models or distillation. And I don't know how to make this prediction, but at some point we're going to be seeing lots of these very large scale models operating on the edge as well. So the time scales are extremely compressed, like Adam mentioned, 12 months from now, hard to say. >> We were talking on theCUBE prior to this session here. We had theCUBE conversation here and then the Wall Street Journal just picked up on the same theme, which is the printing press moment created the enlightenment stage of the history. Here we're in the whole nother automating intellect efficiency, doing heavy lifting, the creative class coming back, a whole nother level of reality around the corner that's being hyped up. The question is, is this justified? Is there really a breakthrough here or is this just another result of continued progress with AI? Can you guys weigh in, because there's two schools of thought. There's the, "Oh my God, we're entering a new enlightenment tech phase, of the equivalent of the printing press in all areas. Then there's, Ah, it's just AI (indistinct) inch by inch. What's your guys' opinion? >> Yeah, I think on the one hand when you're down in the weeds of building AI systems all day, every day, like we are, it's easy to look at this as an incremental progress. Like we have customers who've been building on foundation models since we started the company four years ago, particular in computer vision for classification tasks, starting with pre-trained models, things like that. So that part of it doesn't feel real new, but what does feel new is just when you apply these things to language with all the breakthroughs and computational efficiency, algorithmic improvements, things like that, when you actually sit down and interact with ChatGPT or one of the other systems that's out there that's building on top of LLMs, it really is breathtaking, like, the level of understanding that they have and how quickly you can accelerate your development efforts and get an actual working system in place that solves a really important real world problem and makes people way faster, way more efficient. So I do think there's definitely something there. It's more than just incremental improvement. This feels like a real trajectory inflection point for the adoption of AI. >> John, what's your take on this? As people come into the field, I'm seeing a lot of people move from, hey, I've been coding in Python, I've been doing some development, I've been a software engineer, I'm a computer science student. I'm coding in C++ old school, OG systems person. Where do they come in? Where's the focus, where's the action? Where are the breakthroughs? Where are people jumping in and rolling up their sleeves and getting dirty with this stuff? >> Yeah, all over the place. And it's funny you mentioned students in a different life. I wore a university professor hat and so I'm very, very familiar with the teaching aspects of this. And I will say toward Adam's point, this really is a leap forward in that techniques like in a co-pilot for example, everybody's using them right now and they really do accelerate the way that we develop. When I think about the areas where people are really, really focusing right now, tooling is certainly one of them. Like you and I were chatting about LangChain right before this interview started, two or three people can sit down and create an amazing set of pipes that connect different aspects of the LLM ecosystem. Two, I would say is in engineering. So like distributed training might be one, or just understanding better ways to even be able to train large models, understanding better ways to then distill them or run them. So like this heavy interaction now between engineering and what I might call traditional machine learning from 10 years ago where you had to know a lot of math, you had to know calculus very well, things like that. Now you also need to be, again, a very strong engineer, which is exciting. >> I interviewed Swami when he talked about the news. He's ahead of Amazon's machine learning and AI when they announced Hugging Face announcement. And I reminded him how Amazon was easy to get into if you were developing a startup back in 2007,8, and that the language models had that similar problem. It's step up a lot of content and a lot of expense to get provisioned up, now it's easy. So this is the next wave of innovation. So how do you guys see that from where we are right now? Are we at that point where it's that moment where it's that cloud-like experience for LLMs and large language models? >> Yeah, go ahead John. >> I think the answer is yes. We see a number of large companies that are training these and serving these, some of which are being co-interviewed in this episode. I think we're at that. Like, you can hit one of these with a simple, single line of Python, hitting an API, you can boot this up in seconds if you want. It's easy. >> Got it. >> So I (audio cuts out). >> Well let's take a step back and talk about the company. You guys being featured here on the Showcase. Arthur, what drove you to start the company? How'd this all come together? What's the origination story? Obviously you got a big customers, how'd get started? What are you guys doing? How do you make money? Give a quick overview. >> Yeah, I think John and I come at it from slightly different angles, but for myself, I have been a part of a number of technology companies. I joined Capital One, they acquired my last company and shortly after I joined, they asked me to start their AI team. And so even though I've been doing AI for a long time, I started my career back in DARPA. It was the first time I was really working at scale in AI at an organization where there were hundreds of millions of dollars in revenue at stake with the operation of these models and that they were impacting millions of people's financial livelihoods. And so it just got me hyper-focused on these issues around making sure that your AI worked well and it worked well for your company and it worked well for the people who were being affected by it. At the time when I was doing this 2016, 2017, 2018, there just wasn't any tooling out there to support this production management model monitoring life phase of the life cycle. And so we basically left to start the company that I wanted. And John has a his own story. I'll let let you share that one, John. >> Go ahead John, you're up. >> Yeah, so I'm coming at this from a different world. So I'm on leave now from a tenured role in academia where I was leading a large lab focusing on the intersection of machine learning and economics. And so questions like fairness or the response to the dynamism on the underlying environment have been around for quite a long time in that space. And so I've been thinking very deeply about some of those more like R and D style questions as well as having deployed some automation code across a couple of different industries, some in online advertising, some in the healthcare space and so on, where concerns of, again, fairness come to bear. And so Adam and I connected to understand the space of what that might look like in the 2018 20 19 realm from a quantitative and from a human-centered point of view. And so booted things up from there. >> Yeah, bring that applied engineering R and D into the Capital One, DNA that he had at scale. I could see that fit. I got to ask you now, next step, as you guys move out and think about LLMs and the recent AI news around the generative models and the foundational models like ChatGPT, how should we be looking at that news and everyone watching might be thinking the same thing. I know at the board level companies like, we should refactor our business, this is the future. It's that kind of moment, and the tech team's like, okay, boss, how do we do this again? Or are they prepared? How should we be thinking? How should people watching be thinking about LLMs? >> Yeah, I think they really are transformative. And so, I mean, we're seeing companies all over the place. Everything from large tech companies to a lot of our large enterprise customers are launching significant projects at core parts of their business. And so, yeah, I would be surprised, if you're serious about becoming an AI native company, which most leading companies are, then this is a trend that you need to be taking seriously. And we're seeing the adoption rate. It's funny, I would say the AI adoption in the broader business world really started, let's call it four or five years ago, and it was a relatively slow adoption rate, but I think all that kind of investment in and scaling the maturity curve has paid off because the rate at which people are adopting and deploying systems based on this is tremendous. I mean, this has all just happened in the few months and we're already seeing people get systems into production. So, now there's a lot of things you have to guarantee in order to put these in production in a way that basically is added into your business and doesn't cause more headaches than it solves. And so that's where we help customers is where how do you put these out there in a way that they're going to represent your company well, they're going to perform well, they're going to do their job and do it properly. >> So in the use case, as a customer, as I think about this, there's workflows. They might have had an ML AI ops team that's around IT. Their inference engines are out there. They probably don't have a visibility on say how much it costs, they're kicking the tires. When you look at the deployment, there's a cost piece, there's a workflow piece, there's fairness you mentioned John, what should be, I should be thinking about if I'm going to be deploying stuff into production, I got to think about those things. What's your opinion? >> Yeah, I'm happy to dive in on that one. So monitoring in general is extremely important once you have one of these LLMs in production, and there have been some changes versus traditional monitoring that we can dive deeper into that LLMs are really accelerated. But a lot of that bread and butter style of things you should be looking out for remain just as important as they are for what you might call traditional machine learning models. So the underlying environment of data streams, the way users interact with these models, these are all changing over time. And so any performance metrics that you care about, traditional ones like an accuracy, if you can define that for an LLM, ones around, for example, fairness or bias. If that is a concern for your particular use case and so on. Those need to be tracked. Now there are some interesting changes that LLMs are bringing along as well. So most ML models in production that we see are relatively static in the sense that they're not getting flipped in more than maybe once a day or once a week or they're just set once and then not changed ever again. With LLMs, there's this ongoing value alignment or collection of preferences from users that is often constantly updating the model. And so that opens up all sorts of vectors for, I won't say attack, but for problems to arise in production. Like users might learn to use your system in a different way and thus change the way those preferences are getting collected and thus change your system in ways that you never intended. So maybe that went through governance already internally at the company and now it's totally, totally changed and it's through no fault of your own, but you need to be watching over that for sure. >> Talk about the reinforced learnings from human feedback. How's that factoring in to the LLMs? Is that part of it? Should people be thinking about that? Is that a component that's important? >> It certainly is, yeah. So this is one of the big tweaks that happened with InstructGPT, which is the basis model behind ChatGPT and has since gone on to be used all over the place. So value alignment I think is through RLHF like you mentioned is a very interesting space to get into and it's one that you need to watch over. Like, you're asking humans for feedback over outputs from a model and then you're updating the model with respect to that human feedback. And now you've thrown humans into the loop here in a way that is just going to complicate things. And it certainly helps in many ways. You can ask humans to, let's say that you're deploying an internal chat bot at an enterprise, you could ask humans to align that LLM behind the chatbot to, say company values. And so you're listening feedback about these company values and that's going to scoot that chatbot that you're running internally more toward the kind of language that you'd like to use internally on like a Slack channel or something like that. Watching over that model I think in that specific case, that's a compliance and HR issue as well. So while it is part of the greater LLM stack, you can also view that as an independent bit to watch over. >> Got it, and these are important factors. When people see the Bing news, they freak out how it's doing great. Then it goes off the rails, it goes big, fails big. (laughing) So these models people see that, is that human interaction or is that feedback, is that not accepting it or how do people understand how to take that input in and how to build the right apps around LLMs? This is a tough question. >> Yeah, for sure. So some of the examples that you'll see online where these chatbots go off the rails are obviously humans trying to break the system, but some of them clearly aren't. And that's because these are large statistical models and we don't know what's going to pop out of them all the time. And even if you're doing as much in-house testing at the big companies like the Go-HERE's and the OpenAI's of the world, to try to prevent things like toxicity or racism or other sorts of bad content that might lead to bad pr, you're never going to catch all of these possible holes in the model itself. And so, again, it's very, very important to keep watching over that while it's in production. >> On the business model side, how are you guys doing? What's the approach? How do you guys engage with customers? Take a minute to explain the customer engagement. What do they need? What do you need? How's that work? >> Yeah, I can talk a little bit about that. So it's really easy to get started. It's literally a matter of like just handing out an API key and people can get started. And so we also offer alternative, we also offer versions that can be installed on-prem for models that, we find a lot of our customers have models that deal with very sensitive data. So you can run it in your cloud account or use our cloud version. And so yeah, it's pretty easy to get started with this stuff. We find people start using it a lot of times during the validation phase 'cause that way they can start baselining performance models, they can do champion challenger, they can really kind of baseline the performance of, maybe they're considering different foundation models. And so it's a really helpful tool for understanding differences in the way these models perform. And then from there they can just flow that into their production inferencing, so that as these systems are out there, you have really kind of real time monitoring for anomalies and for all sorts of weird behaviors as well as that continuous feedback loop that helps you make make your product get better and observability and you can run all sorts of aggregated reports to really understand what's going on with these models when they're out there deciding. I should also add that we just today have another way to adopt Arthur and that is we are in the AWS marketplace, and so we are available there just to make it that much easier to use your cloud credits, skip the procurement process, and get up and running really quickly. >> And that's great 'cause Amazon's got SageMaker, which handles a lot of privacy stuff, all kinds of cool things, or you can get down and dirty. So I got to ask on the next one, production is a big deal, getting stuff into production. What have you guys learned that you could share to folks watching? Is there a cost issue? I got to monitor, obviously you brought that up, we talked about the even reinforcement issues, all these things are happening. What is the big learnings that you could share for people that are going to put these into production to watch out for, to plan for, or be prepared for, hope for the best plan for the worst? What's your advice? >> I can give a couple opinions there and I'm sure Adam has. Well, yeah, the big one from my side is, again, I had mentioned this earlier, it's just the input data streams because humans are also exploring how they can use these systems to begin with. It's really, really hard to predict the type of inputs you're going to be seeing in production. Especially, we always talk about chatbots, but then any generative text tasks like this, let's say you're taking in news articles and summarizing them or something like that, it's very hard to get a good sampling even of the set of news articles in such a way that you can really predict what's going to pop out of that model. So to me, it's, adversarial maybe isn't the word that I would use, but it's an unnatural shifting input distribution of like prompts that you might see for these models. That's certainly one. And then the second one that I would talk about is, it can be hard to understand the costs, the inference time costs behind these LLMs. So the pricing on these is always changing as the models change size, it might go up, it might go down based on model size, based on energy cost and so on, but your pricing per token or per a thousand tokens and that I think can be difficult for some clients to wrap their head around. Again, you don't know how these systems are going to be used after all so it can be tough. And so again that's another metric that really should be tracked. >> Yeah, and there's a lot of trade off choices in there with like, how many tokens do you want at each step and in the sequence and based on, you have (indistinct) and you reject these tokens and so based on how your system's operating, that can make the cost highly variable. And that's if you're using like an API version that you're paying per token. A lot of people also choose to run these internally and as John mentioned, the inference time on these is significantly higher than a traditional classifi, even NLP classification model or tabular data model, like orders of magnitude higher. And so you really need to understand how that, as you're constantly iterating on these models and putting out new versions and new features in these models, how that's affecting the overall scale of that inference cost because you can use a lot of computing power very quickly with these profits. >> Yeah, scale, performance, price all come together. I got to ask while we're here on the secret sauce of the company, if you had to describe to people out there watching, what's the secret sauce of the company? What's the key to your success? >> Yeah, so John leads our research team and they've had a number of really cool, I think AI as much as it's been hyped for a while, it's still commercial AI at least is really in its infancy. And so the way we're able to pioneer new ways to think about performance for computer vision NLP LLMs is probably the thing that I'm proudest about. John and his team publish papers all the time at Navs and other places. But I think it's really being able to define what performance means for basically any kind of model type and give people really powerful tools to understand that on an ongoing basis. >> John, secret sauce, how would you describe it? You got all the action happening all around you. >> Yeah, well I going to appreciate Adam talking me up like that. No, I. (all laughing) >> Furrier: Robs to you. >> I would also say a couple of other things here. So we have a very strong engineering team and so I think some early hires there really set the standard at a very high bar that we've maintained as we've grown. And I think that's really paid dividends as scalabilities become even more of a challenge in these spaces, right? And so that's not just scalability when it comes to LLMs, that's scalability when it comes to millions of inferences per day, that kind of thing as well in traditional ML models. And I think that's compared to potential competitors, that's really... Well, it's made us able to just operate more efficiently and pass that along to the client. >> Yeah, and I think the infancy comment is really important because it's the beginning. You really is a long journey ahead. A lot of change coming, like I said, it's a huge wave. So I'm sure you guys got a lot of plannings at the foundation even for your own company, so I appreciate the candid response there. Final question for you guys is, what should the top things be for a company in 2023? If I'm going to set the agenda and I'm a customer moving forward, putting the pedal to the metal, so to speak, what are the top things I should be prioritizing or I need to do to be successful with AI in 2023? >> Yeah, I think, so number one, as we talked about, we've been talking about this entire episode, the things are changing so quickly and the opportunities for business transformation and really disrupting different applications, different use cases, is almost, I don't think we've even fully comprehended how big it is. And so really digging in to your business and understanding where I can apply these new sets of foundation models is, that's a top priority. The interesting thing is I think there's another force at play, which is the macroeconomic conditions and a lot of places are, they're having to work harder to justify budgets. So in the past, couple years ago maybe, they had a blank check to spend on AI and AI development at a lot of large enterprises that was limited primarily by the amount of talent they could scoop up. Nowadays these expenditures are getting scrutinized more. And so one of the things that we really help our customers with is like really calculating the ROI on these things. And so if you have models out there performing and you have a new version that you can put out that lifts the performance by 3%, how many tens of millions of dollars does that mean in business benefit? Or if I want to go to get approval from the CFO to spend a few million dollars on this new project, how can I bake in from the beginning the tools to really show the ROI along the way? Because I think in these systems when done well for a software project, the ROI can be like pretty spectacular. Like we see over a hundred percent ROI in the first year on some of these projects. And so, I think in 2023, you just need to be able to show what you're getting for that spend. >> It's a needle moving moment. You see it all the time with some of these aha moments or like, whoa, blown away. John, I want to get your thoughts on this because one of the things that comes up a lot for companies that I talked to, that are on my second wave, I would say coming in, maybe not, maybe the front wave of adopters is talent and team building. You mentioned some of the hires you got were game changing for you guys and set the bar high. As you move the needle, new developers going to need to come in. What's your advice given that you've been a professor, you've seen students, I know a lot of computer science people want to shift, they might not be yet skilled in AI, but they're proficient in programming, is that's going to be another opportunity with open source when things are happening. How do you talk to that next level of talent that wants to come in to this market to supplement teams and be on teams, lead teams? Any advice you have for people who want to build their teams and people who are out there and want to be a coder in AI? >> Yeah, I've advice, and this actually works for what it would take to be a successful AI company in 2023 as well, which is, just don't be afraid to iterate really quickly with these tools. The space is still being explored on what they can be used for. A lot of the tasks that they're used for now right? like creating marketing content using a machine learning is not a new thing to do. It just works really well now. And so I'm excited to see what the next year brings in terms of folks from outside of core computer science who are, other engineers or physicists or chemists or whatever who are learning how to use these increasingly easy to use tools to leverage LLMs for tasks that I think none of us have really thought about before. So that's really, really exciting. And so toward that I would say iterate quickly. Build things on your own, build demos, show them the friends, host them online and you'll learn along the way and you'll have somebody to show for it. And also you'll help us explore that space. >> Guys, congratulations with Arthur. Great company, great picks and shovels opportunities out there for everybody. Iterate fast, get in quickly and don't be afraid to iterate. Great advice and thank you for coming on and being part of the AWS showcase, thanks. >> Yeah, thanks for having us on John. Always a pleasure. >> Yeah, great stuff. Adam Wenchel, John Dickerson with Arthur. Thanks for coming on theCUBE. I'm John Furrier, your host. Generative AI and AWS. Keep it right there for more action with theCUBE. Thanks for watching. (upbeat music)

(upbeat music) >> Hello, everyone, and welcome to theCUBE's presentation of the "AWS Startup Showcase." This is season three, episode one. The focus of this episode is AI/ML: Top Startups Building Foundational Models, Infrastructure, and AI. It's great topics, super-relevant, and it's part of our ongoing coverage of startups in the AWS ecosystem. I'm your host, John Furrier, with theCUBE. Today, we're excited to be joined by Jay Marshall, VP of Business Development at Neural Magic. Jay, thanks for coming on theCUBE. >> Hey, John, thanks so much. Thanks for having us. >> We had a great CUBE conversation with you guys. This is very much about the company focuses. It's a feature presentation for the "Startup Showcase," and the machine learning at scale is the topic, but in general, it's more, (laughs) and we should call it "Machine Learning and AI: How to Get Started," because everybody is retooling their business. Companies that aren't retooling their business right now with AI first will be out of business, in my opinion. You're seeing massive shift. This is really truly the beginning of the next-gen machine learning AI trend. It's really seeing ChatGPT. Everyone sees that. That went mainstream. But this is just the beginning. This is scratching the surface of this next-generation AI with machine learning powering it, and with all the goodness of cloud, cloud scale, and how horizontally scalable it is. The resources are there. You got the Edge. Everything's perfect for AI 'cause data infrastructure's exploding in value. AI is just the applications. This is a super topic, so what do you guys see in this general area of opportunities right now in the headlines? And I'm sure you guys' phone must be ringing off the hook, metaphorically speaking, or emails and meetings and Zooms. What's going on over there at Neural Magic? >> No, absolutely, and you pretty much nailed most of it. I think that, you know, my background, we've seen for the last 20-plus years. Even just getting enterprise applications kind of built and delivered at scale, obviously, amazing things with AWS and the cloud to help accelerate that. And we just kind of figured out in the last five or so years how to do that productively and efficiently, kind of from an operations perspective. Got development and operations teams. We even came up with DevOps, right? But now, we kind of have this new kind of persona and new workload that developers have to talk to, and then it has to be deployed on those ITOps solutions. And so you pretty much nailed it. Folks are saying, "Well, how do I do this?" These big, generational models or foundational models, as we're calling them, they're great, but enterprises want to do that with their data, on their infrastructure, at scale, at the edge. So for us, yeah, we're helping enterprises accelerate that through optimizing models and then delivering them at scale in a more cost-effective fashion. >> Yeah, and I think one of the things, the benefits of OpenAI we saw, was not only is it open source, then you got also other models that are more proprietary, is that it shows the world that this is really happening, right? It's a whole nother level, and there's also new landscape kind of maps coming out. You got the generative AI, and you got the foundational models, large LLMs. Where do you guys fit into the landscape? Because you guys are in the middle of this. How do you talk to customers when they say, "I'm going down this road. I need help. I'm going to stand this up." This new AI infrastructure and applications, where do you guys fit in the landscape? >> Right, and really, the answer is both. I think today, when it comes to a lot of what for some folks would still be considered kind of cutting edge around computer vision and natural language processing, a lot of our optimization tools and our runtime are based around most of the common computer vision and natural language processing models. So your YOLOs, your BERTs, you know, your DistilBERTs and what have you, so we work to help optimize those, again, who've gotten great performance and great value for customers trying to get those into production. But when you get into the LLMs, and you mentioned some of the open source components there, our research teams have kind of been right in the trenches with those. So kind of the GPT open source equivalent being OPT, being able to actually take, you know, a multi-$100 billion parameter model and sparsify that or optimize that down, shaving away a ton of parameters, and being able to run it on smaller infrastructure. So I think the evolution here, you know, all this stuff came out in the last six months in terms of being turned loose into the wild, but we're staying in the trenches with folks so that we can help optimize those as well and not require, again, the heavy compute, the heavy cost, the heavy power consumption as those models evolve as well. So we're staying right in with everybody while they're being built, but trying to get folks into production today with things that help with business value today. >> Jay, I really appreciate you coming on theCUBE, and before we came on camera, you said you just were on a customer call. I know you got a lot of activity. What specific things are you helping enterprises solve? What kind of problems? Take us through the spectrum from the beginning, people jumping in the deep end of the pool, some people kind of coming in, starting out slow. What are the scale? Can you scope the kind of use cases and problems that are emerging that people are calling you for? >> Absolutely, so I think if I break it down to kind of, like, your startup, or I maybe call 'em AI native to kind of steal from cloud native years ago, that group, it's pretty much, you know, part and parcel for how that group already runs. So if you have a data science team and an ML engineering team, you're building models, you're training models, you're deploying models. You're seeing firsthand the expense of starting to try to do that at scale. So it's really just a pure operational efficiency play. They kind of speak natively to our tools, which we're doing in the open source. So it's really helping, again, with the optimization of the models they've built, and then, again, giving them an alternative to expensive proprietary hardware accelerators to have to run them. Now, on the enterprise side, it varies, right? You have some kind of AI native folks there that already have these teams, but you also have kind of, like, AI curious, right? Like, they want to do it, but they don't really know where to start, and so for there, we actually have an open source toolkit that can help you get into this optimization, and then again, that runtime, that inferencing runtime, purpose-built for CPUs. It allows you to not have to worry, again, about do I have a hardware accelerator available? How do I integrate that into my application stack? If I don't already know how to build this into my infrastructure, does my ITOps teams, do they know how to do this, and what does that runway look like? How do I cost for this? How do I plan for this? When it's just x86 compute, we've been doing that for a while, right? So it obviously still requires more, but at least it's a little bit more predictable. >> It's funny you mentioned AI native. You know, born in the cloud was a phrase that was out there. Now, you have startups that are born in AI companies. So I think you have this kind of cloud kind of vibe going on. You have lift and shift was a big discussion. Then you had cloud native, kind of in the cloud, kind of making it all work. Is there a existing set of things? People will throw on this hat, and then what's the difference between AI native and kind of providing it to existing stuff? 'Cause we're a lot of people take some of these tools and apply it to either existing stuff almost, and it's not really a lift and shift, but it's kind of like bolting on AI to something else, and then starting with AI first or native AI. >> Absolutely. It's a- >> How would you- >> It's a great question. I think that probably, where I'd probably pull back to kind of allow kind of retail-type scenarios where, you know, for five, seven, nine years or more even, a lot of these folks already have data science teams, you know? I mean, they've been doing this for quite some time. The difference is the introduction of these neural networks and deep learning, right? Those kinds of models are just a little bit of a paradigm shift. So, you know, I obviously was trying to be fun with the term AI native, but I think it's more folks that kind of came up in that neural network world, so it's a little bit more second nature, whereas I think for maybe some traditional data scientists starting to get into neural networks, you have the complexity there and the training overhead, and a lot of the aspects of getting a model finely tuned and hyperparameterization and all of these aspects of it. It just adds a layer of complexity that they're just not as used to dealing with. And so our goal is to help make that easy, and then of course, make it easier to run anywhere that you have just kind of standard infrastructure. >> Well, the other point I'd bring out, and I'd love to get your reaction to, is not only is that a neural network team, people who have been focused on that, but also, if you look at some of the DataOps lately, AIOps markets, a lot of data engineering, a lot of scale, folks who have been kind of, like, in that data tsunami cloud world are seeing, they kind of been in this, right? They're, like, been experiencing that. >> No doubt. I think it's funny the data lake concept, right? And you got data oceans now. Like, the metaphors just keep growing on us, but where it is valuable in terms of trying to shift the mindset, I've always kind of been a fan of some of the naming shift. I know with AWS, they always talk about purpose-built databases. And I always liked that because, you know, you don't have one database that can do everything. Even ones that say they can, like, you still have to do implementation detail differences. So sitting back and saying, "What is my use case, and then which database will I use it for?" I think it's kind of similar here. And when you're building those data teams, if you don't have folks that are doing data engineering, kind of that data harvesting, free processing, you got to do all that before a model's even going to care about it. So yeah, it's definitely a central piece of this as well, and again, whether or not you're going to be AI negative as you're making your way to kind of, you know, on that journey, you know, data's definitely a huge component of it. >> Yeah, you would have loved our Supercloud event we had. Talk about naming and, you know, around data meshes was talked about a lot. You're starting to see the control plane layers of data. I think that was the beginning of what I saw as that data infrastructure shift, to be horizontally scalable. So I have to ask you, with Neural Magic, when your customers and the people that are prospects for you guys, they're probably asking a lot of questions because I think the general thing that we see is, "How do I get started? Which GPU do I use?" I mean, there's a lot of things that are kind of, I won't say technical or targeted towards people who are living in that world, but, like, as the mainstream enterprises come in, they're going to need a playbook. What do you guys see, what do you guys offer your clients when they come in, and what do you recommend? >> Absolutely, and I think where we hook in specifically tends to be on the training side. So again, I've built a model. Now, I want to really optimize that model. And then on the runtime side when you want to deploy it, you know, we run that optimized model. And so that's where we're able to provide. We even have a labs offering in terms of being able to pair up our engineering teams with a customer's engineering teams, and we can actually help with most of that pipeline. So even if it is something where you have a dataset and you want some help in picking a model, you want some help training it, you want some help deploying that, we can actually help there as well. You know, there's also a great partner ecosystem out there, like a lot of folks even in the "Startup Showcase" here, that extend beyond into kind of your earlier comment around data engineering or downstream ITOps or the all-up MLOps umbrella. So we can absolutely engage with our labs, and then, of course, you know, again, partners, which are always kind of key to this. So you are spot on. I think what's happened with the kind of this, they talk about a hockey stick. This is almost like a flat wall now with the rate of innovation right now in this space. And so we do have a lot of folks wanting to go straight from curious to native. And so that's definitely where the partner ecosystem comes in so hard 'cause there just isn't anybody or any teams out there that, I literally do from, "Here's my blank database, and I want an API that does all the stuff," right? Like, that's a big chunk, but we can definitely help with the model to delivery piece. >> Well, you guys are obviously a featured company in this space. Talk about the expertise. A lot of companies are like, I won't say faking it till they make it. You can't really fake security. You can't really fake AI, right? So there's going to be a learning curve. They'll be a few startups who'll come out of the gate early. You guys are one of 'em. Talk about what you guys have as expertise as a company, why you're successful, and what problems do you solve for customers? >> No, appreciate that. Yeah, we actually, we love to tell the story of our founder, Nir Shavit. So he's a 20-year professor at MIT. Actually, he was doing a lot of work on kind of multicore processing before there were even physical multicores, and actually even did a stint in computational neurobiology in the 2010s, and the impetus for this whole technology, has a great talk on YouTube about it, where he talks about the fact that his work there, he kind of realized that the way neural networks encode and how they're executed by kind of ramming data layer by layer through these kind of HPC-style platforms, actually was not analogous to how the human brain actually works. So we're on one side, we're building neural networks, and we're trying to emulate neurons. We're not really executing them that way. So our team, which one of the co-founders, also an ex-MIT, that was kind of the birth of why can't we leverage this super-performance CPU platform, which has those really fat, fast caches attached to each core, and actually start to find a way to break that model down in a way that I can execute things in parallel, not having to do them sequentially? So it is a lot of amazing, like, talks and stuff that show kind of the magic, if you will, a part of the pun of Neural Magic, but that's kind of the foundational layer of all the engineering that we do here. And in terms of how we're able to bring it to reality for customers, I'll give one customer quote where it's a large retailer, and it's a people-counting application. So a very common application. And that customer's actually been able to show literally double the amount of cameras being run with the same amount of compute. So for a one-to-one perspective, two-to-one, business leaders usually like that math, right? So we're able to show pure cost savings, but even performance-wise, you know, we have some of the common models like your ResNets and your YOLOs, where we can actually even perform better than hardware-accelerated solutions. So we're trying to do, I need to just dumb it down to better, faster, cheaper, but from a commodity perspective, that's where we're accelerating. >> That's not a bad business model. Make things easier to use, faster, and reduce the steps it takes to do stuff. So, you know, that's always going to be a good market. Now, you guys have DeepSparse, which we've talked about on our CUBE conversation prior to this interview, delivers ML models through the software so the hardware allows for a decoupling, right? >> Yep. >> Which is going to drive probably a cost advantage. Also, it's also probably from a deployment standpoint it must be easier. Can you share the benefits? Is it a cost side? Is it more of a deployment? What are the benefits of the DeepSparse when you guys decouple the software from the hardware on the ML models? >> No you actually, you hit 'em both 'cause that really is primarily the value. Because ultimately, again, we're so early. And I came from this world in a prior life where I'm doing Java development, WebSphere, WebLogic, Tomcat open source, right? When we were trying to do innovation, we had innovation buckets, 'cause everybody wanted to be on the web and have their app and a browser, right? We got all the money we needed to build something and show, hey, look at the thing on the web, right? But when you had to get in production, that was the challenge. So to what you're speaking to here, in this situation, we're able to show we're just a Python package. So whether you just install it on the operating system itself, or we also have a containerized version you can drop on any container orchestration platform, so ECS or EKS on AWS. And so you get all the auto-scaling features. So when you think about that kind of a world where you have everything from real-time inferencing to kind of after hours batch processing inferencing, the fact that you can auto scale that hardware up and down and it's CPU based, so you're paying by the minute instead of maybe paying by the hour at a lower cost shelf, it does everything from pure cost to, again, I can have my standard IT team say, "Hey, here's the Kubernetes in the container," and it just runs on the infrastructure we're already managing. So yeah, operational, cost and again, and many times even performance. (audio warbles) CPUs if I want to. >> Yeah, so that's easier on the deployment too. And you don't have this kind of, you know, blank check kind of situation where you don't know what's on the backend on the cost side. >> Exactly. >> And you control the actual hardware and you can manage that supply chain. >> And keep in mind, exactly. Because the other thing that sometimes gets lost in the conversation, depending on where a customer is, some of these workloads, like, you know, you and I remember a world where even like the roundtrip to the cloud and back was a problem for folks, right? We're used to extremely low latency. And some of these workloads absolutely also adhere to that. But there's some workloads where the latency isn't as important. And we actually even provide the tuning. Now, if we're giving you five milliseconds of latency and you don't need that, you can tune that back. So less CPU, lower cost. Now, throughput and other things come into play. But that's the kind of configurability and flexibility we give for operations. >> All right, so why should I call you if I'm a customer or prospect Neural Magic, what problem do I have or when do I know I need you guys? When do I call you in and what does my environment look like? When do I know? What are some of the signals that would tell me that I need Neural Magic? >> No, absolutely. So I think in general, any neural network, you know, the process I mentioned before called sparcification, it's, you know, an optimization process that we specialize in. Any neural network, you know, can be sparcified. So I think if it's a deep-learning neural network type model. If you're trying to get AI into production, you have cost concerns even performance-wise. I certainly hate to be too generic and say, "Hey, we'll talk to everybody." But really in this world right now, if it's a neural network, it's something where you're trying to get into production, you know, we are definitely offering, you know, kind of an at-scale performant deployable solution for deep learning models. >> So neural network you would define as what? Just devices that are connected that need to know about each other? What's the state-of-the-art current definition of neural network for customers that may think they have a neural network or might not know they have a neural network architecture? What is that definition for neural network? >> That's a great question. So basically, machine learning models that fall under this kind of category, you hear about transformers a lot, or I mentioned about YOLO, the YOLO family of computer vision models, or natural language processing models like BERT. If you have a data science team or even developers, some even regular, I used to call myself a nine to five developer 'cause I worked in the enterprise, right? So like, hey, we found a new open source framework, you know, I used to use Spring back in the day and I had to go figure it out. There's developers that are pulling these models down and they're figuring out how to get 'em into production, okay? So I think all of those kinds of situations, you know, if it's a machine learning model of the deep learning variety that's, you know, really specifically where we shine. >> Okay, so let me pretend I'm a customer for a minute. I have all these videos, like all these transcripts, I have all these people that we've interviewed, CUBE alumnis, and I say to my team, "Let's AI-ify, sparcify theCUBE." >> Yep. >> What do I do? I mean, do I just like, my developers got to get involved and they're going to be like, "Well, how do I upload it to the cloud? Do I use a GPU?" So there's a thought process. And I think a lot of companies are going through that example of let's get on this AI, how can it help our business? >> Absolutely. >> What does that progression look like? Take me through that example. I mean, I made up theCUBE example up, but we do have a lot of data. We have large data models and we have people and connect to the internet and so we kind of seem like there's a neural network. I think every company might have a neural network in place. >> Well, and I was going to say, I think in general, you all probably do represent even the standard enterprise more than most. 'Cause even the enterprise is going to have a ton of video content, a ton of text content. So I think it's a great example. So I think that that kind of sea or I'll even go ahead and use that term data lake again, of data that you have, you're probably going to want to be setting up kind of machine learning pipelines that are going to be doing all of the pre-processing from kind of the raw data to kind of prepare it into the format that say a YOLO would actually use or let's say BERT for natural language processing. So you have all these transcripts, right? So we would do a pre-processing path where we would create that into the file format that BERT, the machine learning model would know how to train off of. So that's kind of all the pre-processing steps. And then for training itself, we actually enable what's called sparse transfer learning. So that's transfer learning is a very popular method of doing training with existing models. So we would be able to retrain that BERT model with your transcript data that we have now done the pre-processing with to get it into the proper format. And now we have a BERT natural language processing model that's been trained on your data. And now we can deploy that onto DeepSparse runtime so that now you can ask that model whatever questions, or I should say pass, you're not going to ask it those kinds of questions ChatGPT, although we can do that too. But you're going to pass text through the BERT model and it's going to give you answers back. It could be things like sentiment analysis or text classification. You just call the model, and now when you pass text through it, you get the answers better, faster or cheaper. I'll use that reference again. >> Okay, we can create a CUBE bot to give us questions on the fly from the the AI bot, you know, from our previous guests. >> Well, and I will tell you using that as an example. So I had mentioned OPT before, kind of the open source version of ChatGPT. So, you know, typically that requires multiple GPUs to run. So our research team, I may have mentioned earlier, we've been able to sparcify that over 50% already and run it on only a single GPU. And so in that situation, you could train OPT with that corpus of data and do exactly what you say. Actually we could use Alexa, we could use Alexa to actually respond back with voice. How about that? We'll do an API call and we'll actually have an interactive Alexa-enabled bot. >> Okay, we're going to be a customer, let's put it on the list. But this is a great example of what you guys call software delivered AI, a topic we chatted about on theCUBE conversation. This really means this is a developer opportunity. This really is the convergence of the data growth, the restructuring, how data is going to be horizontally scalable, meets developers. So this is an AI developer model going on right now, which is kind of unique. >> It is, John, I will tell you what's interesting. And again, folks don't always think of it this way, you know, the AI magical goodness is now getting pushed in the middle where the developers and IT are operating. And so it again, that paradigm, although for some folks seem obvious, again, if you've been around for 20 years, that whole all that plumbing is a thing, right? And so what we basically help with is when you deploy the DeepSparse runtime, we have a very rich API footprint. And so the developers can call the API, ITOps can run it, or to your point, it's developer friendly enough that you could actually deploy our off-the-shelf models. We have something called the SparseZoo where we actually publish pre-optimized or pre-sparcified models. And so developers could literally grab those right off the shelf with the training they've already had and just put 'em right into their applications and deploy them as containers. So yeah, we enable that for sure as well. >> It's interesting, DevOps was infrastructure as code and we had a last season, a series on data as code, which we kind of coined. This is data as code. This is a whole nother level of opportunity where developers just want to have programmable data and apps with AI. This is a whole new- >> Absolutely. >> Well, absolutely great, great stuff. Our news team at SiliconANGLE and theCUBE said you guys had a little bit of a launch announcement you wanted to make here on the "AWS Startup Showcase." So Jay, you have something that you want to launch here? >> Yes, and thank you John for teeing me up. So I'm going to try to put this in like, you know, the vein of like an AWS, like main stage keynote launch, okay? So we're going to try this out. So, you know, a lot of our product has obviously been built on top of x86. I've been sharing that the past 15 minutes or so. And with that, you know, we're seeing a lot of acceleration for folks wanting to run on commodity infrastructure. But we've had customers and prospects and partners tell us that, you know, ARM and all of its kind of variance are very compelling, both cost performance-wise and also obviously with Edge. And wanted to know if there was anything we could do from a runtime perspective with ARM. And so we got the work and, you know, it's a hard problem to solve 'cause the instructions set for ARM is very different than the instruction set for x86, and our deep tensor column technology has to be able to work with that lower level instruction spec. But working really hard, the engineering team's been at it and we are happy to announce here at the "AWS Startup Showcase," that DeepSparse inference now has, or inference runtime now has support for AWS Graviton instances. So it's no longer just x86, it is also ARM and that obviously also opens up the door to Edge and further out the stack so that optimize once run anywhere, we're not going to open up. So it is an early access. So if you go to neuralmagic.com/graviton, you can sign up for early access, but we're excited to now get into the ARM side of the fence as well on top of Graviton. >> That's awesome. Our news team is going to jump on that news. We'll get it right up. We get a little scoop here on the "Startup Showcase." Jay Marshall, great job. That really highlights the flexibility that you guys have when you decouple the software from the hardware. And again, we're seeing open source driving a lot more in AI ops now with with machine learning and AI. So to me, that makes a lot of sense. And congratulations on that announcement. Final minute or so we have left, give a summary of what you guys are all about. Put a plug in for the company, what you guys are looking to do. I'm sure you're probably hiring like crazy. Take the last few minutes to give a plug for the company and give a summary. >> No, I appreciate that so much. So yeah, joining us out neuralmagic.com, you know, part of what we didn't spend a lot of time here, our optimization tools, we are doing all of that in the open source. It's called SparseML and I mentioned SparseZoo briefly. So we really want the data scientists community and ML engineering community to join us out there. And again, the DeepSparse runtime, it's actually free to use for trial purposes and for personal use. So you can actually run all this on your own laptop or on an AWS instance of your choice. We are now live in the AWS marketplace. So push button, deploy, come try us out and reach out to us on neuralmagic.com. And again, sign up for the Graviton early access. >> All right, Jay Marshall, Vice President of Business Development Neural Magic here, talking about performant, cost effective machine learning at scale. This is season three, episode one, focusing on foundational models as far as building data infrastructure and AI, AI native. I'm John Furrier with theCUBE. Thanks for watching. (bright upbeat music)

(soft music) >> Hello, everyone. Welcome to theCUBE's presentation of the AWS Startup Showcase. AI and Machine Learning: Top Startups Building Foundational Model Infrastructure. This is season 3, episode 1 of the ongoing series covering the exciting stuff from the AWS ecosystem, talking about machine learning and AI. I'm your host, John Furrier and today we are excited to be joined by Luis Ceze who's the CEO of OctoML and Anna Connolly, VP of customer success and experience OctoML. Great to have you on again, Luis. Anna, thanks for coming on. Appreciate it. >> Thank you, John. It's great to be here. >> Thanks for having us. >> I love the company. We had a CUBE conversation about this. You guys are really addressing how to run foundational models faster for less. And this is like the key theme. But before we get into it, this is a hot trend, but let's explain what you guys do. Can you set the narrative of what the company's about, why it was founded, what's your North Star and your mission? >> Yeah, so John, our mission is to make AI sustainable and accessible for everyone. And what we offer customers is, you know, a way of taking their models into production in the most efficient way possible by automating the process of getting a model and optimizing it for a variety of hardware and making cost-effective. So better, faster, cheaper model deployment. >> You know, the big trend here is AI. Everyone's seeing the ChatGPT, kind of the shot heard around the world. The BingAI and this fiasco and the ongoing experimentation. People are into it, and I think the business impact is clear. I haven't seen this in all of my career in the technology industry of this kind of inflection point. And every senior leader I talk to is rethinking about how to rebuild their business with AI because now the large language models have come in, these foundational models are here, they can see value in their data. This is a 10 year journey in the big data world. Now it's impacting that, and everyone's rebuilding their company around this idea of being AI first 'cause they see ways to eliminate things and make things more efficient. And so now they telling 'em to go do it. And they're like, what do we do? So what do you guys think? Can you explain what is this wave of AI and why is it happening, why now, and what should people pay attention to? What does it mean to them? >> Yeah, I mean, it's pretty clear by now that AI can do amazing things that captures people's imaginations. And also now can show things that are really impactful in businesses, right? So what people have the opportunity to do today is to either train their own model that adds value to their business or find open models out there that can do very valuable things to them. So the next step really is how do you take that model and put it into production in a cost-effective way so that the business can actually get value out of it, right? >> Anna, what's your take? Because customers are there, you're there to make 'em successful, you got the new secret weapon for their business. >> Yeah, I think we just see a lot of companies struggle to get from a trained model into a model that is deployed in a cost-effective way that actually makes sense for the application they're building. I think that's a huge challenge we see today, kind of across the board across all of our customers. >> Well, I see this, everyone asking the same question. I have data, I want to get value out of it. I got to get these big models, I got to train it. What's it going to cost? So I think there's a reality of, okay, I got to do it. Then no one has any visibility on what it costs. When they get into it, this is going to break the bank. So I have to ask you guys, the cost of training these models is on everyone's mind. OctoML, your company's focus on the cost side of it as well as the efficiency side of running these models in production. Why are the production costs such a concern and where specifically are people looking at it and why did it get here? >> Yeah, so training costs get a lot of attention because normally a large number, but we shouldn't forget that it's a large, typically one time upfront cost that customers pay. But, you know, when the model is put into production, the cost grows directly with model usage and you actually want your model to be used because it's adding value, right? So, you know, the question that a customer faces is, you know, they have a model, they have a trained model and now what? So how much would it cost to run in production, right? And now without the big wave in generative AI, which rightfully is getting a lot of attention because of the amazing things that it can do. It's important for us to keep in mind that generative AI models like ChatGPT are huge, expensive energy hogs. They cost a lot to run, right? And given that model usage growth directly, model cost grows directly with usage, what you want to do is make sure that once you put a model into production, you have the best cost structure possible so that you're not surprised when it's gets popular, right? So let me give you an example. So if you have a model that costs, say 1 to $2 million to train, but then it costs about one to two cents per session to use it, right? So if you have a million active users, even if they use just once a day, it's 10 to $20,000 a day to operate that model in production. And that very, very quickly, you know, get beyond what you paid to train it. >> Anna, these aren't small numbers, and it's cost to train and cost to operate, it kind of reminds me of when the cloud came around and the data center versus cloud options. Like, wait a minute, one, it costs a ton of cash to deploy, and then running it. This is kind of a similar dynamic. What are you seeing? >> Yeah, absolutely. I think we are going to see increasingly the cost and production outpacing the costs and training by a lot. I mean, people talk about training costs now because that's what they're confronting now because people are so focused on getting models performant enough to even use in an application. And now that we have them and they're that capable, we're really going to start to see production costs go up a lot. >> Yeah, Luis, if you don't mind, I know this might be a little bit of a tangent, but, you know, training's super important. I get that. That's what people are doing now, but then there's the deployment side of production. Where do people get caught up and miss the boat or misconfigure? What's the gotcha? Where's the trip wire or so to speak? Where do people mess up on the cost side? What do they do? Is it they don't think about it, they tie it to proprietary hardware? What's the issue? >> Yeah, several things, right? So without getting really technical, which, you know, I might get into, you know, you have to understand relationship between performance, you know, both in terms of latency and throughput and cost, right? So reducing latency is important because you improve responsiveness of the model. But it's really important to keep in mind that it often leads diminishing returns. Below a certain latency, making it faster won't make a measurable difference in experience, but it's going to cost a lot more. So understanding that is important. Now, if you care more about throughputs, which is the time it takes for you to, you know, units per period of time, you care about time to solution, we should think about this throughput per dollar. And understand what you want is the highest throughput per dollar, which may come at the cost of higher latency, which you're not going to care about, right? So, and the reality here, John, is that, you know, humans and especially folks in this space want to have the latest and greatest hardware. And often they commit a lot of money to get access to them and have to commit upfront before they understand the needs that their models have, right? So common mistake here, one is not spending time to understand what you really need, and then two, over-committing and using more hardware than you actually need. And not giving yourself enough freedom to get your workload to move around to the more cost-effective choice, right? So this is just a metaphoric choice. And then another thing that's important here too is making a model run faster on the hardware directly translates to lower cost, right? So, but it takes a lot of engineers, you need to think of ways of producing very efficient versions of your model for the target hardware that you're going to use. >> Anna, what's the customer angle here? Because price performance has been around for a long time, people get that, but now latency and throughput, that's key because we're starting to see this in apps. I mean, there's an end user piece. I even seeing it on the infrastructure side where they're taking a heavy lifting away from operational costs. So you got, you know, application specific to the user and/or top of the stack, and then you got actually being used in operations where they want both. >> Yeah, absolutely. Maybe I can illustrate this with a quick story with the customer that we had recently been working with. So this customer is planning to run kind of a transformer based model for tech generation at super high scale on Nvidia T4 GPU, so kind of a commodity GPU. And the scale was so high that they would've been paying hundreds of thousands of dollars in cloud costs per year just to serve this model alone. You know, one of many models in their application stack. So we worked with this team to optimize our model and then benchmark across several possible targets. So that matching the hardware that Luis was just talking about, including the newer kind of Nvidia A10 GPUs. And what they found during this process was pretty interesting. First, the team was able to shave a quarter of their spend just by using better optimization techniques on the T4, the older hardware. But actually moving to a newer GPU would allow them to serve this model in a sub two milliseconds latency, so super fast, which was able to unlock an entirely new kind of user experience. So they were able to kind of change the value they're delivering in their application just because they were able to move to this new hardware easily. So they ultimately decided to plan their deployment on the more expensive A10 because of this, but because of the hardware specific optimizations that we helped them with, they managed to even, you know, bring costs down from what they had originally planned. And so if you extend this kind of example to everything that's happening with generative AI, I think the story we just talked about was super relevant, but the scale can be even higher, you know, it can be tenfold that. We were recently conducting kind of this internal study using GPT-J as a proxy to illustrate the experience of just a company trying to use one of these large language models with an example scenario of creating a chatbot to help job seekers prepare for interviews. So if you imagine kind of a conservative usage scenario where the model generates just 3000 words per user per day, which is, you know, pretty conservative for how people are interacting with these models. It costs 5 cents a session and if you're a company and your app goes viral, so from, you know, beginning of the year there's nobody, at the end of the year there's a million daily active active users in that year alone, going from zero to a million. You'll be spending about $6 million a year, which is pretty unmanageable. That's crazy, right? >> Yeah. >> For a company or a product that's just launching. So I think, you know, for us we see the real way to make these kind of advancements accessible and sustainable, as we said is to bring down cost to serve using these techniques. >> That's a great story and I think that illustrates this idea that deployment cost can vary from situation to situation, from model to model and that the efficiency is so strong with this new wave, it eliminates heavy lifting, creates more efficiency, automates intellect. I mean, this is the trend, this is radical, this is going to increase. So the cost could go from nominal to millions, literally, potentially. So, this is what customers are doing. Yeah, that's a great story. What makes sense on a financial, is there a cost of ownership? Is there a pattern for best practice for training? What do you guys advise cuz this is a lot of time and money involved in all potential, you know, good scenarios of upside. But you can get over your skis as they say, and be successful and be out of business if you don't manage it. I mean, that's what people are talking about, right? >> Yeah, absolutely. I think, you know, we see kind of three main vectors to reduce cost. I think one is make your deployment process easier overall, so that your engineering effort to even get your app running goes down. Two, would be get more from the compute you're already paying for, you're already paying, you know, for your instances in the cloud, but can you do more with that? And then three would be shop around for lower cost hardware to match your use case. So on the first one, I think making the deployment easier overall, there's a lot of manual work that goes into benchmarking, optimizing and packaging models for deployment. And because the performance of machine learning models can be really hardware dependent, you have to go through this process for each target you want to consider running your model on. And this is hard, you know, we see that every day. But for teams who want to incorporate some of these large language models into their applications, it might be desirable because licensing a model from a large vendor like OpenAI can leave you, you know, over provision, kind of paying for capabilities you don't need in your application or can lock you into them and you lose flexibility. So we have a customer whose team actually prepares models for deployment in a SaaS application that many of us use every day. And they told us recently that without kind of an automated benchmarking and experimentation platform, they were spending several days each to benchmark a single model on a single hardware type. So this is really, you know, manually intensive and then getting more from the compute you're already paying for. We do see customers who leave money on the table by running models that haven't been optimized specifically for the hardware target they're using, like Luis was mentioning. And for some teams they just don't have the time to go through an optimization process and for others they might lack kind of specialized expertise and this is something we can bring. And then on shopping around for different hardware types, we really see a huge variation in model performance across hardware, not just CPU vs. GPU, which is, you know, what people normally think of. But across CPU vendors themselves, high memory instances and across cloud providers even. So the best strategy here is for teams to really be able to, we say, look before you leap by running real world benchmarking and not just simulations or predictions to find the best software, hardware combination for their workload. >> Yeah. You guys sound like you have a very impressive customer base deploying large language models. Where would you categorize your current customer base? And as you look out, as you guys are growing, you have new customers coming in, take me through the progression. Take me through the profile of some of your customers you have now, size, are they hyperscalers, are they big app folks, are they kicking the tires? And then as people are out there scratching heads, I got to get in this game, what's their psychology like? Are they coming in with specific problems or do they have specific orientation point of view about what they want to do? Can you share some data around what you're seeing? >> Yeah, I think, you know, we have customers that kind of range across the spectrum of sophistication from teams that basically don't have MLOps expertise in their company at all. And so they're really looking for us to kind of give a full service, how should I do everything from, you know, optimization, find the hardware, prepare for deployment. And then we have teams that, you know, maybe already have their serving and hosting infrastructure up and ready and they already have models in production and they're really just looking to, you know, take the extra juice out of the hardware and just do really specific on that optimization piece. I think one place where we're doing a lot more work now is kind of in the developer tooling, you know, model selection space. And that's kind of an area that we're creating more tools for, particularly within the PyTorch ecosystem to bring kind of this power earlier in the development cycle so that as people are grabbing a model off the shelf, they can, you know, see how it might perform and use that to inform their development process. >> Luis, what's the big, I like this idea of picking the models because isn't that like going to the market and picking the best model for your data? It's like, you know, it's like, isn't there a certain approaches? What's your view on this? 'Cause this is where everyone, I think it's going to be a land rush for this and I want to get your thoughts. >> For sure, yeah. So, you know, I guess I'll start with saying the one main takeaway that we got from the GPT-J study is that, you know, having a different understanding of what your model's compute and memory requirements are, very quickly, early on helps with the much smarter AI model deployments, right? So, and in fact, you know, Anna just touched on this, but I want to, you know, make sure that it's clear that OctoML is putting that power into user's hands right now. So in partnership with AWS, we are launching this new PyTorch native profiler that allows you with a single, you know, one line, you know, code decorator allows you to see how your code runs on a variety of different hardware after accelerations. So it gives you very clear, you know, data on how you should think about your model deployments. And this ties back to choices of models. So like, if you have a set of choices that are equally good of models in terms of functionality and you want to understand after acceleration how are you going to deploy, how much they're going to cost or what are the options using a automated process of making a decision is really, really useful. And in fact, so I think these events can get early access to this by signing up for the Octopods, you know, this is exclusive group for insiders here, so you can go to OctoML.ai/pods to sign up. >> So that Octopod, is that a program? What is that, is that access to code? Is that a beta, what is that? Explain, take a minute and explain Octopod. >> I think the Octopod would be a group of people who is interested in experiencing this functionality. So it is the friends and users of OctoML that would be the Octopod. And then yes, after you sign up, we would provide you essentially the tool in code form for you to try out in your own. I mean, part of the benefit of this is that it happens in your own local environment and you're in control of everything kind of within the workflow that developers are already using to create and begin putting these models into their applications. So it would all be within your control. >> Got it. I think the big question I have for you is when do you, when does that one of your customers know they need to call you? What's their environment look like? What are they struggling with? What are the conversations they might be having on their side of the fence? If anyone's watching this, they're like, "Hey, you know what, I've got my team, we have a lot of data. Do we have our own language model or do I use someone else's?" There's a lot of this, I will say discovery going on around what to do, what path to take, what does that customer look like, if someone's listening, when do they know to call you guys, OctoML? >> Well, I mean the most obvious one is that you have a significant spend on AI/ML, come and talk to us, you know, putting AIML into production. So that's the clear one. In fact, just this morning I was talking to someone who is in life sciences space and is having, you know, 15 to $20 million a year cloud related to AI/ML deployment is a clear, it's a pretty clear match right there, right? So that's on the cost side. But I also want to emphasize something that Anna said earlier that, you know, the hardware and software complexity involved in putting model into production is really high. So we've been able to abstract that away, offering a clean automation flow enables one, to experiment early on, you know, how models would run and get them to production. And then two, once they are into production, gives you an automated flow to continuously updating your model and taking advantage of all this acceleration and ability to run the model on the right hardware. So anyways, let's say one then is cost, you know, you have significant cost and then two, you have an automation needs. And Anna please compliment that. >> Yeah, Anna you can please- >> Yeah, I think that's exactly right. Maybe the other time is when you are expecting a big scale up in serving your application, right? You're launching a new feature, you expect to get a lot of usage or, and you want to kind of anticipate maybe your CTO, your CIO, whoever pays your cloud bills is going to come after you, right? And so they want to know, you know, what's the return on putting this model essentially into my application stack? Am I going to, is the usage going to match what I'm paying for it? And then you can understand that. >> So you guys have a lot of the early adopters, they got big data teams, they're pushed in the production, they want to get a little QA, test the waters, understand, use your technology to figure it out. Is there any cases where people have gone into production, they have to pull it out? It's like the old lemon laws with your car, you buy a car and oh my god, it's not the way I wanted it. I mean, I can imagine the early people through the wall, so to speak, in the wave here are going to be bloody in the sense that they've gone in and tried stuff and get stuck with huge bills. Are you seeing that? Are people pulling stuff out of production and redeploying? Or I can imagine that if I had a bad deployment, I'd want to refactor that or actually replatform that. Do you see that too? >> Definitely after a sticker shock, yes, your customers will come and make sure that, you know, the sticker shock won't happen again. >> Yeah. >> But then there's another more thorough aspect here that I think we likely touched on, be worth elaborating a bit more is just how are you going to scale in a way that's feasible depending on the allocation that you get, right? So as we mentioned several times here, you know, model deployment is so hardware dependent and so complex that you tend to get a model for a hardware choice and then you want to scale that specific type of instance. But what if, when you want to scale because suddenly luckily got popular and, you know, you want to scale it up and then you don't have that instance anymore. So how do you live with whatever you have at that moment is something that we see customers needing as well. You know, so in fact, ideally what we want is customers to not think about what kind of specific instances they want. What they want is to know what their models need. Say, they know the SLA and then find a set of hybrid targets and instances that hit the SLA whenever they're also scaling, they're going to scale with more freedom, right? Instead of having to wait for AWS to give them more specific allocation for a specific instance. What if you could live with other types of hardware and scale up in a more free way, right? So that's another thing that we see customers, you know, like they need more freedom to be able to scale with whatever is available. >> Anna, you touched on this with the business model impact to that 6 million cost, if that goes out of control, there's a business model aspect and there's a technical operation aspect to the cost side too. You want to be mindful of riding the wave in a good way, but not getting over your skis. So that brings up the point around, you know, confidence, right? And teamwork. Because if you're in production, there's probably a team behind it. Talk about the team aspect of your customers. I mean, they're dedicated, they go put stuff into production, they're developers, there're data. What's in it for them? Are they getting better, are they in the beach, you know, reading the book. Are they, you know, are there easy street for them? What's the customer benefit to the teams? >> Yeah, absolutely. With just a few clicks of a button, you're in production, right? That's the dream. So yeah, I mean I think that, you know, we illustrated it before a little bit. I think the automated kind of benchmarking and optimization process, like when you think about the effort it takes to get that data by hand, which is what people are doing today, they just don't do it. So they're making decisions without the best information because it's, you know, there just isn't the bandwidth to get the information that they need to make the best decision and then know exactly how to deploy it. So I think it's actually bringing kind of a new insight and capability to these teams that they didn't have before. And then maybe another aspect on the team side is that it's making the hand-off of the models from the data science teams to the model deployment teams more seamless. So we have, you know, we have seen in the past that this kind of transition point is the place where there are a lot of hiccups, right? The data science team will give a model to the production team and it'll be too slow for the application or it'll be too expensive to run and it has to go back and be changed and kind of this loop. And so, you know, with the PyTorch profiler that Luis was talking about, and then also, you know, the other ways we do optimization that kind of prevents that hand-off problem from happening. >> Luis and Anna, you guys have a great company. Final couple minutes left. Talk about the company, the people there, what's the culture like, you know, if Intel has Moore's law, which is, you know, doubling the performance in few years, what's the culture like there? Is it, you know, more throughput, better pricing? Explain what's going on with the company and put a plug in. Luis, we'll start with you. >> Yeah, absolutely. I'm extremely proud of the team that we built here. You know, we have a people first culture, you know, very, very collaborative and folks, we all have a shared mission here of making AI more accessible and sustainable. We have a very diverse team in terms of backgrounds and life stories, you know, to do what we do here, we need a team that has expertise in software engineering, in machine learning, in computer architecture. Even though we don't build chips, we need to understand how they work, right? So, and then, you know, the fact that we have this, this very really, really varied set of backgrounds makes the environment, you know, it's say very exciting to learn more about, you know, assistance end-to-end. But also makes it for a very interesting, you know, work environment, right? So people have different backgrounds, different stories. Some of them went to grad school, others, you know, were in intelligence agencies and now are working here, you know. So we have a really interesting set of people and, you know, life is too short not to work with interesting humans. You know, that's something that I like to think about, you know. >> I'm sure your off-site meetings are a lot of fun, people talking about computer architectures, silicon advances, the next GPU, the big data models coming in. Anna, what's your take? What's the culture like? What's the company vibe and what are you guys looking to do? What's the customer success pattern? What's up? >> Yeah, absolutely. I mean, I, you know, second all of the great things that Luis just said about the team. I think one that I, an additional one that I'd really like to underscore is kind of this customer obsession, to use a term you all know well. And focus on the end users and really making the experiences that we're bringing to our user who are developers really, you know, useful and valuable for them. And so I think, you know, all of these tools that we're trying to put in the hands of users, the industry and the market is changing so rapidly that our products across the board, you know, all of the companies that, you know, are part of the showcase today, we're all evolving them so quickly and we can only do that kind of really hand in glove with our users. So that would be another thing I'd emphasize. >> I think the change dynamic, the power dynamics of this industry is just the beginning. I'm very bullish that this is going to be probably one of the biggest inflection points in history of the computer industry because of all the dynamics of the confluence of all the forces, which you mentioned some of them, I mean PC, you know, interoperability within internetworking and you got, you know, the web and then mobile. Now we have this, I mean, I wouldn't even put social media even in the close to this. Like, this is like, changes user experience, changes infrastructure. There's going to be massive accelerations in performance on the hardware side from AWS's of the world and cloud and you got the edge and more data. This is really what big data was going to look like. This is the beginning. Final question, what do you guys see going forward in the future? >> Well, it's undeniable that machine learning and AI models are becoming an integral part of an interesting application today, right? So, and the clear trends here are, you know, more and more competitional needs for these models because they're only getting more and more powerful. And then two, you know, seeing the complexity of the infrastructure where they run, you know, just considering the cloud, there's like a wide variety of choices there, right? So being able to live with that and making the most out of it in a way that does not require, you know, an impossible to find team is something that's pretty clear. So the need for automation, abstracting with the complexity is definitely here. And we are seeing this, you know, trends are that you also see models starting to move to the edge as well. So it's clear that we're seeing, we are going to live in a world where there's no large models living in the cloud. And then, you know, edge models that talk to these models in the cloud to form, you know, an end-to-end truly intelligent application. >> Anna? >> Yeah, I think, you know, our, Luis said it at the beginning. Our vision is to make AI sustainable and accessible. And I think as this technology just expands in every company and every team, that's going to happen kind of on its own. And we're here to help support that. And I think you can't do that without tools like those like OctoML. >> I think it's going to be an error of massive invention, creativity, a lot of the format heavy lifting is going to allow the talented people to automate their intellect. I mean, this is really kind of what we see going on. And Luis, thank you so much. Anna, thanks for coming on this segment. Thanks for coming on theCUBE and being part of the AWS Startup Showcase. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(upbeat music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase AI/ML Top Startups Building Foundation Model Infrastructure. This is season three, episode one of our ongoing series covering exciting startups from the AWS ecosystem to talk about data and analytics. I'm your host, Lisa Martin and today we're excited to be joined by two guests from Astronomer. Steven Hillion joins us, it's Chief Data Officer and Jeff Fletcher, it's director of ML. They're here to talk about machine learning and data orchestration. Guys, thank you so much for joining us today. >> Thank you. >> It's great to be here. >> Before we get into machine learning let's give the audience an overview of Astronomer. Talk about what that is, Steven. Talk about what you mean by data orchestration. >> Yeah, let's start with Astronomer. We're the Airflow company basically. The commercial developer behind the open-source project, Apache Airflow. I don't know if you've heard of Airflow. It's sort of de-facto standard these days for orchestrating data pipelines, data engineering pipelines, and as we'll talk about later, machine learning pipelines. It's really is the de-facto standard. I think we're up to about 12 million downloads a month. That's actually as a open-source project. I think at this point it's more popular by some measures than Slack. Airflow was created by Airbnb some years ago to manage all of their data pipelines and manage all of their workflows and now it powers the data ecosystem for organizations as diverse as Electronic Arts, Conde Nast is one of our big customers, a big user of Airflow. And also not to mention the biggest banks on Wall Street use Airflow and Astronomer to power the flow of data throughout their organizations. >> Talk about that a little bit more, Steven, in terms of the business impact. You mentioned some great customer names there. What is the business impact or outcomes that a data orchestration strategy enables businesses to achieve? >> Yeah, I mean, at the heart of it is quite simply, scheduling and managing data pipelines. And so if you have some enormous retailer who's managing the flow of information throughout their organization they may literally have thousands or even tens of thousands of data pipelines that need to execute every day to do things as simple as delivering metrics for the executives to consume at the end of the day, to producing on a weekly basis new machine learning models that can be used to drive product recommendations. One of our customers, for example, is a British food delivery service. And you get those recommendations in your application that says, "Well, maybe you want to have samosas with your curry." That sort of thing is powered by machine learning models that they train on a regular basis to reflect changing conditions in the market. And those are produced through Airflow and through the Astronomer platform, which is essentially a managed platform for running airflow. So at its simplest it really is just scheduling and managing those workflows. But that's easier said than done of course. I mean if you have 10 thousands of those things then you need to make sure that they all run that they all have sufficient compute resources. If things fail, how do you track those down across those 10,000 workflows? How easy is it for an average data scientist or data engineer to contribute their code, their Python notebooks or their SQL code into a production environment? And then you've got reproducibility, governance, auditing, like managing data flows across an organization which we think of as orchestrating them is much more than just scheduling. It becomes really complicated pretty quickly. >> I imagine there's a fair amount of complexity there. Jeff, let's bring you into the conversation. Talk a little bit about Astronomer through your lens, data orchestration and how it applies to MLOps. >> So I come from a machine learning background and for me the interesting part is that machine learning requires the expansion into orchestration. A lot of the same things that you're using to go and develop and build pipelines in a standard data orchestration space applies equally well in a machine learning orchestration space. What you're doing is you're moving data between different locations, between different tools, and then tasking different types of tools to act on that data. So extending it made logical sense from a implementation perspective. And a lot of my focus at Astronomer is really to explain how Airflow can be used well in a machine learning context. It is being used well, it is being used a lot by the customers that we have and also by users of the open source version. But it's really being able to explain to people why it's a natural extension for it and how well it fits into that. And a lot of it is also extending some of the infrastructure capabilities that Astronomer provides to those customers for them to be able to run some of the more platform specific requirements that come with doing machine learning pipelines. >> Let's get into some of the things that make Astronomer unique. Jeff, sticking with you, when you're in customer conversations, what are some of the key differentiators that you articulate to customers? >> So a lot of it is that we are not specific to one cloud provider. So we have the ability to operate across all of the big cloud providers. I know, I'm certain we have the best developers that understand how best practices implementations for data orchestration works. So we spend a lot of time talking to not just the business outcomes and the business users of the product, but also also for the technical people, how to help them better implement things that they may have come across on a Stack Overflow article or not necessarily just grown with how the product has migrated. So it's the ability to run it wherever you need to run it and also our ability to help you, the customer, better implement and understand those workflows that I think are two of the primary differentiators that we have. >> Lisa: Got it. >> I'll add another one if you don't mind. >> You can go ahead, Steven. >> Is lineage and dependencies between workflows. One thing we've done is to augment core Airflow with Lineage services. So using the Open Lineage framework, another open source framework for tracking datasets as they move from one workflow to another one, team to another, one data source to another is a really key component of what we do and we bundle that within the service so that as a developer or as a production engineer, you really don't have to worry about lineage, it just happens. Jeff, may show us some of this later that you can actually see as data flows from source through to a data warehouse out through a Python notebook to produce a predictive model or a dashboard. Can you see how those data products relate to each other? And when something goes wrong, figure out what upstream maybe caused the problem, or if you're about to change something, figure out what the impact is going to be on the rest of the organization. So Lineage is a big deal for us. >> Got it. >> And just to add on to that, the other thing to think about is that traditional Airflow is actually a complicated implementation. It required quite a lot of time spent understanding or was almost a bespoke language that you needed to be able to develop in two write these DAGs, which is like fundamental pipelines. So part of what we are focusing on is tooling that makes it more accessible to say a data analyst or a data scientist who doesn't have or really needs to gain the necessary background in how the semantics of Airflow DAGs works to still be able to get the benefit of what Airflow can do. So there is new features and capabilities built into the astronomer cloud platform that effectively obfuscates and removes the need to understand some of the deep work that goes on. But you can still do it, you still have that capability, but we are expanding it to be able to have orchestrated and repeatable processes accessible to more teams within the business. >> In terms of accessibility to more teams in the business. You talked about data scientists, data analysts, developers. Steven, I want to talk to you, as the chief data officer, are you having more and more conversations with that role and how is it emerging and evolving within your customer base? >> Hmm. That's a good question, and it is evolving because I think if you look historically at the way that Airflow has been used it's often from the ground up. You have individual data engineers or maybe single data engineering teams who adopt Airflow 'cause it's very popular. Lots of people know how to use it and they bring it into an organization and say, "Hey, let's use this to run our data pipelines." But then increasingly as you turn from pure workflow management and job scheduling to the larger topic of orchestration you realize it gets pretty complicated, you want to have coordination across teams, and you want to have standardization for the way that you manage your data pipelines. And so having a managed service for Airflow that exists in the cloud is easy to spin up as you expand usage across the organization. And thinking long term about that in the context of orchestration that's where I think the chief data officer or the head of analytics tends to get involved because they really want to think of this as a strategic investment that they're making. Not just per team individual Airflow deployments, but a network of data orchestrators. >> That network is key. Every company these days has to be a data company. We talk about companies being data driven. It's a common word, but it's true. It's whether it is a grocer or a bank or a hospital, they've got to be data companies. So talk to me a little bit about Astronomer's business model. How is this available? How do customers get their hands on it? >> Jeff, go ahead. >> Yeah, yeah. So we have a managed cloud service and we have two modes of operation. One, you can bring your own cloud infrastructure. So you can say here is an account in say, AWS or Azure and we can go and deploy the necessary infrastructure into that, or alternatively we can host everything for you. So it becomes a full SaaS offering. But we then provide a platform that connects at the backend to your internal IDP process. So however you are authenticating users to make sure that the correct people are accessing the services that they need with role-based access control. From there we are deploying through Kubernetes, the different services and capabilities into either your cloud account or into an account that we host. And from there Airflow does what Airflow does, which is its ability to then reach to different data systems and data platforms and to then run the orchestration. We make sure we do it securely, we have all the necessary compliance certifications required for GDPR in Europe and HIPAA based out of the US, and a whole bunch host of others. So it is a secure platform that can run in a place that you need it to run, but it is a managed Airflow that includes a lot of the extra capabilities like the cloud developer environment and the open lineage services to enhance the overall airflow experience. >> Enhance the overall experience. So Steven, going back to you, if I'm a Conde Nast or another organization, what are some of the key business outcomes that I can expect? As one of the things I think we've learned during the pandemic is access to realtime data is no longer a nice to have for organizations. It's really an imperative. It's that demanding consumer that wants to have that personalized, customized, instant access to a product or a service. So if I'm a Conde Nast or I'm one of your customers, what can I expect my business to be able to achieve as a result of data orchestration? >> Yeah, I think in a nutshell it's about providing a reliable, scalable, and easy to use service for developing and running data workflows. And talking of demanding customers, I mean, I'm actually a customer myself, as you mentioned, I'm the head of data for Astronomer. You won't be surprised to hear that we actually use Astronomer and Airflow to run all of our data pipelines. And so I can actually talk about my experience. When I started I was of course familiar with Airflow, but it always seemed a little bit unapproachable to me if I was introducing that to a new team of data scientists. They don't necessarily want to have to think about learning something new. But I think because of the layers that Astronomer has provided with our Astro service around Airflow it was pretty easy for me to get up and running. Of course I've got an incentive for doing that. I work for the Airflow company, but we went from about, at the beginning of last year, about 500 data tasks that we were running on a daily basis to about 15,000 every day. We run something like a million data operations every month within my team. And so as one outcome, just the ability to spin up new production workflows essentially in a single day you go from an idea in the morning to a new dashboard or a new model in the afternoon, that's really the business outcome is just removing that friction to operationalizing your machine learning and data workflows. >> And I imagine too, oh, go ahead, Jeff. >> Yeah, I think to add to that, one of the things that becomes part of the business cycle is a repeatable capabilities for things like reporting, for things like new machine learning models. And the impediment that has existed is that it's difficult to take that from a team that's an analyst team who then provide that or a data science team that then provide that to the data engineering team who have to work the workflow all the way through. What we're trying to unlock is the ability for those teams to directly get access to scheduling and orchestrating capabilities so that a business analyst can have a new report for C-suite execs that needs to be done once a week, but the time to repeatability for that report is much shorter. So it is then immediately in the hands of the person that needs to see it. It doesn't have to go into a long list of to-dos for a data engineering team that's already overworked that they eventually get it to it in a month's time. So that is also a part of it is that the realizing, orchestration I think is fairly well and a lot of people get the benefit of being able to orchestrate things within a business, but it's having more people be able to do it and shorten the time that that repeatability is there is one of the main benefits from good managed orchestration. >> So a lot of workforce productivity improvements in what you're doing to simplify things, giving more people access to data to be able to make those faster decisions, which ultimately helps the end user on the other end to get that product or the service that they're expecting like that. Jeff, I understand you have a demo that you can share so we can kind of dig into this. >> Yeah, let me take you through a quick look of how the whole thing works. So our starting point is our cloud infrastructure. This is the login. You go to the portal. You can see there's a a bunch of workspaces that are available. Workspaces are like individual places for people to operate in. I'm not going to delve into all the deep technical details here, but starting point for a lot of our data science customers is we have what we call our Cloud IDE, which is a web-based development environment for writing and building out DAGs without actually having to know how the underpinnings of Airflow work. This is an internal one, something that we use. You have a notebook-like interface that lets you write python code and SQL code and a bunch of specific bespoke type of blocks if you want. They all get pulled together and create a workflow. So this is a workflow, which gets compiled to something that looks like a complicated set of Python code, which is the DAG. I then have a CICD process pipeline where I commit this through to my GitHub repo. So this comes to a repo here, which is where these DAGs that I created in the previous step exist. I can then go and say, all right, I want to see how those particular DAGs have been running. We then get to the actual Airflow part. So this is the managed Airflow component. So we add the ability for teams to fairly easily bring up an Airflow instance and write code inside our notebook-like environment to get it into that instance. So you can see it's been running. That same process that we built here that graph ends up here inside this, but you don't need to know how the fundamentals of Airflow work in order to get this going. Then we can run one of these, it runs in the background and we can manage how it goes. And from there, every time this runs, it's emitting to a process underneath, which is the open lineage service, which is the lineage integration that allows me to come in here and have a look and see this was that actual, that same graph that we built, but now it's the historic version. So I know where things started, where things are going, and how it ran. And then I can also do a comparison. So if I want to see how this particular run worked compared to one historically, I can grab one from a previous date and it will show me the comparison between the two. So that combination of managed Airflow, getting Airflow up and running very quickly, but the Cloud IDE that lets you write code and know how to get something into a repeatable format get that into Airflow and have that attached to the lineage process adds what is a complete end-to-end orchestration process for any business looking to get the benefit from orchestration. >> Outstanding. Thank you so much Jeff for digging into that. So one of my last questions, Steven is for you. This is exciting. There's a lot that you guys are enabling organizations to achieve here to really become data-driven companies. So where can folks go to get their hands on this? >> Yeah, just go to astronomer.io and we have plenty of resources. If you're new to Airflow, you can read our documentation, our guides to getting started. We have a CLI that you can download that is really I think the easiest way to get started with Airflow. But you can actually sign up for a trial. You can sign up for a guided trial where our teams, we have a team of experts, really the world experts on getting Airflow up and running. And they'll take you through that trial and allow you to actually kick the tires and see how this works with your data. And I think you'll see pretty quickly that it's very easy to get started with Airflow, whether you're doing that from the command line or doing that in our cloud service. And all of that is available on our website >> astronomer.io. Jeff, last question for you. What are you excited about? There's so much going on here. What are some of the things, maybe you can give us a sneak peek coming down the road here that prospects and existing customers should be excited about? >> I think a lot of the development around the data awareness components, so one of the things that's traditionally been complicated with orchestration is you leave your data in the place that you're operating on and we're starting to have more data processing capability being built into Airflow. And from a Astronomer perspective, we are adding more capabilities around working with larger datasets, doing bigger data manipulation with inside the Airflow process itself. And that lends itself to better machine learning implementation. So as we start to grow and as we start to get better in the machine learning context, well, in the data awareness context, it unlocks a lot more capability to do and implement proper machine learning pipelines. >> Awesome guys. Exciting stuff. Thank you so much for talking to me about Astronomer, machine learning, data orchestration, and really the value in it for your customers. Steve and Jeff, we appreciate your time. >> Thank you. >> My pleasure, thanks. >> And we thank you for watching. This is season three, episode one of our ongoing series covering exciting startups from the AWS ecosystem. I'm your host, Lisa Martin. You're watching theCUBE, the leader in live tech coverage. (upbeat music)

(upbeat music) >> Hello everyone. Welcome to theCube's presentation of the "AWS Startup Showcase." The topic this episode is AI and machine learning, top startups building foundational model infrastructure. This is season three, episode one of the ongoing series covering exciting startups from the AWS ecosystem. And this time we're talking about AI and machine learning. I'm your host, John Furrier. I'm excited I'm joined today by Robert Nishihara, who's the co-founder and CEO of a hot startup called Anyscale. He's here to talk about Ray, the open source project, Anyscale's infrastructure for foundation as well. Robert, thank you for joining us today. >> Yeah, thanks so much as well. >> I've been following your company since the founding pre pandemic and you guys really had a great vision scaled up and in a perfect position for this big wave that we all see with ChatGPT and OpenAI that's gone mainstream. Finally, AI has broken out through the ropes and now gone mainstream, so I think you guys are really well positioned. I'm looking forward to to talking with you today. But before we get into it, introduce the core mission for Anyscale. Why do you guys exist? What is the North Star for Anyscale? >> Yeah, like you mentioned, there's a tremendous amount of excitement about AI right now. You know, I think a lot of us believe that AI can transform just every different industry. So one of the things that was clear to us when we started this company was that the amount of compute needed to do AI was just exploding. Like to actually succeed with AI, companies like OpenAI or Google or you know, these companies getting a lot of value from AI, were not just running these machine learning models on their laptops or on a single machine. They were scaling these applications across hundreds or thousands or more machines and GPUs and other resources in the Cloud. And so to actually succeed with AI, and this has been one of the biggest trends in computing, maybe the biggest trend in computing in, you know, in recent history, the amount of compute has been exploding. And so to actually succeed with that AI, to actually build these scalable applications and scale the AI applications, there's a tremendous software engineering lift to build the infrastructure to actually run these scalable applications. And that's very hard to do. So one of the reasons many AI projects and initiatives fail is that, or don't make it to production, is the need for this scale, the infrastructure lift, to actually make it happen. So our goal here with Anyscale and Ray, is to make that easy, is to make scalable computing easy. So that as a developer or as a business, if you want to do AI, if you want to get value out of AI, all you need to know is how to program on your laptop. Like, all you need to know is how to program in Python. And if you can do that, then you're good to go. Then you can do what companies like OpenAI or Google do and get value out of machine learning. >> That programming example of how easy it is with Python reminds me of the early days of Cloud, when infrastructure as code was talked about was, it was just code the infrastructure programmable. That's super important. That's what AI people wanted, first program AI. That's the new trend. And I want to understand, if you don't mind explaining, the relationship that Anyscale has to these foundational models and particular the large language models, also called LLMs, was seen with like OpenAI and ChatGPT. Before you get into the relationship that you have with them, can you explain why the hype around foundational models? Why are people going crazy over foundational models? What is it and why is it so important? >> Yeah, so foundational models and foundation models are incredibly important because they enable businesses and developers to get value out of machine learning, to use machine learning off the shelf with these large models that have been trained on tons of data and that are useful out of the box. And then, of course, you know, as a business or as a developer, you can take those foundational models and repurpose them or fine tune them or adapt them to your specific use case and what you want to achieve. But it's much easier to do that than to train them from scratch. And I think there are three, for people to actually use foundation models, there are three main types of workloads or problems that need to be solved. One is training these foundation models in the first place, like actually creating them. The second is fine tuning them and adapting them to your use case. And the third is serving them and actually deploying them. Okay, so Ray and Anyscale are used for all of these three different workloads. Companies like OpenAI or Cohere that train large language models. Or open source versions like GPTJ are done on top of Ray. There are many startups and other businesses that fine tune, that, you know, don't want to train the large underlying foundation models, but that do want to fine tune them, do want to adapt them to their purposes, and build products around them and serve them, those are also using Ray and Anyscale for that fine tuning and that serving. And so the reason that Ray and Anyscale are important here is that, you know, building and using foundation models requires a huge scale. It requires a lot of data. It requires a lot of compute, GPUs, TPUs, other resources. And to actually take advantage of that and actually build these scalable applications, there's a lot of infrastructure that needs to happen under the hood. And so you can either use Ray and Anyscale to take care of that and manage the infrastructure and solve those infrastructure problems. Or you can build the infrastructure and manage the infrastructure yourself, which you can do, but it's going to slow your team down. It's going to, you know, many of the businesses we work with simply don't want to be in the business of managing infrastructure and building infrastructure. They want to focus on product development and move faster. >> I know you got a keynote presentation we're going to go to in a second, but I think you hit on something I think is the real tipping point, doing it yourself, hard to do. These are things where opportunities are and the Cloud did that with data centers. Turned a data center and made it an API. The heavy lifting went away and went to the Cloud so people could be more creative and build their product. In this case, build their creativity. Is that kind of what's the big deal? Is that kind of a big deal happening that you guys are taking the learnings and making that available so people don't have to do that? >> That's exactly right. So today, if you want to succeed with AI, if you want to use AI in your business, infrastructure work is on the critical path for doing that. To do AI, you have to build infrastructure. You have to figure out how to scale your applications. That's going to change. We're going to get to the point, and you know, with Ray and Anyscale, we're going to remove the infrastructure from the critical path so that as a developer or as a business, all you need to focus on is your application logic, what you want the the program to do, what you want your application to do, how you want the AI to actually interface with the rest of your product. Now the way that will happen is that Ray and Anyscale will still, the infrastructure work will still happen. It'll just be under the hood and taken care of by Ray in Anyscale. And so I think something like this is really necessary for AI to reach its potential, for AI to have the impact and the reach that we think it will, you have to make it easier to do. >> And just for clarification to point out, if you don't mind explaining the relationship of Ray and Anyscale real quick just before we get into the presentation. >> So Ray is an open source project. We created it. We were at Berkeley doing machine learning. We started Ray so that, in order to provide an easy, a simple open source tool for building and running scalable applications. And Anyscale is the managed version of Ray, basically we will run Ray for you in the Cloud, provide a lot of tools around the developer experience and managing the infrastructure and providing more performance and superior infrastructure. >> Awesome. I know you got a presentation on Ray and Anyscale and you guys are positioning as the infrastructure for foundational models. So I'll let you take it away and then when you're done presenting, we'll come back, I'll probably grill you with a few questions and then we'll close it out so take it away. >> Robert: Sounds great. So I'll say a little bit about how companies are using Ray and Anyscale for foundation models. The first thing I want to mention is just why we're doing this in the first place. And the underlying observation, the underlying trend here, and this is a plot from OpenAI, is that the amount of compute needed to do machine learning has been exploding. It's been growing at something like 35 times every 18 months. This is absolutely enormous. And other people have written papers measuring this trend and you get different numbers. But the point is, no matter how you slice and dice it, it' a astronomical rate. Now if you compare that to something we're all familiar with, like Moore's Law, which says that, you know, the processor performance doubles every roughly 18 months, you can see that there's just a tremendous gap between the needs, the compute needs of machine learning applications, and what you can do with a single chip, right. So even if Moore's Law were continuing strong and you know, doing what it used to be doing, even if that were the case, there would still be a tremendous gap between what you can do with the chip and what you need in order to do machine learning. And so given this graph, what we've seen, and what has been clear to us since we started this company, is that doing AI requires scaling. There's no way around it. It's not a nice to have, it's really a requirement. And so that led us to start Ray, which is the open source project that we started to make it easy to build these scalable Python applications and scalable machine learning applications. And since we started the project, it's been adopted by a tremendous number of companies. Companies like OpenAI, which use Ray to train their large models like ChatGPT, companies like Uber, which run all of their deep learning and classical machine learning on top of Ray, companies like Shopify or Spotify or Instacart or Lyft or Netflix, ByteDance, which use Ray for their machine learning infrastructure. Companies like Ant Group, which makes Alipay, you know, they use Ray across the board for fraud detection, for online learning, for detecting money laundering, you know, for graph processing, stream processing. Companies like Amazon, you know, run Ray at a tremendous scale and just petabytes of data every single day. And so the project has seen just enormous adoption since, over the past few years. And one of the most exciting use cases is really providing the infrastructure for building training, fine tuning, and serving foundation models. So I'll say a little bit about, you know, here are some examples of companies using Ray for foundation models. Cohere trains large language models. OpenAI also trains large language models. You can think about the workloads required there are things like supervised pre-training, also reinforcement learning from human feedback. So this is not only the regular supervised learning, but actually more complex reinforcement learning workloads that take human input about what response to a particular question, you know is better than a certain other response. And incorporating that into the learning. There's open source versions as well, like GPTJ also built on top of Ray as well as projects like Alpa coming out of UC Berkeley. So these are some of the examples of exciting projects in organizations, training and creating these large language models and serving them using Ray. Okay, so what actually is Ray? Well, there are two layers to Ray. At the lowest level, there's the core Ray system. This is essentially low level primitives for building scalable Python applications. Things like taking a Python function or a Python class and executing them in the cluster setting. So Ray core is extremely flexible and you can build arbitrary scalable applications on top of Ray. So on top of Ray, on top of the core system, what really gives Ray a lot of its power is this ecosystem of scalable libraries. So on top of the core system you have libraries, scalable libraries for ingesting and pre-processing data, for training your models, for fine tuning those models, for hyper parameter tuning, for doing batch processing and batch inference, for doing model serving and deployment, right. And a lot of the Ray users, the reason they like Ray is that they want to run multiple workloads. They want to train and serve their models, right. They want to load their data and feed that into training. And Ray provides common infrastructure for all of these different workloads. So this is a little overview of what Ray, the different components of Ray. So why do people choose to go with Ray? I think there are three main reasons. The first is the unified nature. The fact that it is common infrastructure for scaling arbitrary workloads, from data ingest to pre-processing to training to inference and serving, right. This also includes the fact that it's future proof. AI is incredibly fast moving. And so many people, many companies that have built their own machine learning infrastructure and standardized on particular workflows for doing machine learning have found that their workflows are too rigid to enable new capabilities. If they want to do reinforcement learning, if they want to use graph neural networks, they don't have a way of doing that with their standard tooling. And so Ray, being future proof and being flexible and general gives them that ability. Another reason people choose Ray in Anyscale is the scalability. This is really our bread and butter. This is the reason, the whole point of Ray, you know, making it easy to go from your laptop to running on thousands of GPUs, making it easy to scale your development workloads and run them in production, making it easy to scale, you know, training to scale data ingest, pre-processing and so on. So scalability and performance, you know, are critical for doing machine learning and that is something that Ray provides out of the box. And lastly, Ray is an open ecosystem. You can run it anywhere. You can run it on any Cloud provider. Google, you know, Google Cloud, AWS, Asure. You can run it on your Kubernetes cluster. You can run it on your laptop. It's extremely portable. And not only that, it's framework agnostic. You can use Ray to scale arbitrary Python workloads. You can use it to scale and it integrates with libraries like TensorFlow or PyTorch or JAX or XG Boost or Hugging Face or PyTorch Lightning, right, or Scikit-learn or just your own arbitrary Python code. It's open source. And in addition to integrating with the rest of the machine learning ecosystem and these machine learning frameworks, you can use Ray along with all of the other tooling in the machine learning ecosystem. That's things like weights and biases or ML flow, right. Or you know, different data platforms like Databricks, you know, Delta Lake or Snowflake or tools for model monitoring for feature stores, all of these integrate with Ray. And that's, you know, Ray provides that kind of flexibility so that you can integrate it into the rest of your workflow. And then Anyscale is the scalable compute platform that's built on top, you know, that provides Ray. So Anyscale is a managed Ray service that runs in the Cloud. And what Anyscale does is it offers the best way to run Ray. And if you think about what you get with Anyscale, there are fundamentally two things. One is about moving faster, accelerating the time to market. And you get that by having the managed service so that as a developer you don't have to worry about managing infrastructure, you don't have to worry about configuring infrastructure. You also, it provides, you know, optimized developer workflows. Things like easily moving from development to production, things like having the observability tooling, the debug ability to actually easily diagnose what's going wrong in a distributed application. So things like the dashboards and the other other kinds of tooling for collaboration, for monitoring and so on. And then on top of that, so that's the first bucket, developer productivity, moving faster, faster experimentation and iteration. The second reason that people choose Anyscale is superior infrastructure. So this is things like, you know, cost deficiency, being able to easily take advantage of spot instances, being able to get higher GPU utilization, things like faster cluster startup times and auto scaling. Things like just overall better performance and faster scheduling. And so these are the kinds of things that Anyscale provides on top of Ray. It's the managed infrastructure. It's fast, it's like the developer productivity and velocity as well as performance. So this is what I wanted to share about Ray in Anyscale. >> John: Awesome. >> Provide that context. But John, I'm curious what you think. >> I love it. I love the, so first of all, it's a platform because that's the platform architecture right there. So just to clarify, this is an Anyscale platform, not- >> That's right. >> Tools. So you got tools in the platform. Okay, that's key. Love that managed service. Just curious, you mentioned Python multiple times, is that because of PyTorch and TensorFlow or Python's the most friendly with machine learning or it's because it's very common amongst all developers? >> That's a great question. Python is the language that people are using to do machine learning. So it's the natural starting point. Now, of course, Ray is actually designed in a language agnostic way and there are companies out there that use Ray to build scalable Java applications. But for the most part right now we're focused on Python and being the best way to build these scalable Python and machine learning applications. But, of course, down the road there always is that potential. >> So if you're slinging Python code out there and you're watching that, you're watching this video, get on Anyscale bus quickly. Also, I just, while you were giving the presentation, I couldn't help, since you mentioned OpenAI, which by the way, congratulations 'cause they've had great scale, I've noticed in their rapid growth 'cause they were the fastest company to the number of users than anyone in the history of the computer industry, so major successor, OpenAI and ChatGPT, huge fan. I'm not a skeptic at all. I think it's just the beginning, so congratulations. But I actually typed into ChatGPT, what are the top three benefits of Anyscale and came up with scalability, flexibility, and ease of use. Obviously, scalability is what you guys are called. >> That's pretty good. >> So that's what they came up with. So they nailed it. Did you have an inside prompt training, buy it there? Only kidding. (Robert laughs) >> Yeah, we hard coded that one. >> But that's the kind of thing that came up really, really quickly if I asked it to write a sales document, it probably will, but this is the future interface. This is why people are getting excited about the foundational models and the large language models because it's allowing the interface with the user, the consumer, to be more human, more natural. And this is clearly will be in every application in the future. >> Absolutely. This is how people are going to interface with software, how they're going to interface with products in the future. It's not just something, you know, not just a chat bot that you talk to. This is going to be how you get things done, right. How you use your web browser or how you use, you know, how you use Photoshop or how you use other products. Like you're not going to spend hours learning all the APIs and how to use them. You're going to talk to it and tell it what you want it to do. And of course, you know, if it doesn't understand it, it's going to ask clarifying questions. You're going to have a conversation and then it'll figure it out. >> This is going to be one of those things, we're going to look back at this time Robert and saying, "Yeah, from that company, that was the beginning of that wave." And just like AWS and Cloud Computing, the folks who got in early really were in position when say the pandemic came. So getting in early is a good thing and that's what everyone's talking about is getting in early and playing around, maybe replatforming or even picking one or few apps to refactor with some staff and managed services. So people are definitely jumping in. So I have to ask you the ROI cost question. You mentioned some of those, Moore's Law versus what's going on in the industry. When you look at that kind of scale, the first thing that jumps out at people is, "Okay, I love it. Let's go play around." But what's it going to cost me? Am I going to be tied to certain GPUs? What's the landscape look like from an operational standpoint, from the customer? Are they locked in and the benefit was flexibility, are you flexible to handle any Cloud? What is the customers, what are they looking at? Basically, that's my question. What's the customer looking at? >> Cost is super important here and many of the companies, I mean, companies are spending a huge amount on their Cloud computing, on AWS, and on doing AI, right. And I think a lot of the advantage of Anyscale, what we can provide here is not only better performance, but cost efficiency. Because if we can run something faster and more efficiently, it can also use less resources and you can lower your Cloud spending, right. We've seen companies go from, you know, 20% GPU utilization with their current setup and the current tools they're using to running on Anyscale and getting more like 95, you know, 100% GPU utilization. That's something like a five x improvement right there. So depending on the kind of application you're running, you know, it's a significant cost savings. We've seen companies that have, you know, processing petabytes of data every single day with Ray going from, you know, getting order of magnitude cost savings by switching from what they were previously doing to running their application on Ray. And when you have applications that are spending, you know, potentially $100 million a year and getting a 10 X cost savings is just absolutely enormous. So these are some of the kinds of- >> Data infrastructure is super important. Again, if the customer, if you're a prospect to this and thinking about going in here, just like the Cloud, you got infrastructure, you got the platform, you got SaaS, same kind of thing's going to go on in AI. So I want to get into that, you know, ROI discussion and some of the impact with your customers that are leveraging the platform. But first I hear you got a demo. >> Robert: Yeah, so let me show you, let me give you a quick run through here. So what I have open here is the Anyscale UI. I've started a little Anyscale Workspace. So Workspaces are the Anyscale concept for interactive developments, right. So here, imagine I'm just, you want to have a familiar experience like you're developing on your laptop. And here I have a terminal. It's not on my laptop. It's actually in the cloud running on Anyscale. And I'm just going to kick this off. This is going to train a large language model, so OPT. And it's doing this on 32 GPUs. We've got a cluster here with a bunch of CPU cores, bunch of memory. And as that's running, and by the way, if I wanted to run this on instead of 32 GPUs, 64, 128, this is just a one line change when I launch the Workspace. And what I can do is I can pull up VS code, right. Remember this is the interactive development experience. I can look at the actual code. Here it's using Ray train to train the torch model. We've got the training loop and we're saying that each worker gets access to one GPU and four CPU cores. And, of course, as I make the model larger, this is using deep speed, as I make the model larger, I could increase the number of GPUs that each worker gets access to, right. And how that is distributed across the cluster. And if I wanted to run on CPUs instead of GPUs or a different, you know, accelerator type, again, this is just a one line change. And here we're using Ray train to train the models, just taking my vanilla PyTorch model using Hugging Face and then scaling that across a bunch of GPUs. And, of course, if I want to look at the dashboard, I can go to the Ray dashboard. There are a bunch of different visualizations I can look at. I can look at the GPU utilization. I can look at, you know, the CPU utilization here where I think we're currently loading the model and running that actual application to start the training. And some of the things that are really convenient here about Anyscale, both I can get that interactive development experience with VS code. You know, I can look at the dashboards. I can monitor what's going on. It feels, I have a terminal, it feels like my laptop, but it's actually running on a large cluster. And I can, with however many GPUs or other resources that I want. And so it's really trying to combine the best of having the familiar experience of programming on your laptop, but with the benefits, you know, being able to take advantage of all the resources in the Cloud to scale. And it's like when, you know, you're talking about cost efficiency. One of the biggest reasons that people waste money, one of the silly reasons for wasting money is just forgetting to turn off your GPUs. And what you can do here is, of course, things will auto terminate if they're idle. But imagine you go to sleep, I have this big cluster. You can turn it off, shut off the cluster, come back tomorrow, restart the Workspace, and you know, your big cluster is back up and all of your code changes are still there. All of your local file edits. It's like you just closed your laptop and came back and opened it up again. And so this is the kind of experience we want to provide for our users. So that's what I wanted to share with you. >> Well, I think that whole, couple of things, lines of code change, single line of code change, that's game changing. And then the cost thing, I mean human error is a big deal. People pass out at their computer. They've been coding all night or they just forget about it. I mean, and then it's just like leaving the lights on or your water running in your house. It's just, at the scale that it is, the numbers will add up. That's a huge deal. So I think, you know, compute back in the old days, there's no compute. Okay, it's just compute sitting there idle. But you know, data cranking the models is doing, that's a big point. >> Another thing I want to add there about cost efficiency is that we make it really easy to use, if you're running on Anyscale, to use spot instances and these preemptable instances that can just be significantly cheaper than the on-demand instances. And so when we see our customers go from what they're doing before to using Anyscale and they go from not using these spot instances 'cause they don't have the infrastructure around it, the fault tolerance to handle the preemption and things like that, to being able to just check a box and use spot instances and save a bunch of money. >> You know, this was my whole, my feature article at Reinvent last year when I met with Adam Selipsky, this next gen Cloud is here. I mean, it's not auto scale, it's infrastructure scale. It's agility. It's flexibility. I think this is where the world needs to go. Almost what DevOps did for Cloud and what you were showing me that demo had this whole SRE vibe. And remember Google had site reliability engines to manage all those servers. This is kind of like an SRE vibe for data at scale. I mean, a similar kind of order of magnitude. I mean, I might be a little bit off base there, but how would you explain it? >> It's a nice analogy. I mean, what we are trying to do here is get to the point where developers don't think about infrastructure. Where developers only think about their application logic. And where businesses can do AI, can succeed with AI, and build these scalable applications, but they don't have to build, you know, an infrastructure team. They don't have to develop that expertise. They don't have to invest years in building their internal machine learning infrastructure. They can just focus on the Python code, on their application logic, and run the stuff out of the box. >> Awesome. Well, I appreciate the time. Before we wrap up here, give a plug for the company. I know you got a couple websites. Again, go, Ray's got its own website. You got Anyscale. You got an event coming up. Give a plug for the company looking to hire. Put a plug in for the company. >> Yeah, absolutely. Thank you. So first of all, you know, we think AI is really going to transform every industry and the opportunity is there, right. We can be the infrastructure that enables all of that to happen, that makes it easy for companies to succeed with AI, and get value out of AI. Now we have, if you're interested in learning more about Ray, Ray has been emerging as the standard way to build scalable applications. Our adoption has been exploding. I mentioned companies like OpenAI using Ray to train their models. But really across the board companies like Netflix and Cruise and Instacart and Lyft and Uber, you know, just among tech companies. It's across every industry. You know, gaming companies, agriculture, you know, farming, robotics, drug discovery, you know, FinTech, we see it across the board. And all of these companies can get value out of AI, can really use AI to improve their businesses. So if you're interested in learning more about Ray and Anyscale, we have our Ray Summit coming up in September. This is going to highlight a lot of the most impressive use cases and stories across the industry. And if your business, if you want to use LLMs, you want to train these LLMs, these large language models, you want to fine tune them with your data, you want to deploy them, serve them, and build applications and products around them, give us a call, talk to us. You know, we can really take the infrastructure piece, you know, off the critical path and make that easy for you. So that's what I would say. And, you know, like you mentioned, we're hiring across the board, you know, engineering, product, go-to-market, and it's an exciting time. >> Robert Nishihara, co-founder and CEO of Anyscale, congratulations on a great company you've built and continuing to iterate on and you got growth ahead of you, you got a tailwind. I mean, the AI wave is here. I think OpenAI and ChatGPT, a customer of yours, have really opened up the mainstream visibility into this new generation of applications, user interface, roll of data, large scale, how to make that programmable so we're going to need that infrastructure. So thanks for coming on this season three, episode one of the ongoing series of the hot startups. In this case, this episode is the top startups building foundational model infrastructure for AI and ML. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(light airy music) >> Hello, everyone, welcome to theCUBE's presentation of the AWS Startup Showcase, AI and machine learning. "Top Startups Building Generative AI on AWS." This is season three, episode one of the ongoing series covering the exciting startups from the AWS ecosystem, talking about AI machine learning. We have three great guests Bratin Saha, VP, Vice President of Machine Learning and AI Services at Amazon Web Services. Tom Mason, the CTO of Stability AI, and Aidan Gomez, CEO and co-founder of Cohere. Two practitioners doing startups and AWS. Gentlemen, thank you for opening up this session, this episode. Thanks for coming on. >> Thank you. >> Thank you. >> Thank you. >> So the topic is hype versus reality. So I think we're all on the reality is great, hype is great, but the reality's here. I want to get into it. Generative AI's got all the momentum, it's going mainstream, it's kind of come out of the behind the ropes, it's now mainstream. We saw the success of ChatGPT, opens up everyone's eyes, but there's so much more going on. Let's jump in and get your early perspectives on what should people be talking about right now? What are you guys working on? We'll start with AWS. What's the big focus right now for you guys as you come into this market that's highly active, highly hyped up, but people see value right out of the gate? >> You know, we have been working on generative AI for some time. In fact, last year we released Code Whisperer, which is about using generative AI for software development and a number of customers are using it and getting real value out of it. So generative AI is now something that's mainstream that can be used by enterprise users. And we have also been partnering with a number of other companies. So, you know, stability.ai, we've been partnering with them a lot. We want to be partnering with other companies as well. In seeing how we do three things, you know, first is providing the most efficient infrastructure for generative AI. And that is where, you know, things like Trainium, things like Inferentia, things like SageMaker come in. And then next is the set of models and then the third is the kind of applications like Code Whisperer and so on. So, you know, it's early days yet, but clearly there's a lot of amazing capabilities that will come out and something that, you know, our customers are starting to pay a lot of attention to. >> Tom, talk about your company and what your focus is and why the Amazon Web Services relationship's important for you? >> So yeah, we're primarily committed to making incredible open source foundation models and obviously stable effusions been our kind of first big model there, which we trained all on AWS. We've been working with them over the last year and a half to develop, obviously a big cluster, and bring all that compute to training these models at scale, which has been a really successful partnership. And we're excited to take it further this year as we develop commercial strategy of the business and build out, you know, the ability for enterprise customers to come and get all the value from these models that we think they can get. So we're really excited about the future. We got hugely exciting pipeline for this year with new modalities and video models and wonderful things and trying to solve images for once and for all and get the kind of general value and value proposition correct for customers. So it's a really exciting time and very honored to be part of it. >> It's great to see some of your customers doing so well out there. Congratulations to your team. Appreciate that. Aidan, let's get into what you guys do. What does Cohere do? What are you excited about right now? >> Yeah, so Cohere builds large language models, which are the backbone of applications like ChatGPT and GPT-3. We're extremely focused on solving the issues with adoption for enterprise. So it's great that you can make a super flashy demo for consumers, but it takes a lot to actually get it into billion user products and large global enterprises. So about six months ago, we released our command models, which are some of the best that exist for large language models. And in December, we released our multilingual text understanding models and that's on over a hundred different languages and it's trained on, you know, authentic data directly from native speakers. And so we're super excited to continue pushing this into enterprise and solving those barriers for adoption, making this transformation a reality. >> Just real quick, while I got you there on the new products coming out. Where are we in the progress? People see some of the new stuff out there right now. There's so much more headroom. Can you just scope out in your mind what that looks like? Like from a headroom standpoint? Okay, we see ChatGPT. "Oh yeah, it writes my papers for me, does some homework for me." I mean okay, yawn, maybe people say that, (Aidan chuckles) people excited or people are blown away. I mean, it's helped theCUBE out, it helps me, you know, feed up a little bit from my write-ups but it's not always perfect. >> Yeah, at the moment it's like a writing assistant, right? And it's still super early in the technologies trajectory. I think it's fascinating and it's interesting but its impact is still really limited. I think in the next year, like within the next eight months, we're going to see some major changes. You've already seen the very first hints of that with stuff like Bing Chat, where you augment these dialogue models with an external knowledge base. So now the models can be kept up to date to the millisecond, right? Because they can search the web and they can see events that happened a millisecond ago. But that's still limited in the sense that when you ask the question, what can these models actually do? Well they can just write text back at you. That's the extent of what they can do. And so the real project, the real effort, that I think we're all working towards is actually taking action. So what happens when you give these models the ability to use tools, to use APIs? What can they do when they can actually affect change out in the real world, beyond just streaming text back at the user? I think that's the really exciting piece. >> Okay, so I wanted to tee that up early in the segment 'cause I want to get into the customer applications. We're seeing early adopters come in, using the technology because they have a lot of data, they have a lot of large language model opportunities and then there's a big fast follower wave coming behind it. I call that the people who are going to jump in the pool early and get into it. They might not be advanced. Can you guys share what customer applications are being used with large language and vision models today and how they're using it to transform on the early adopter side, and how is that a tell sign of what's to come? >> You know, one of the things we have been seeing both with the text models that Aidan talked about as well as the vision models that stability.ai does, Tom, is customers are really using it to change the way you interact with information. You know, one example of a customer that we have, is someone who's kind of using that to query customer conversations and ask questions like, you know, "What was the customer issue? How did we solve it?" And trying to get those kinds of insights that was previously much harder to do. And then of course software is a big area. You know, generating software, making that, you know, just deploying it in production. Those have been really big areas that we have seen customers start to do. You know, looking at documentation, like instead of you know, searching for stuff and so on, you know, you just have an interactive way, in which you can just look at the documentation for a product. You know, all of this goes to where we need to take the technology. One of which is, you know, the models have to be there but they have to work reliably in a production setting at scale, with privacy, with security, and you know, making sure all of this is happening, is going to be really key. That is what, you know, we at AWS are looking to do, which is work with partners like stability and others and in the open source and really take all of these and make them available at scale to customers, where they work reliably. >> Tom, Aidan, what's your thoughts on this? Where are customers landing on this first use cases or set of low-hanging fruit use cases or applications? >> Yeah, so I think like the first group of adopters that really found product market fit were the copywriting companies. So one great example of that is HyperWrite. Another one is Jasper. And so for Cohere, that's the tip of the iceberg, like there's a very long tail of usage from a bunch of different applications. HyperWrite is one of our customers, they help beat writer's block by drafting blog posts, emails, and marketing copy. We also have a global audio streaming platform, which is using us the power of search engine that can comb through podcast transcripts, in a bunch of different languages. Then a global apparel brand, which is using us to transform how they interact with their customers through a virtual assistant, two dozen global news outlets who are using us for news summarization. So really like, these large language models, they can be deployed all over the place into every single industry sector, language is everywhere. It's hard to think of any company on Earth that doesn't use language. So it's, very, very- >> We're doing it right now. We got the language coming in. >> Exactly. >> We'll transcribe this puppy. All right. Tom, on your side, what do you see the- >> Yeah, we're seeing some amazing applications of it and you know, I guess that's partly been, because of the growth in the open source community and some of these applications have come from there that are then triggering this secondary wave of innovation, which is coming a lot from, you know, controllability and explainability of the model. But we've got companies like, you know, Jasper, which Aidan mentioned, who are using stable diffusion for image generation in block creation, content creation. We've got Lensa, you know, which exploded, and is built on top of stable diffusion for fine tuning so people can bring themselves and their pets and you know, everything into the models. So we've now got fine tuned stable diffusion at scale, which is democratized, you know, that process, which is really fun to see your Lensa, you know, exploded. You know, I think it was the largest growing app in the App Store at one point. And lots of other examples like NightCafe and Lexica and Playground. So seeing lots of cool applications. >> So much applications, we'll probably be a customer for all you guys. We'll definitely talk after. But the challenges are there for people adopting, they want to get into what you guys see as the challenges that turn into opportunities. How do you see the customers adopting generative AI applications? For example, we have massive amounts of transcripts, timed up to all the videos. I don't even know what to do. Do I just, do I code my API there. So, everyone has this problem, every vertical has these use cases. What are the challenges for people getting into this and adopting these applications? Is it figuring out what to do first? Or is it a technical setup? Do they stand up stuff, they just go to Amazon? What do you guys see as the challenges? >> I think, you know, the first thing is coming up with where you think you're going to reimagine your customer experience by using generative AI. You know, we talked about Ada, and Tom talked about a number of these ones and you know, you pick up one or two of these, to get that robust. And then once you have them, you know, we have models and we'll have more models on AWS, these large language models that Aidan was talking about. Then you go in and start using these models and testing them out and seeing whether they fit in use case or not. In many situations, like you said, John, our customers want to say, "You know, I know you've trained these models on a lot of publicly available data, but I want to be able to customize it for my use cases. Because, you know, there's some knowledge that I have created and I want to be able to use that." And then in many cases, and I think Aidan mentioned this. You know, you need these models to be up to date. Like you can't have it staying. And in those cases, you augmented with a knowledge base, you know you have to make sure that these models are not hallucinating. And so you need to be able to do the right kind of responsible AI checks. So, you know, you start with a particular use case, and there are a lot of them. Then, you know, you can come to AWS, and then look at one of the many models we have and you know, we are going to have more models for other modalities as well. And then, you know, play around with the models. We have a playground kind of thing where you can test these models on some data and then you can probably, you will probably want to bring your own data, customize it to your own needs, do some of the testing to make sure that the model is giving the right output and then just deploy it. And you know, we have a lot of tools. >> Yeah. >> To make this easy for our customers. >> How should people think about large language models? Because do they think about it as something that they tap into with their IP or their data? Or is it a large language model that they apply into their system? Is the interface that way? What's the interaction look like? >> In many situations, you can use these models out of the box. But in typical, in most of the other situations, you will want to customize it with your own data or with your own expectations. So the typical use case would be, you know, these are models are exposed through APIs. So the typical use case would be, you know you're using these APIs a little bit for testing and getting familiar and then there will be an API that will allow you to train this model further on your data. So you use that AI, you know, make sure you augmented the knowledge base. So then you use those APIs to customize the model and then just deploy it in an application. You know, like Tom was mentioning, a number of companies that are using these models. So once you have it, then you know, you again, use an endpoint API and use it in an application. >> All right, I love the example. I want to ask Tom and Aidan, because like most my experience with Amazon Web Service in 2007, I would stand up in EC2, put my code on there, play around, if it didn't work out, I'd shut it down. Is that a similar dynamic we're going to see with the machine learning where developers just kind of log in and stand up infrastructure and play around and then have a cloud-like experience? >> So I can go first. So I mean, we obviously, with AWS working really closely with the SageMaker team, do fantastic platform there for ML training and inference. And you know, going back to your point earlier, you know, where the data is, is hugely important for companies. Many companies bringing their models to their data in AWS on-premise for them is hugely important. Having the models to be, you know, open sources, makes them explainable and transparent to the adopters of those models. So, you know, we are really excited to work with the SageMaker team over the coming year to bring companies to that platform and make the most of our models. >> Aidan, what's your take on developers? Do they just need to have a team in place, if we want to interface with you guys? Let's say, can they start learning? What do they got to do to set up? >> Yeah, so I think for Cohere, our product makes it much, much easier to people, for people to get started and start building, it solves a lot of the productionization problems. But of course with SageMaker, like Tom was saying, I think that lowers a barrier even further because it solves problems like data privacy. So I want to underline what Bratin was saying earlier around when you're fine tuning or when you're using these models, you don't want your data being incorporated into someone else's model. You don't want it being used for training elsewhere. And so the ability to solve for enterprises, that data privacy and that security guarantee has been hugely important for Cohere, and that's very easy to do through SageMaker. >> Yeah. >> But the barriers for using this technology are coming down super quickly. And so for developers, it's just becoming completely intuitive. I love this, there's this quote from Andrej Karpathy. He was saying like, "It really wasn't on my 2022 list of things to happen that English would become, you know, the most popular programming language." And so the barrier is coming down- >> Yeah. >> Super quickly and it's exciting to see. >> It's going to be awesome for all the companies here, and then we'll do more, we're probably going to see explosion of startups, already seeing that, the maps, ecosystem maps, the landscape maps are happening. So this is happening and I'm convinced it's not yesterday's chat bot, it's not yesterday's AI Ops. It's a whole another ballgame. So I have to ask you guys for the final question before we kick off the company's showcasing here. How do you guys gauge success of generative AI applications? Is there a lens to look through and say, okay, how do I see success? It could be just getting a win or is it a bigger picture? Bratin we'll start with you. How do you gauge success for generative AI? >> You know, ultimately it's about bringing business value to our customers. And making sure that those customers are able to reimagine their experiences by using generative AI. Now the way to get their ease, of course to deploy those models in a safe, effective manner, and ensuring that all of the robustness and the security guarantees and the privacy guarantees are all there. And we want to make sure that this transitions from something that's great demos to actual at scale products, which means making them work reliably all of the time not just some of the time. >> Tom, what's your gauge for success? >> Look, I think this, we're seeing a completely new form of ways to interact with data, to make data intelligent, and directly to bring in new revenue streams into business. So if businesses can use our models to leverage that and generate completely new revenue streams and ultimately bring incredible new value to their customers, then that's fantastic. And we hope we can power that revolution. >> Aidan, what's your take? >> Yeah, reiterating Bratin and Tom's point, I think that value in the enterprise and value in market is like a huge, you know, it's the goal that we're striving towards. I also think that, you know, the value to consumers and actual users and the transformation of the surface area of technology to create experiences like ChatGPT that are magical and it's the first time in human history we've been able to talk to something compelling that's not a human. I think that in itself is just extraordinary and so exciting to see. >> It really brings up a whole another category of markets. B2B, B2C, it's B2D, business to developer. Because I think this is kind of the big trend the consumers have to win. The developers coding the apps, it's a whole another sea change. Reminds me everyone use the "Moneyball" movie as example during the big data wave. Then you know, the value of data. There's a scene in "Moneyball" at the end, where Billy Beane's getting the offer from the Red Sox, then the owner says to the Red Sox, "If every team's not rebuilding their teams based upon your model, there'll be dinosaurs." I think that's the same with AI here. Every company will have to need to think about their business model and how they operate with AI. So it'll be a great run. >> Completely Agree >> It'll be a great run. >> Yeah. >> Aidan, Tom, thank you so much for sharing about your experiences at your companies and congratulations on your success and it's just the beginning. And Bratin, thanks for coming on representing AWS. And thank you, appreciate for what you do. Thank you. >> Thank you, John. Thank you, Aidan. >> Thank you John. >> Thanks so much. >> Okay, let's kick off season three, episode one. I'm John Furrier, your host. Thanks for watching. (light airy music)

(upbeat electronic music) >> Hello everyone, welcome to this CUBE conversation here from the studios in the CUBE in Palo Alto, California. I'm John Furrier, your host. We're featuring a startup, Astronomer. Astronomer.io is the URL, check it out. And we're going to have a great conversation around one of the most important topics hitting the industry, and that is the future of machine learning and AI, and the data that powers it underneath it. There's a lot of things that need to get done, and we're excited to have some of the co-founders of Astronomer here. Viraj Parekh, who is co-founder of Astronomer, and Paola Peraza Calderon, another co-founder, both with Astronomer. Thanks for coming on. First of all, how many co-founders do you guys have? >> You know, I think the answer's around six or seven. I forget the exact, but there's really been a lot of people around the table who've worked very hard to get this company to the point that it's at. We have long ways to go, right? But there's been a lot of people involved that have been absolutely necessary for the path we've been on so far. >> Thanks for that, Viraj, appreciate that. The first question I want to get out on the table, and then we'll get into some of the details, is take a minute to explain what you guys are doing. How did you guys get here? Obviously, multiple co-founders, sounds like a great project. The timing couldn't have been better. ChatGPT has essentially done so much public relations for the AI industry to kind of highlight this shift that's happening. It's real, we've been chronicalizing, take a minute to explain what you guys do. >> Yeah, sure, we can get started. So, yeah, when Viraj and I joined Astronomer in 2017, we really wanted to build a business around data, and we were using an open source project called Apache Airflow that we were just using sort of as customers ourselves. And over time, we realized that there was actually a market for companies who use Apache Airflow, which is a data pipeline management tool, which we'll get into, and that running Airflow is actually quite challenging, and that there's a big opportunity for us to create a set of commercial products and an opportunity to grow that open source community and actually build a company around that. So the crux of what we do is help companies run data pipelines with Apache Airflow. And certainly we've grown in our ambitions beyond that, but that's sort of the crux of what we do for folks. >> You know, data orchestration, data management has always been a big item in the old classic data infrastructure. But with AI, you're seeing a lot more emphasis on scale, tuning, training. Data orchestration is the center of the value proposition, when you're looking at coordinating resources, it's one of the most important things. Can you guys explain what data orchestration entails? What does it mean? Take us through the definition of what data orchestration entails. >> Yeah, for sure. I can take this one, and Viraj, feel free to jump in. So if you google data orchestration, here's what you're going to get. You're going to get something that says, "Data orchestration is the automated process" "for organizing silo data from numerous" "data storage points, standardizing it," "and making it accessible and prepared for data analysis." And you say, "Okay, but what does that actually mean," right, and so let's give sort of an an example. So let's say you're a business and you have sort of the following basic asks of your data team, right? Okay, give me a dashboard in Sigma, for example, for the number of customers or monthly active users, and then make sure that that gets updated on an hourly basis. And then number two, a consistent list of active customers that I have in HubSpot so that I can send them a monthly product newsletter, right? Two very basic asks for all sorts of companies and organizations. And when that data team, which has data engineers, data scientists, ML engineers, data analysts get that request, they're looking at an ecosystem of data sources that can help them get there, right? And that includes application databases, for example, that actually have in product user behavior and third party APIs from tools that the company uses that also has different attributes and qualities of those customers or users. And that data team needs to use tools like Fivetran to ingest data, a data warehouse, like Snowflake or Databricks to actually store that data and do analysis on top of it, a tool like DBT to do transformations and make sure that data is standardized in the way that it needs to be, a tool like Hightouch for reverse ETL. I mean, we could go on and on. There's so many partners of ours in this industry that are doing really, really exciting and critical things for those data movements. And the whole point here is that data teams have this plethora of tooling that they use to both ingest the right data and come up with the right interfaces to transform and interact with that data. And data orchestration, in our view, is really the heartbeat of all of those processes, right? And tangibly the unit of data orchestration is a data pipeline, a set of tasks or jobs that each do something with data over time and eventually run that on a schedule to make sure that those things are happening continuously as time moves on and the company advances. And so, for us, we're building a business around Apache Airflow, which is a workflow management tool that allows you to author, run, and monitor data pipelines. And so when we talk about data orchestration, we talk about sort of two things. One is that crux of data pipelines that, like I said, connect that large ecosystem of data tooling in your company. But number two, it's not just that data pipeline that needs to run every day, right? And Viraj will probably touch on this as we talk more about Astronomer and our value prop on top of Airflow. But then it's all the things that you need to actually run data and production and make sure that it's trustworthy, right? So it's actually not just that you're running things on a schedule, but it's also things like CICD tooling, secure secrets management, user permissions, monitoring, data lineage, documentation, things that enable other personas in your data team to actually use those tools. So long-winded way of saying that it's the heartbeat, we think, of of the data ecosystem, and certainly goes beyond scheduling, but again, data pipelines are really at the center of it. >> One of the things that jumped out, Viraj, if you can get into this, I'd like to hear more about how you guys look at all those little tools that are out. You mentioned a variety of things. You look at the data infrastructure, it's not just one stack. You've got an analytic stack, you've got a realtime stack, you've got a data lake stack, you got an AI stack potentially. I mean you have these stacks now emerging in the data world that are fundamental, that were once served by either a full package, old school software, and then a bunch of point solution. You mentioned Fivetran there, I would say in the analytics stack. Then you got S3, they're on the data lake stack. So all these things are kind of munged together. >> Yeah. >> How do you guys fit into that world? You make it easier, or like, what's the deal? >> Great question, right? And you know, I think that one of the biggest things we've found in working with customers over the last however many years is that if a data team is using a bunch of tools to get what they need done, and the number of tools they're using is growing exponentially and they're kind of roping things together here and there, that's actually a sign of a productive team, not a bad thing, right? It's because that team is moving fast. They have needs that are very specific to them, and they're trying to make something that's exactly tailored to their business. So a lot of times what we find is that customers have some sort of base layer, right? That's kind of like, it might be they're running most of the things in AWS, right? And then on top of that, they'll be using some of the things AWS offers, things like SageMaker, Redshift, whatever, but they also might need things that their cloud can't provide. Something like Fivetran, or Hightouch, those are other tools. And where data orchestration really shines, and something that we've had the pleasure of helping our customers build, is how do you take all those requirements, all those different tools and whip them together into something that fulfills a business need? So that somebody can read a dashboard and trust the number that it says, or somebody can make sure that the right emails go out to their customers. And Airflow serves as this amazing kind of glue between that data stack, right? It's to make it so that for any use case, be it ELT pipelines, or machine learning, or whatever, you need different things to do them, and Airflow helps tie them together in a way that's really specific for a individual business' needs. >> Take a step back and share the journey of what you guys went through as a company startup. So you mentioned Apache, open source. I was just having an interview with a VC, we were talking about foundational models. You got a lot of proprietary and open source development going on. It's almost the iPhone/Android moment in this whole generative space and foundational side. This is kind of important, the open source piece of it. Can you share how you guys started? And I can imagine your customers probably have their hair on fire and are probably building stuff on their own. Are you guys helping them? Take us through, 'cause you guys are on the front end of a big, big wave, and that is to make sense of the chaos, rain it in. Take us through your journey and why this is important. >> Yeah, Paola, I can take a crack at this, then I'll kind of hand it over to you to fill in whatever I miss in details. But you know, like Paola is saying, the heart of our company is open source, because we started using Airflow as an end user and started to say like, "Hey wait a second," "more and more people need this." Airflow, for background, started at Airbnb, and they were actually using that as a foundation for their whole data stack. Kind of how they made it so that they could give you recommendations, and predictions, and all of the processes that needed orchestrated. Airbnb created Airflow, gave it away to the public, and then fast forward a couple years and we're building a company around it, and we're really excited about that. >> That's a beautiful thing. That's exactly why open source is so great. >> Yeah, yeah. And for us, it's really been about watching the community and our customers take these problems, find a solution to those problems, standardize those solutions, and then building on top of that, right? So we're reaching to a point where a lot of our earlier customers who started to just using Airflow to get the base of their BI stack down and their reporting in their ELP infrastructure, they've solved that problem and now they're moving on to things like doing machine learning with their data, because now that they've built that foundation, all the connective tissue for their data arriving on time and being orchestrated correctly is happening, they can build a layer on top of that. And it's just been really, really exciting kind of watching what customers do once they're empowered to pick all the tools that they need, tie them together in the way they need to, and really deliver real value to their business. >> Can you share some of the use cases of these customers? Because I think that's where you're starting to see the innovation. What are some of the companies that you're working with, what are they doing? >> Viraj, I'll let you take that one too. (group laughs) >> So you know, a lot of it is... It goes across the gamut, right? Because it doesn't matter what you are, what you're doing with data, it needs to be orchestrated. So there's a lot of customers using us for their ETL and ELT reporting, right? Just getting data from other disparate sources into one place and then building on top of that. Be it building dashboards, answering questions for the business, building other data products and so on and so forth. From there, these use cases evolve a lot. You do see folks doing things like fraud detection, because Airflow's orchestrating how transactions go, transactions get analyzed. They do things like analyzing marketing spend to see where your highest ROI is. And then you kind of can't not talk about all of the machine learning that goes on, right? Where customers are taking data about their own customers, kind of analyze and aggregating that at scale, and trying to automate decision making processes. So it goes from your most basic, what we call data plumbing, right? Just to make sure data's moving as needed, all the ways to your more exciting expansive use cases around automated decision making and machine learning. >> And I'd say, I mean, I'd say that's one of the things that I think gets me most excited about our future, is how critical Airflow is to all of those processes, and I think when you know a tool is valuable is when something goes wrong and one of those critical processes doesn't work. And we know that our system is so mission critical to answering basic questions about your business and the growth of your company for so many organizations that we work with. So it's, I think, one of the things that gets Viraj and I and the rest of our company up every single morning is knowing how important the work that we do for all of those use cases across industries, across company sizes, and it's really quite energizing. >> It was such a big focus this year at AWS re:Invent, the role of data. And I think one of the things that's exciting about the open AI and all the movement towards large language models is that you can integrate data into these models from outside. So you're starting to see the integration easier to deal with. Still a lot of plumbing issues. So a lot of things happening. So I have to ask you guys, what is the state of the data orchestration area? Is it ready for disruption? Has it already been disrupted? Would you categorize it as a new first inning kind of opportunity, or what's the state of the data orchestration area right now? Both technically and from a business model standpoint. How would you guys describe that state of the market? >> Yeah, I mean, I think in a lot of ways, in some ways I think we're category creating. Schedulers have been around for a long time. I released a data presentation sort of on the evolution of going from something like Kron, which I think was built in like the 1970s out of Carnegie Mellon. And that's a long time ago, that's 50 years ago. So sort of like the basic need to schedule and do something with your data on a schedule is not a new concept. But to our point earlier, I think everything that you need around your ecosystem, first of all, the number of data tools and developer tooling that has come out industry has 5X'd over the last 10 years. And so obviously as that ecosystem grows, and grows, and grows, and grows, the need for orchestration only increases. And I think, as Astronomer, I think we... And we work with so many different types of companies, companies that have been around for 50 years, and companies that got started not even 12 months ago. And so I think for us it's trying to, in a ways, category create and adjust sort of what we sell and the value that we can provide for companies all across that journey. There are folks who are just getting started with orchestration, and then there's folks who have such advanced use case, 'cause they're hitting sort of a ceiling and only want to go up from there. And so I think we, as a company, care about both ends of that spectrum, and certainly want to build and continue building products for companies of all sorts, regardless of where they are on the maturity curve of data orchestration. >> That's a really good point, Paola. And I think the other thing to really take into account is it's the companies themselves, but also individuals who have to do their jobs. If you rewind the clock like 5 or 10 years ago, data engineers would be the ones responsible for orchestrating data through their org. But when we look at our customers today, it's not just data engineers anymore. There's data analysts who sit a lot closer to the business, and the data scientists who want to automate things around their models. So this idea that orchestration is this new category is right on the money. And what we're finding is the need for it is spreading to all parts of the data team, naturally where Airflow's emerged as an open source standard and we're hoping to take things to the next level. >> That's awesome. We've been up saying that the data market's kind of like the SRE with servers, right? You're going to need one person to deal with a lot of data, and that's data engineering, and then you're got to have the practitioners, the democratization. Clearly that's coming in what you're seeing. So I have to ask, how do you guys fit in from a value proposition standpoint? What's the pitch that you have to customers, or is it more inbound coming into you guys? Are you guys doing a lot of outreach, customer engagements? I'm sure they're getting a lot of great requirements from customers. What's the current value proposition? How do you guys engage? >> Yeah, I mean, there's so many... Sorry, Viraj, you can jump in. So there's so many companies using Airflow, right? So the baseline is that the open source project that is Airflow that came out of Airbnb, over five years ago at this point, has grown exponentially in users and continues to grow. And so the folks that we sell to primarily are folks who are already committed to using Apache Airflow, need data orchestration in their organization, and just want to do it better, want to do it more efficiently, want to do it without managing that infrastructure. And so our baseline proposition is for those organizations. Now to Viraj's point, obviously I think our ambitions go beyond that, both in terms of the personas that we addressed and going beyond that data engineer, but really it's to start at the baseline, as we continue to grow our our company, it's really making sure that we're adding value to folks using Airflow and help them do so in a better way, in a larger way, in a more efficient way, and that's really the crux of who we sell to. And so to answer your question on, we get a lot of inbound because they're... >> You have a built in audience. (laughs) >> The world that use it. Those are the folks who we talk to and come to our website and chat with us and get value from our content. I mean, the power of the opensource community is really just so, so big, and I think that's also one of the things that makes this job fun. >> And you guys are in a great position. Viraj, you can comment a little, get your reaction. There's been a big successful business model to starting a company around these big projects for a lot of reasons. One is open source is continuing to be great, but there's also supply chain challenges in there. There's also we want to continue more innovation and more code and keeping it free and and flowing. And then there's the commercialization of productizing it, operationalizing it. This is a huge new dynamic, I mean, in the past 5 or so years, 10 years, it's been happening all on CNCF from other areas like Apache, Linux Foundation, they're all implementing this. This is a huge opportunity for entrepreneurs to do this. >> Yeah, yeah. Open source is always going to be core to what we do, because we wouldn't exist without the open source community around us. They are huge in numbers. Oftentimes they're nameless people who are working on making something better in a way that everybody benefits from it. But open source is really hard, especially if you're a company whose core competency is running a business, right? Maybe you're running an e-commerce business, or maybe you're running, I don't know, some sort of like, any sort of business, especially if you're a company running a business, you don't really want to spend your time figuring out how to run open source software. You just want to use it, you want to use the best of it, you want to use the community around it, you want to be able to google something and get answers for it, you want the benefits of open source. You don't have the time or the resources to invest in becoming an expert in open source, right? And I think that dynamic is really what's given companies like us an ability to kind of form businesses around that in the sense that we'll make it so people get the best of both worlds. You'll get this vast open ecosystem that you can build on top of, that you can benefit from, that you can learn from. But you won't have to spend your time doing undifferentiated heavy lifting. You can do things that are just specific to your business. >> It's always been great to see that business model evolve. We used a debate 10 years ago, can there be another Red Hat? And we said, not really the same, but there'll be a lot of little ones that'll grow up to be big soon. Great stuff. Final question, can you guys share the history of the company? The milestones of Astromer's journey in data orchestration? >> Yeah, we could. So yeah, I mean, I think, so Viraj and I have obviously been at Astronomer along with our other founding team and leadership folks for over five years now. And it's been such an incredible journey of learning, of hiring really amazing people, solving, again, mission critical problems for so many types of organizations. We've had some funding that has allowed us to invest in the team that we have and in the software that we have, and that's been really phenomenal. And so that investment, I think, keeps us confident, even despite these sort of macroeconomic conditions that we're finding ourselves in. And so honestly, the milestones for us are focusing on our product, focusing on our customers over the next year, focusing on that market for us that we know can get valuable out of what we do, and making developers' lives better, and growing the open source community and making sure that everything that we're doing makes it easier for folks to get started, to contribute to the project and to feel a part of the community that we're cultivating here. >> You guys raised a little bit of money. How much have you guys raised? >> Don't know what the total is, but it's in the ballpark over $200 million. It feels good to... >> A little bit of capital. Got a little bit of cap to work with there. Great success. I know as a Series C Financing, you guys have been down. So you're up and running, what's next? What are you guys looking to do? What's the big horizon look like for you from a vision standpoint, more hiring, more product, what is some of the key things you're looking at doing? >> Yeah, it's really a little of all of the above, right? Kind of one of the best and worst things about working at earlier stage startups is there's always so much to do and you often have to just kind of figure out a way to get everything done. But really investing our product over the next, at least over the course of our company lifetime. And there's a lot of ways we want to make it more accessible to users, easier to get started with, easier to use, kind of on all areas there. And really, we really want to do more for the community, right, like I was saying, we wouldn't be anything without the large open source community around us. And we want to figure out ways to give back more in more creative ways, in more code driven ways, in more kind of events and everything else that we can keep those folks galvanized and just keep them happy using Airflow. >> Paola, any final words as we close out? >> No, I mean, I'm super excited. I think we'll keep growing the team this year. We've got a couple of offices in the the US, which we're excited about, and a fully global team that will only continue to grow. So Viraj and I are both here in New York, and we're excited to be engaging with our coworkers in person finally, after years of not doing so. We've got a bustling office in San Francisco as well. So growing those teams and continuing to hire all over the world, and really focusing on our product and the open source community is where our heads are at this year. So, excited. >> Congratulations. 200 million in funding, plus. Good runway, put that money in the bank, squirrel it away. It's a good time to kind of get some good interest on it, but still grow. Congratulations on all the work you guys do. We appreciate you and the open source community does, and good luck with the venture, continue to be successful, and we'll see you at the Startup Showcase. >> Thank you. >> Yeah, thanks so much, John. Appreciate it. >> Okay, that's the CUBE Conversation featuring astronomer.io, that's the website. Astronomer is doing well. Multiple rounds of funding, over 200 million in funding. Open source continues to lead the way in innovation. Great business model, good solution for the next gen cloud scale data operations, data stacks that are emerging. I'm John Furrier, your host, thanks for watching. (soft upbeat music)

(soft music) >> Hello everyone, welcome to this Cube conversation here from the studios of theCube in Palo Alto, California. John Furrier, your host. We're featuring a startup, Astronomer, astronomer.io is the url. Check it out. And we're going to have a great conversation around one of the most important topics hitting the industry, and that is the future of machine learning and AI and the data that powers it underneath it. There's a lot of things that need to get done, and we're excited to have some of the co-founders of Astronomer here. Viraj Parekh, who is co-founder and Paola Peraza Calderon, another co-founder, both with Astronomer. Thanks for coming on. First of all, how many co-founders do you guys have? >> You know, I think the answer's around six or seven. I forget the exact, but there's really been a lot of people around the table, who've worked very hard to get this company to the point that it's at. And we have long ways to go, right? But there's been a lot of people involved that are, have been absolutely necessary for the path we've been on so far. >> Thanks for that, Viraj, appreciate that. The first question I want to get out on the table, and then we'll get into some of the details, is take a minute to explain what you guys are doing. How did you guys get here? Obviously, multiple co-founders sounds like a great project. The timing couldn't have been better. ChatGPT has essentially done so much public relations for the AI industry. Kind of highlight this shift that's happening. It's real. We've been chronologicalizing, take a minute to explain what you guys do. >> Yeah, sure. We can get started. So yeah, when Astronomer, when Viraj and I joined Astronomer in 2017, we really wanted to build a business around data and we were using an open source project called Apache Airflow, that we were just using sort of as customers ourselves. And over time, we realized that there was actually a market for companies who use Apache Airflow, which is a data pipeline management tool, which we'll get into. And that running Airflow is actually quite challenging and that there's a lot of, a big opportunity for us to create a set of commercial products and opportunity to grow that open source community and actually build a company around that. So the crux of what we do is help companies run data pipelines with Apache Airflow. And certainly we've grown in our ambitions beyond that, but that's sort of the crux of what we do for folks. >> You know, data orchestration, data management has always been a big item, you know, in the old classic data infrastructure. But with AI you're seeing a lot more emphasis on scale, tuning, training. You know, data orchestration is the center of the value proposition when you're looking at coordinating resources, it's one of the most important things. Could you guys explain what data orchestration entails? What does it mean? Take us through the definition of what data orchestration entails. >> Yeah, for sure. I can take this one and Viraj feel free to jump in. So if you google data orchestration, you know, here's what you're going to get. You're going to get something that says, data orchestration is the automated process for organizing silo data from numerous data storage points to organizing it and making it accessible and prepared for data analysis. And you say, okay, but what does that actually mean, right? And so let's give sort of an example. So let's say you're a business and you have sort of the following basic asks of your data team, right? Hey, give me a dashboard in Sigma, for example, for the number of customers or monthly active users and then make sure that that gets updated on an hourly basis. And then number two, a consistent list of active customers that I have in HubSpot so that I can send them a monthly product newsletter, right? Two very basic asks for all sorts of companies and organizations. And when that data team, which has data engineers, data scientists, ML engineers, data analysts get that request, they're looking at an ecosystem of data sources that can help them get there, right? And that includes application databases, for example, that actually have end product user behavior and third party APIs from tools that the company uses that also has different attributes and qualities of those customers or users. And that data team needs to use tools like Fivetran, to ingest data, a data warehouse like Snowflake or Databricks to actually store that data and do analysis on top of it, a tool like DBT to do transformations and make sure that that data is standardized in the way that it needs to be, a tool like Hightouch for reverse ETL. I mean, we could go on and on. There's so many partners of ours in this industry that are doing really, really exciting and critical things for those data movements. And the whole point here is that, you know, data teams have this plethora of tooling that they use to both ingest the right data and come up with the right interfaces to transform and interact with that data. And data orchestration in our view is really the heartbeat of all of those processes, right? And tangibly the unit of data orchestration, you know, is a data pipeline, a set of tasks or jobs that each do something with data over time and eventually run that on a schedule to make sure that those things are happening continuously as time moves on. And, you know, the company advances. And so, you know, for us, we're building a business around Apache Airflow, which is a workflow management tool that allows you to author, run and monitor data pipelines. And so when we talk about data orchestration, we talk about sort of two things. One is that crux of data pipelines that, like I said, connect that large ecosystem of data tooling in your company. But number two, it's not just that data pipeline that needs to run every day, right? And Viraj will probably touch on this as we talk more about Astronomer and our value prop on top of Airflow. But then it's all the things that you need to actually run data and production and make sure that it's trustworthy, right? So it's actually not just that you're running things on a schedule, but it's also things like CI/CD tooling, right? Secure secrets management, user permissions, monitoring, data lineage, documentation, things that enable other personas in your data team to actually use those tools. So long-winded way of saying that, it's the heartbeat that we think of the data ecosystem and certainly goes beyond scheduling, but again, data pipelines are really at the center of it. >> You know, one of the things that jumped out Viraj, if you can get into this, I'd like to hear more about how you guys look at all those little tools that are out there. You mentioned a variety of things. You know, if you look at the data infrastructure, it's not just one stack. You've got an analytic stack, you've got a realtime stack, you've got a data lake stack, you got an AI stack potentially. I mean you have these stacks now emerging in the data world that are >> Yeah. - >> fundamental, but we're once served by either a full package, old school software, and then a bunch of point solution. You mentioned Fivetran there, I would say in the analytics stack. Then you got, you know, S3, they're on the data lake stack. So all these things are kind of munged together. >> Yeah. >> How do you guys fit into that world? You make it easier or like, what's the deal? >> Great question, right? And you know, I think that one of the biggest things we've found in working with customers over, you know, the last however many years, is that like if a data team is using a bunch of tools to get what they need done and the number of tools they're using is growing exponentially and they're kind of roping things together here and there, that's actually a sign of a productive team, not a bad thing, right? It's because that team is moving fast. They have needs that are very specific to them and they're trying to make something that's exactly tailored to their business. So a lot of times what we find is that customers have like some sort of base layer, right? That's kind of like, you know, it might be they're running most of the things in AWS, right? And then on top of that, they'll be using some of the things AWS offers, you know, things like SageMaker, Redshift, whatever. But they also might need things that their Cloud can't provide, you know, something like Fivetran or Hightouch or anything of those other tools and where data orchestration really shines, right? And something that we've had the pleasure of helping our customers build, is how do you take all those requirements, all those different tools and whip them together into something that fulfills a business need, right? Something that makes it so that somebody can read a dashboard and trust the number that it says or somebody can make sure that the right emails go out to their customers. And Airflow serves as this amazing kind of glue between that data stack, right? It's to make it so that for any use case, be it ELT pipelines or machine learning or whatever, you need different things to do them and Airflow helps tie them together in a way that's really specific for a individual business's needs. >> Take a step back and share the journey of what your guys went through as a company startup. So you mentioned Apache open source, you know, we were just, I was just having an interview with the VC, we were talking about foundational models. You got a lot of proprietary and open source development going on. It's almost the iPhone, Android moment in this whole generative space and foundational side. This is kind of important, the open source piece of it. Can you share how you guys started? And I can imagine your customers probably have their hair on fire and are probably building stuff on their own. How do you guys, are you guys helping them? Take us through, 'cuz you guys are on the front end of a big, big wave and that is to make sense of the chaos, reigning it in. Take us through your journey and why this is important. >> Yeah Paola, I can take a crack at this and then I'll kind of hand it over to you to fill in whatever I miss in details. But you know, like Paola is saying, the heart of our company is open source because we started using Airflow as an end user and started to say like, "Hey wait a second". Like more and more people need this. Airflow, for background, started at Airbnb and they were actually using that as the foundation for their whole data stack. Kind of how they made it so that they could give you recommendations and predictions and all of the processes that need to be or needed to be orchestrated. Airbnb created Airflow, gave it away to the public and then, you know, fast forward a couple years and you know, we're building a company around it and we're really excited about that. >> That's a beautiful thing. That's exactly why open source is so great. >> Yeah, yeah. And for us it's really been about like watching the community and our customers take these problems, find solution to those problems, build standardized solutions, and then building on top of that, right? So we're reaching to a point where a lot of our earlier customers who started to just using Airflow to get the base of their BI stack down and their reporting and their ELP infrastructure, you know, they've solved that problem and now they're moving onto things like doing machine learning with their data, right? Because now that they've built that foundation, all the connective tissue for their data arriving on time and being orchestrated correctly is happening, they can build the layer on top of that. And it's just been really, really exciting kind of watching what customers do once they're empowered to pick all the tools that they need, tie them together in the way they need to, and really deliver real value to their business. >> Can you share some of the use cases of these customers? Because I think that's where you're starting to see the innovation. What are some of the companies that you're working with, what are they doing? >> Raj, I'll let you take that one too. (all laughing) >> Yeah. (all laughing) So you know, a lot of it is, it goes across the gamut, right? Because all doesn't matter what you are, what you're doing with data, it needs to be orchestrated. So there's a lot of customers using us for their ETL and ELT reporting, right? Just getting data from all the disparate sources into one place and then building on top of that, be it building dashboards, answering questions for the business, building other data products and so on and so forth. From there, these use cases evolve a lot. You do see folks doing things like fraud detection because Airflow's orchestrating how transactions go. Transactions get analyzed, they do things like analyzing marketing spend to see where your highest ROI is. And then, you know, you kind of can't not talk about all of the machine learning that goes on, right? Where customers are taking data about their own customers kind of analyze and aggregating that at scale and trying to automate decision making processes. So it goes from your most basic, what we call like data plumbing, right? Just to make sure data's moving as needed. All the ways to your more exciting and sexy use cases around like automated decision making and machine learning. >> And I'd say, I mean, I'd say that's one of the things that I think gets me most excited about our future is how critical Airflow is to all of those processes, you know? And I think when, you know, you know a tool is valuable is when something goes wrong and one of those critical processes doesn't work. And we know that our system is so mission critical to answering basic, you know, questions about your business and the growth of your company for so many organizations that we work with. So it's, I think one of the things that gets Viraj and I, and the rest of our company up every single morning, is knowing how important the work that we do for all of those use cases across industries, across company sizes. And it's really quite energizing. >> It was such a big focus this year at AWS re:Invent, the role of data. And I think one of the things that's exciting about the open AI and all the movement towards large language models, is that you can integrate data into these models, right? From outside, right? So you're starting to see the integration easier to deal with, still a lot of plumbing issues. So a lot of things happening. So I have to ask you guys, what is the state of the data orchestration area? Is it ready for disruption? Is it already been disrupted? Would you categorize it as a new first inning kind of opportunity or what's the state of the data orchestration area right now? Both, you know, technically and from a business model standpoint, how would you guys describe that state of the market? >> Yeah, I mean I think, I think in a lot of ways we're, in some ways I think we're categoric rating, you know, schedulers have been around for a long time. I recently did a presentation sort of on the evolution of going from, you know, something like KRON, which I think was built in like the 1970s out of Carnegie Mellon. And you know, that's a long time ago. That's 50 years ago. So it's sort of like the basic need to schedule and do something with your data on a schedule is not a new concept. But to our point earlier, I think everything that you need around your ecosystem, first of all, the number of data tools and developer tooling that has come out the industry has, you know, has some 5X over the last 10 years. And so obviously as that ecosystem grows and grows and grows and grows, the need for orchestration only increases. And I think, you know, as Astronomer, I think we, and there's, we work with so many different types of companies, companies that have been around for 50 years and companies that got started, you know, not even 12 months ago. And so I think for us, it's trying to always category create and adjust sort of what we sell and the value that we can provide for companies all across that journey. There are folks who are just getting started with orchestration and then there's folks who have such advanced use case 'cuz they're hitting sort of a ceiling and only want to go up from there. And so I think we as a company, care about both ends of that spectrum and certainly have want to build and continue building products for companies of all sorts, regardless of where they are on the maturity curve of data orchestration. >> That's a really good point Paola. And I think the other thing to really take into account is it's the companies themselves, but also individuals who have to do their jobs. You know, if you rewind the clock like five or 10 years ago, data engineers would be the ones responsible for orchestrating data through their org. But when we look at our customers today, it's not just data engineers anymore. There's data analysts who sit a lot closer to the business and the data scientists who want to automate things around their models. So this idea that orchestration is this new category is spot on, is right on the money. And what we're finding is it's spreading, the need for it, is spreading to all parts of the data team naturally where Airflows have emerged as an open source standard and we're hoping to take things to the next level. >> That's awesome. You know, we've been up saying that the data market's kind of like the SRE with servers, right? You're going to need one person to deal with a lot of data and that's data engineering and then you're going to have the practitioners, the democratization. Clearly that's coming in what you're seeing. So I got to ask, how do you guys fit in from a value proposition standpoint? What's the pitch that you have to customers or is it more inbound coming into you guys? Are you guys doing a lot of outreach, customer engagements? I'm sure they're getting a lot of great requirements from customers. What's the current value proposition? How do you guys engage? >> Yeah, I mean we've, there's so many, there's so many. Sorry Raj, you can jump in. - >> It's okay. So there's so many companies using Airflow, right? So our, the baseline is that the open source project that is Airflow that was, that came out of Airbnb, you know, over five years ago at this point, has grown exponentially in users and continues to grow. And so the folks that we sell to primarily are folks who are already committed to using Apache Airflow, need data orchestration in the organization and just want to do it better, want to do it more efficiently, want to do it without managing that infrastructure. And so our baseline proposition is for those organizations. Now to Raj's point, obviously I think our ambitions go beyond that, both in terms of the personas that we addressed and going beyond that data engineer, but really it's for, to start at the baseline. You know, as we continue to grow our company, it's really making sure that we're adding value to folks using Airflow and help them do so in a better way, in a larger way and a more efficient way. And that's really the crux of who we sell to. And so to answer your question on, we actually, we get a lot of inbound because they're are so many - >> A built-in audience. >> In the world that use it, that those are the folks who we talk to and come to our website and chat with us and get value from our content. I mean the power of the open source community is really just so, so big. And I think that's also one of the things that makes this job fun, so. >> And you guys are in a great position, Viraj, you can comment, to get your reaction. There's been a big successful business model to starting a company around these big projects for a lot of reasons. One is open source is continuing to be great, but there's also supply chain challenges in there. There's also, you know, we want to continue more innovation and more code and keeping it free and and flowing. And then there's the commercialization of product-izing it, operationalizing it. This is a huge new dynamic. I mean, in the past, you know, five or so years, 10 years, it's been happening all on CNCF from other areas like Apache, Linux Foundation, they're all implementing this. This is a huge opportunity for entrepreneurs to do this. >> Yeah, yeah. Open source is always going to be core to what we do because, you know, we wouldn't exist without the open source community around us. They are huge in numbers. Oftentimes they're nameless people who are working on making something better in a way that everybody benefits from it. But open source is really hard, especially if you're a company whose core competency is running a business, right? Maybe you're running e-commerce business or maybe you're running, I don't know, some sort of like any sort of business, especially if you're a company running a business, you don't really want to spend your time figuring out how to run open source software. You just want to use it, you want to use the best of it, you want to use the community around it. You want to take, you want to be able to google something and get answers for it. You want the benefits of open source. You don't want to have, you don't have the time or the resources to invest in becoming an expert in open source, right? And I think that dynamic is really what's given companies like us an ability to kind of form businesses around that, in the sense that we'll make it so people get the best of both worlds. You'll get this vast open ecosystem that you can build on top of, you can benefit from, that you can learn from, but you won't have to spend your time doing undifferentiated heavy lifting. You can do things that are just specific to your business. >> It's always been great to see that business model evolved. We used to debate 10 years ago, can there be another red hat? And we said, not really the same, but there'll be a lot of little ones that'll grow up to be big soon. Great stuff. Final question, can you guys share the history of the company, the milestones of the Astronomer's journey in data orchestration? >> Yeah, we could. So yeah, I mean, I think, so Raj and I have obviously been at astronomer along with our other founding team and leadership folks, for over five years now. And it's been such an incredible journey of learning, of hiring really amazing people. Solving again, mission critical problems for so many types of organizations. You know, we've had some funding that has allowed us to invest in the team that we have and in the software that we have. And that's been really phenomenal. And so that investment, I think, keeps us confident even despite these sort of macroeconomic conditions that we're finding ourselves in. And so honestly, the milestones for us are focusing on our product, focusing on our customers over the next year, focusing on that market for us, that we know can get value out of what we do. And making developers' lives better and growing the open source community, you know, and making sure that everything that we're doing makes it easier for folks to get started to contribute to the project and to feel a part of the community that we're cultivating here. >> You guys raised a little bit of money. How much have you guys raised? >> I forget what the total is, but it's in the ballpark of 200, over $200 million. So it feels good - >> A little bit of capital. Got a little bit of cash to work with there. Great success. I know it's a Series C financing, you guys been down, so you're up and running. What's next? What are you guys looking to do? What's the big horizon look like for you? And from a vision standpoint, more hiring, more product, what is some of the key things you're looking at doing? >> Yeah, it's really a little of all of the above, right? Like, kind of one of the best and worst things about working at earlier stage startups is there's always so much to do and you often have to just kind of figure out a way to get everything done, but really invest in our product over the next, at least the next, over the course of our company lifetime. And there's a lot of ways we wanting to just make it more accessible to users, easier to get started with, easier to use all kind of on all areas there. And really, we really want to do more for the community, right? Like I was saying, we wouldn't be anything without the large open source community around us. And we want to figure out ways to give back more in more creative ways, in more code driven ways and more kind of events and everything else that we can do to keep those folks galvanized and just keeping them happy using Airflow. >> Paola, any final words as we close out? >> No, I mean, I'm super excited. You know, I think we'll keep growing the team this year. We've got a couple of offices in the US which we're excited about, and a fully global team that will only continue to grow. So Viraj and I are both here in New York and we're excited to be engaging with our coworkers in person. Finally, after years of not doing so, we've got a bustling office in San Francisco as well. So growing those teams and continuing to hire all over the world and really focusing on our product and the open source community is where our heads are at this year, so. >> Congratulations. - >> Excited. 200 million in funding plus good runway. Put that money in the bank, squirrel it away. You know, it's good to kind of get some good interest on it, but still grow. Congratulations on all the work you guys do. We appreciate you and the open sourced community does and good luck with the venture. Continue to be successful and we'll see you at the Startup Showcase. >> Thank you. - >> Yeah, thanks so much, John. Appreciate it. - >> It's theCube conversation, featuring astronomer.io, that's the website. Astronomer is doing well. Multiple rounds of funding, over 200 million in funding. Open source continues to lead the way in innovation. Great business model. Good solution for the next gen, Cloud, scale, data operations, data stacks that are emerging. I'm John Furrier, your host. Thanks for watching. (soft music)

(upbeat music) >> Hello, everyone. Welcome back to theCUBE's presentation of the AWS Startup Showcase. This is season two, episode four of the ongoing series covering the exciting startups from the AWS ecosystem. And talking about cybersecurity. I'm your host, John Furrier. Excited to have two great guests. Ed Casmer, founder and CEO of Cloud Storage Security, back CUBE alumni, and also James Johnson, AVP of Research and Development at iPipeline. Here to talk about cloud storage security antivirus on S3. James, thanks for joining us today. >> Thank you, John. >> Thank you. >> So the topic here is cloud security, storage security. Ed, we had a great CUBE conversation previously, earlier in the month. Companies are modernizing their apps and migrating the cloud. That's fact. Everyone kind of knows that. >> Yeah. >> Been there, done that. Clouds have the infrastructure, they got the OS, they got protection, but the end of the day, the companies are responsible and they're on the hook for their own security of their data. And this is becoming more permanent now that you have hybrid cloud, cloud operations, cloud native applications. This is the core focus right now in the next five years. This is what everyone's talking about. Architecture, how to build apps, workflows, team formation. Everything's being refactored around this. Can you talk about how organizations are adjusting and how they view their data security in light of how applications are being built and specifically around the goodness of say S3? >> Yep, absolutely. Thank you for that. So we've seen S3 grow 20,000% over the last 10 years. And that's primarily because companies like James with iPipeline are delivering solutions that are leveraging this object storage more and above the others. When we look at protection, we typically fall into a couple of categories. The first one is, we have folks that are worried about the access of the data. How are they dealing with it? And so they're looking at configuration aspects. But the big thing that we're seeing is that customers are blind to the fact that the data itself must also be protected and looked at. And so we find these customers who do come to the realization that it needs to happen, finding out, asking themselves, how do I solve for this? And so they need lightweight, cloud native built solutions to deliver that. >> So what's the blind spot? You mentioned there's a blind spot. They're kind of blind to that. What specifically are you seeing? >> Well so, when we get into these conversations, the first thing that we see with customers is I need to predict how I access it. This is everyone's conversation. Who are my users? How do they get into my data? How am I controlling that policy? Am I making sure there's no east-west traffic there, once I've blocked the north-south? But what we really find is that the data is the key packet of this whole process. It's what gets consumed by the downstream users. Whether that's an employee, a customer, a partner. And so it's really, the blind spot is the fact that we find most customers not looking at whether that data is safe to use. >> It's interesting. When you talk about that, I think about all the recent breaches and incidents. "Incidents," they call them. >> Yeah. >> They've really been around user configurations. S3 buckets not configured properly. >> Absolutely. >> And this brings up what you're saying, is that the users and the customers have to be responsible for the configurations, the encryption, the malware aspect of it. Don't just hope that AWS has the magic to do it. Is that kind of what you're getting at here? Is that the similar, am I correlating that properly? >> Absolutely. That's perfect. And we've seen it. We've had our own customers, luckily iPipeline's not one of them, that have actually infected their end users because they weren't looking at the data. >> And that's a huge issue. So James, let's get in, you're a customer partner. Talk about your relationship with these guys and what's it all about? >> Yeah, well, my pipeline is building a digital ecosystem for life insurance and wealth management industries to enable the sale of life insurance to under-insured and uninsured Americans, to make sure that they have the coverage that they need, should something happen. And our solutions have been around for many years. In a traditional data center type of an implementation. And we're in process now of migrating that to the cloud, moving it to AWS, in order to give our customers a better experience, a better resiliency, better reliability. And with that, we have to change the way that we approach file storage and how we approach scanning for vulnerabilities in those files that might come to us via feeds from third parties or that are uploaded directly by end users that come to us from a source that we don't control. So it was really necessary for us to identify a solution that both solved for these vulnerability scanning needs, as well as enabling us to leverage the capabilities that we get with other aspects of our move to the cloud and being able to automatically scale based on load, based on need, to ensure that we get the performance that our customers are looking for. >> So tell me about your journey to the cloud, migrating to the cloud and how you're using S3 specifically. What led you to determine the need for the cloud based AV solution? >> So when we looked to begin moving our applications to the cloud, one of the realizations that we had is that our approach to storing certain types of data was a bit archaic. We were storing binary files in a database, which is not the most efficient way to do things. And we were scanning them with the traditional antivirus engines that would've been scaled in traditional ways. So as our need grew, we would need to spin up additional instances of those engines to keep up with load. And we wanted a solution that was cloud native and would allow us to scan more dynamically without having to manage the underlying details of how many engines do I need to have running for a particular load at a particular time and being able to scan dynamically. And also being able to move that out of the application layer, being able to scan those files behind the scenes. So scanning in, when the file's been saved in S3, it allows us to scan and release the file once it's been deemed safe rather than blocking the user while they wait for that scan to take place. >> Awesome. Well, thanks for sharing that. I got to ask Ed, and James, same question next. It's, how does all this factor in to audits and self compliance? Because when you start getting into this level of sophistication, I'm sure it probably impacts reporting workflows. Can you guys share the impact on that piece of it? The reporting? >> Yeah. I'll start with a comment and James will have more applicable things to say. But we're seeing two things. One is, you don't want to be the vendor whose name is in the news for infecting your customer base. So that's number one. So you have to put something like this in place and figure that out. The second part is, we do hear that under SOC 2, under PCI, different aspects of it, there are scanning requirements on your data. Traditionally, we've looked at that as endpoint data and the data that you see in your on-prem world. It doesn't translate as directly to cloud data, but it's certainly applicable. And if you want to achieve SOC 2 or you want to achieve some of these other pieces, you have to be scanning your data as well. >> Furrier: James, what's your take? As practitioner, you're living it. >> Yeah, that's exactly right. There are a number of audits that we go through where this is a question that comes up both from a SOC perspective, as well as our individual customers who reach out and they want to know where we stand from a security perspective and a compliance perspective. And very often this is a question of how are you ensuring that data that is uploaded into the application is safe and doesn't contain any vulnerabilities. >> James, if you don't mind me asking, I have to kind of inquire because I can imagine that you have users on your system but also you have third parties, relationships. How does that impact this? What's the connection? >> That's a good question. We receive data from a number of different locations from our customers directly, from their users and from partners that we have as well as partners that our customers have. And as we ingest that data, from an implementation perspective, the way we've approached this, there's a minimal impact there in each one of those integrations. Because everything comes into the S3 bucket and is scanned before it is available for consumption or distribution. But this allows us to ensure that no matter where that data is coming from, that we are able to verify that it is safe before we allow it into our systems or allow it to continue on to another third party whether that's our customer or somebody else. >> Yeah, I don't mean to get in the weeds there, but it's one of those things where, this is what people are experiencing right now. Ed, we talked about this before. It's not just siloed data anymore. It's interactive data. It's third party data from multiple sources. This is a scanning requirement. >> Agreed. I find it interesting too. I think James brings it up. We've had it in previous conversations that not all data's created equal. Data that comes from third parties that you're not in control of, you feel like you have to scan. And other data you may generate internally. You don't have to be as compelled to scan that although it's a good idea, but you can, as long as you can sift through and determine which data is which and process it appropriately, then you're in good shape. >> Well, James, you're living the cloud security, storage security situation here. I got to ask you, if you zoom out and not get in the weeds and look at the board room or the management conversation. Tell me about how you guys view the data security problem. I mean, obviously it's important. So can you give us a level of how important it is for iPipeline and with your customers and where does this S3 piece fit in? I mean, when you guys look at this holistically, for data security, what's the view, what's the conversation like? >> Yeah. Well, data security is critical. As Ed mentioned a few minutes ago, you don't want to be the company that's in the news because some data was exposed. That's something that nobody has the appetite for. And so data security is first and foremost in everything that we do. And that's really where this solution came into play, in making sure that we had not only a solution but we had a solution that was the right fit for the technology that we're using. There are a number of options. Some of them have been around for a while. But this was focused on S3, which we were using to store these documents that are coming from many different sources. And we have to take all the precautions we can to ensure that something that is malicious doesn't make its way into our ecosystem or into our customers' ecosystems through us. >> What's the primary use case that you see the value here with these guys? What's the aha moment that you had? >> With the cloud storage security specifically, it goes beyond the security aspects of being able to scan for vulnerable files, which is, there are a number of options and they're one of those. But for us, the key was being able to scale dynamically without committing to a particular load whether that's under committing or overcommitting. As we move our applications from a traditional data center type of installation to AWS, we anticipated a lot of growth over time and being able to scale up very dynamically, literally moving a slider within the admin console, was key to us to be able to meet our customer's needs without overspending, by building up something that was dramatically larger than we needed in our initial rollout. >> Not a bad testimonial there, Ed. >> I mean, I agree. >> This really highlights the applications using S3 more in the file workflow for the application in real time. This is where you start to see the rise of ransomware other issues. And scale matters. Can you share your thoughts and reaction to what James just said? >> Yeah. I think it's critical. As the popularity of S3 has increased, so has the fact that it's an attack vector now. And people are going after it whether that's to plant bad malicious files, whether it's to replace code segments that are downloaded and used in other applications, it is a very critical piece. And when you look at scale and you look at the cloud native capability, there are lots of ways to solve it. You can dig a hole with a spoon, but a shovel works a lot better. And in this case, we take a simple example like James. They did a weekend migration, so they've got new data coming in all the time, but we did a massive migration 5,000 files a minute being ingested. And like he said, with a couple of clicks, scale up, process that over sustained period of time and then scale back down. So I've said it before, I said it on the previous one. We don't want to get in the way of someone's workflow. We want to help them secure their data and do it in a timely fashion that they can continue with their proper processing and their normal customer responses. >> Frictionless has to be key. I know you're in the marketplace with your antivirus for S3 on the AWS. People can just download it. So people are interested, go check it out. James, I got to ask you and maybe Ed can chime in over the top, but it seems so obvious. Data. Secure the data. Why is it so hard? Why isn't this so obvious? What's the problem? Why is it so difficult? Why are there so many different solutions? It just seems so obvious. You know, you got ransomware, you got injection of different malicious payloads. There's a ton of things going on around the data. Why is, this so obvious? Why isn't it solved? >> Well, I think there have been solutions available for a long time. But the challenge, the difficulty that I see, is that it is a moving target. As bad actors learn new vulnerabilities, new approaches and as new technology becomes available, that opens additional attack vectors. >> Yeah. >> That's the challenge, is keeping up on the changing world including keeping up on the new ways that people are finding to exploit vulnerabilities. >> And you got sensitive data at iPipeline. You do a lot of insurance, wealth management, all kinds of sensitive data, super valuable. This brings me up, reminds me of the Sony hack Ed, years ago. Companies are responsible for their own militia. I mean, cybersecurity is no government help for sure. I mean, companies are on the hook. As we mentioned earlier at the top of this interview, this really is highlighted that IT departments have to evolve to large scale cloud, cloud native applications, automation, AI machine learning all built in, to keep up at the scale. But also from a defense standpoint. I mean, James you're out there, you're in the front lines, you got to defend yourself basically, and you got to engineer it. >> A hundred percent. And just to go on top of what James was saying is, I think there, one of the big factors and we've seen this. There's skill shortages out there. There's also just a pure lack of understanding. When we look at Amazon S3 or object storage in general, it's not an executable file system. So people sort of assume that, oh, I'm safe. It's not executable. So I'm not worried about it traversing my storage network. And they also probably have the assumption that the cloud providers, Amazon is taking care of this for them. And so it's this aha moment. Like you mentioned earlier, that you start to think, oh it's not about where the data is sitting per se. It's about scanning it as close to the storage spot. So when it gets to the end user, it's safe and secure. And you can't rely on the end user's environment and system to be in place and up to date to handle it. So it's that really, that lack of understanding that drives some of these folks into this. But for a while, we'll walk into customers and they'll say the same thing you said, John. Why haven't I been doing this for so long? And it's because they didn't understand that it was such a risk. That's where that blind spot comes in. >> James, it's just a final note on your environment. What's your goals for the next year? How's things going over there on your side? How you look at the security posture? What's on your agenda for the next year? How are you guys looking at the next level? >> Yeah. Well, our goal as it relates to this is to continue to move our existing applications over to AWS to run natively there. Which includes moving more data into S3 and leveraging the cloud storage security solution to scan that and ensure that there are no vulnerabilities that are getting in. >> And the ingestion, is there like a bottlenecks log jams? How do you guys see that scaling up? I mean, what's the strategy there? Just add more S3? >> Well, S3 itself scales automatically for us and the cloud storage solution gives us leverage to pull to do that. As Ed mentioned, we ingested a large amount of data during our initial migration which created a bottleneck for us. As we were preparing to move our users over, we were able to make an adjustment in the admin console and spin up additional processes entirely behind the scenes and broke the log jam. So I don't see any immediate concerns there, being able to handle the load. >> The term cloud native and hyperscale native, cloud native, one cloud's hybrid. All these things are native. We have antivirus native coming soon. And I mean, this is what we're basically doing is making it native into the workflows. Security native. And soon there's going to be security clouds out there. We're starting to see the rise of these new solutions. Can you guys share any thoughts or vision around how you see the industry evolving and what's needed? What's working and what's needed? Ed, we'll start with you. What's your vision? >> So I think the notion of being able to look at and view the management plane and control that has been where we're at right now. That's what everyone seems to be doing and going after. I think there are niche plays coming up. Storage is one of them, but we're going to get to a point where storage is just a blanket term for where you put your stuff. I mean, it kind of already is that. But in AWS, it's going to be less about S3. Less about work docs, less about EVS. It's going to be just storage and you're going to need a solution that can span all of that to go along with where we're already at the management plane. We're going to keep growing the data plane. >> James, what's your vision for what's needed in the industry? What's the gaps, what's working, and where do you see things going? >> Yeah, well, I think on the security front specifically, Ed's probably a little bit better equipped to speak to them than I am since that his primary focus. But I see the need for just expanded solutions that are cloud native that fit and fit nicely with the Amazon technologies. Whether that comes from Amazon or other partners like Cloud Storage Security to fill those gaps. We are focused on the financial services and insurance industries. That's our niche. And we look to other partners like Ed to help be the experts in these areas. And so that's really what I'm looking for, is the experts that we can partner with that are going to help fill those gaps as they come up and as they change in the future. >> Well, James, I really appreciate you coming on, sharing your story and I'll give you the final word. Put a quick, spend a minute to talk about the company. I know Cloud Storage Security is an AWS partner with the security software competency and is one of I think 16 partners listed in the competency and the data category. So take a minute to explain what's going on with the company, where people can find more information, how they buy and consume the products. >> Okay. >> Put the plug in. >> Yeah, thank you for that. So we are a fast growing startup. We've been in business for two and a half years now. We have achieved our security competency as John indicated. We're one of 16 data protection security competent ISV vendors globally. And our goal is to expand and grow a platform that spans all storage types that you're going to be dealing with and answer basic questions. What do I have and where is it? Is it safe to use? And am I in proper control of it? Am I being alerted appropriate? So we're building this storage security platform, very laser focused on the storage aspect of it. And if people want to find out more information, you're more than welcome to go and try the software out on Amazon marketplace. That's basically where we do most of our transacting. So find it there. Start of free trial. Reach out to us directly from our website. We are happy to help you in any way that you need it. Whether that's storage assessments, figuring out what data is important to you and how to protect it. >> All right, Ed. Thank you so much. Ed Casmer, founder and CEO of Cloud Storage Security. And of course James Johnson, AVP of Research and Development, iPipeline customer. Gentlemen, thank you for sharing your story and featuring the company and the value proposition, certainly needed. This is season two, episode four. Thanks for joining us. Appreciate it. >> Casmer: Thanks John. >> Okay. I'm John Furrier. That is a wrap for this segment of the cybersecurity season two, episode four. The ongoing series covering the exciting startups from Amazon's ecosystem. Thanks for watching. (upbeat music)

>>Hi, everyone. Welcome to the Cube's presentation of the AWS startup showcase. This is season two, episode four of our ongoing series, where we're talking with exciting partners in the AWS ecosystem. This topic on this episode is cybersecurity detect and protect against threats. I have two guests here with me today from hunters, please. Welcome. Laal Asher Doan, the CMO and Oprah. Geier the VP of product management. Thank you both so much for joining us today. >>Thank you for having us, Lisa, >>Our pleasure. Laal let's go ahead and start with you. Give the audience an overview of hunters. What does it do? When was it founded? What's the vision, all that good stuff. >>So hunters was founded in 20 18 2. Co-founders coming out of unit 8,200 in the Israeli defense force, the founders and people in engineering and R and D are mostly coming from both offensive cybersecurity, as well as defensive threat hunting, advanced operations, or, or being able to see in response to advanced attack and with the knowledge that they came with. They wanted to enable security teams in organizations, not just those that are coming from, you know, military background, but those that actually need to defend day in and day out against the growing cyber attacks that are growing in sophistication in the numbers of attacks. And we all know that every organization nowaday is being targeted, is it run somewhere more sophisticated attacks. So this thing has become a real challenge and we all know those challenges that the industry is facing with talent scarcity, with lack of the knowledge and expertise needing to address this. >>So came in with this mindset of, we wanna bring our expertise into the field, build it into a platform into a tool that will actually serve security teams in organizations around the world to defend against cyber attacks. So born and raised in Tel Aviv became a global company. Recently raised a serious CEO of funding funded by the world's rated VCs from stripes, wild benches, supported by snowflake data breaks and Microsoft M 12 also as strategic partners. And we now have broad variety of customers from all industries around the world, from tech to retail, to eCommerce, to banks that we work closely with. So very exciting times, and we are very excited to share today how we work with AWS customers to support the environments. >>Yeah, we're gonna unpack that. So really solid foundation, the company was built on only a few years ago. Laal was there, why a new approach was there a compelling event? Obviously we've seen dramatic changes in the threat landscape in recent years, ransomware becoming a, when it happens to us, not if, but any sort of compelling event that really led the founders to go, ah, this new approach. We gotta go this direction. >>Absolutely. We've seen a tremendous shift of organizations from cloud adoption to adoption of more security tools, both create a scenario, which the tool sets that are currently being used by security organizations. The security teams are not sufficient anymore. They cannot deal with the plethora of the variety of data. They cannot deal with the scale that is needed. And the security teams are really under a tremendous burden of tweaking tools that they have in their environment without too much of automation with a lot of manual work processes. So we've seen a lot of points where the current technology is not supporting the people and the processes that need to support security operations. And with that offer and his product team kind of set a vision of what a new platform should come to replace and enhance what teams are using these days. >>Excellent. Oprah, that's a perfect segue to bring you into the conversation. Talk about that vision and some of those really key challenges and problems that hunters are solving for organizations across any industry. >>Yeah. So as Lial mentioned, and it was very rightful, the problem with the, with the SIM space, that's the, the space that we're disrupting is the well known secret around is it's a broken space. There's a lot of competitors. There's a lot of vendors out there. It's one of the most mature, presumably mature markets in cybersecurity. But it seems like that every single customer and organization we talk to, they don't really like their existing solution. It doesn't really fit what they need. It's a very painful process and it's painful all across their workflow from the time they ingest the data. Everybody knows if you ever had a SIM solution or a soft platform, just getting the data into your environment can take the most amount of your time. The, the, the lion share of whatever your engineers are working on will go to getting the data into the system. >>And then, then keeping it there. It's this black hole that you have to keep feeding with more and more resources as you go along. It's an endless task with a lot of moving pieces, and it's very, very painful before you even get a single moment of value of security use case from your product. That's a big, painful piece. What you then see is once they set it up, their detection engineering is so far behind the curve because of all the different times of things they need to take care of. It used to be limited attack surface. We all know the attack surface here today is enormous. Especially when you talk about something like AWS, there's new services, new things, all the time, more accounts, more things. It keeps moving a lot and keeping track of that. And having someone that can actually look into a new threat when it's released, look into a new attack service, analyze it, deploying the detections in time, test and tweaked and all those things. >>Most organizations don't, don't even how to start approaching this problem. And, and, and that's a big pain for them. When they finally get to investigating something, they lack the context and the knowledge of how to investigate. They have very limited information coming to them and they go on this hunting chase of not hunting the attackers, but hunting the data, looking for the bits and pieces they're missing to complete the picture. It's like this bad boss that gives you very little instructions or, or guidelines. And then you need to kind of try to figure out what is it that they asked, right? That's the same thing with trying to do triaging with very minimal context. You look at the IP and then you try to figure out, you look at the hash, you look at all these different artifacts and you try to figure out yourself, you have very limited insights. And the worst is when you're under the gun, when there's a new emerging threat, that happens like a log for shell. And now you're under the gun and the entire company's looking at you and saying, are we impacted? What's going on? What should we doing? So from, from start to finish, it's a very painful process that impacts everybody in the security organization. A lot of, a lot of cumbersome work with a lot of frustration >>And it's comp companies in any industry over don't have time. You talked about some of the, the time involved here in the lag, and there isn't time in the very dynamic threat landscape that customers are living in. Let's all question for you is your primary target audience, existing SIM customers, cause over mentioned the disruption of the SIM market. I'm just wanting to understand in terms of who you're targeting, what does that look like? >>Definitely looking for customers that have a SIM and don't like, it don't find that it helps them improve the security posture. We also have organizations that are young emerging, have a lot of data, a lot of tech companies that have grown in the last 10, 15 years, or even five years, we have snowflake as a customer. They're booming. They have so much data that going the direction of traditional tools to aggregate the logs, cross correlate them doesn't make any sense with the scale that they need. They need the cloud based approach, SaaS approach that is capable of taking care of the environment. So we both cater to those organizations that we're shifting from on-prem to cloud and need visibility into those two environments and into those cloud natives wanted the cloud don't want to even think of a traditional SIM. >>You mentioned snowflake. We were just at snowflake summit a couple of months ago. I think that was and tremendous company that massive growth, massive growth in data across the board though. So I'm curious, Oprah, if we go back to you, we can dig into some of these data challenges. Obviously data volume and variety is only gonna continue to grow and proliferate and expand data in silos is still a problem. What are some of those main data challenges that hunters helps customers to just eliminate? >>Definitely. So the data challenge starts with getting the right data in the fact that you have so many different products across so many different environments, and you need to try to get them in a, in some location to try to use them for running your queries, your rules, your, your correlation. It's a big prompt. There's no unified standard for anyone. Even if there was, you have a lot of legacy things on premises, as well as your AWS environment, you need to combine all these. You can keep things only OnPrem you can own. Mostly a lot of most organizations are still in hybrid mode. They have they're shifting most of the things to AWS. You still have a lot of things OnPrem that they're gonna shift in the next 3, 4, 5 years. So that hybrid approach is definitely a problem for gathering the data. And when they gather the data, a lot of the times their existing solutions are very cross prohibitive and scale prohibitive from pushing all the data and essential location. >>So they have these data silos. They'll put some of it there. Some of it here, some of them different location, hot storage called storage, long term storage. They don't really, they end up not knowing really where the data is, especially when they need it. The most becomes a huge problem for them. Now with analytics, it's very hard to know upfront what data I'll need, not tomorrow, but maybe in three months to look back and query making these decisions very hard. Changing them later is even harder. Keeping track of all these moving pieces. You know, you have a device, you have some vendor sending you some logs. They changed their APIs. Who's in charge of, of fixing it. Who's in charge of changing your schema. You move from one EDR vendor to the other. How are you making sure that you keep the same level of protection? All these data challenges are very problematic for most customers. The most important thing is to be able to gather as much data as possible, putting in a centralized location and having good monitoring in a continuous flow of, I know what data I'm getting in. I know how much I'm using, and I'm making sure that it's working and flowing. It's going to a central life central place where I can use it at any time that I want. >>We've seen. So sorry. Yes, please. We wanted to add on that. We've seen too much compromise on data that because of prohibitive costs, structure of tools, or because of, in inability to manage the scale teams are compromising or making choices and that paying a price of the latency of being able to then go search. If an incident happened, if you are impacted by something, it all means money and time at the end of the day, when you actually need to answer yourself, am I breached or not? We wanna break out from this compromise. We think that data is something that should not be compromised. It's a commodity today. Everything should be retained, kept and used as appropriately without the team needing to ration what they're gonna use versus what they're not gonna use. >>Correct. That's >>A great point. Go ahead. >>Yeah. And we've seen customers either having entire teams dedicated to just doing this and, or leveraging products and companies that actually build a business around helping you filter the data that you need to put in different data silos, which to me is, is shows how much problem pain and how much this space is broken with what it provides with customers that you have these makeshift solutions to go around the problem instead of facing it head on and saying, okay, let's, let's build something that you're put all your data as much as you want, not have to compromise insecurity. >>You guys both bring up such a great point where data and security is concerned. No business can afford to compromise. Usually compromise is a good thing, but in that case, it's really not companies can't afford that. We know with the, with the threat landscape, the risk, all of the incentives for bad actors that companies need to ensure that they're doing the right things in Aly manner. LA I'm curious, you mentioned the target markets that you're going after. Where are the customer conversations? Is this C conversation from a datasecurity perspective? I would, this is more than the, the CSO. >>It's a CSO conversation, as well as we, we talk on a daily basis with those that lead security operations, head of socks. Those that actually see how the analyst are being overworked are tired, have so many false positives that they need to deal with noise day in, day out, becoming enslaved with the tools that they need to work on and, and tweak. So we have seen that the ones that are most enlightened by a solution like hunters are actually the ones that have to stop reporting to them. They know the daily pain and how much the process is broken. And this is probably one of we, we all talk about, you know, job satisfaction or dissatisfaction, the greatest, the great resignation people are living. This is the real problem in security. And the, so is one of these places that we see this alert, fatigue, people are struggling. It's a stressful work. And if there is anything that we can do to offload the work that is less appealing and have them work on what they sign up for, which is dealing with real threat, solving them, instead of dealing with false positives, this is where we can actually help. >>Can you add a little bit on that? Laal and you mentioned the cybersecurity skills gap, which is massive. We talk about that a lot because it's a huge problem. How is hunters a facilitator of companies that might be experiencing that? >>Absolutely. So we come with approach of, we call it the 80 20 of detection and response. Basically there are about 80% probably. Whoa, it's actually something like 95% of the threats are shared across all organizations in the world. Also 80 to 90% of the environments are similar. People are using similar tools. They're on similar cloud services. We think that everything that goes around detection of threats around those common attacks, scenarios in common attack landscape should come out of the box from a vendor like hunters. So we automate, we write the rules, we cross correlate. We provide those services out of the box. Once you sign to use our solution, your data flows in, and we basically do the processing and the analysis of all the data so that your team can actually focus on the 20% or the, you know, the 5% that are very unique to your organization. >>If you are developing a specific app and you have the knowledge of about the dev SecOps that needs to take place to defend it. Great. Have your team focus on that? If you are a specific actor in a specific space and specific threats that are unique to you, you build your own detections into our tool. But the whole idea that we have, the knowledge, we see attacks across industries and across industries, we have the researchers and the capabilities to be on top of those things. So your team doesn't need to do it on a daily basis because new attacks come almost on a daily basis. Now we read them in the news, we see them. So we do it. So your team doesn't have to, >>And nobody wants to be that next headline where a breach is concerned. I'll close this out here with outcomes. I noticed some big stats on your website. I always gravitate towards that. What are some of the key outcomes that hunters customers are achieving and then specifically AWS customers? >>Absolutely. Well, we already talked a lot about data and being able to ingest it. So we give our customers the predictability, the ability to ingest the data, knowing what the cost is going to be in a very simple cost model. So basically you can ingest everything that you have across all it tools that you have in your environment. And that helped companies reduce up to 75% of the data cost. We we've seen with large customer how much it change when they moved from traditional Sims to using hunters specifically, AWS customers can actually use the AWS credits to buy hunters. If they're interested, just go to AWS marketplace, search for hunters and come to a website. You can use your credits for that. I think we talked also about the security burden. The time spent on writing rules plus correlating incidents. We have seen sometimes a change in, instead of investigating an incident for two days, it is being cut for 20 minutes because we give them the exact story of the entire attack. What are the involved assets? What are the users that are involved, that they can just go see what's happening and then immediately go and remediate it. So big shift in meantime, to detect meantime, to respond. And I'm sure often has a more kind of insights that he's seen with some of our customers around that. >>Yeah. So, so some, some great examples recently there. So there's two things that I've, I've been chatting to customers about. One thing they really get a benefit of is we talked, you talked about the, the, the prong with talent and where that really matters the most is that under the gun mode, we have a service that is, we see it as, as the, the natural progression of the service that we provide called team axon. What team axon does for you is when you are under the gun, when something like log for shell happens, and everybody's looking at you, and time is ticking. Instead of trying to figure out on yourself, team axon will come in, figure out the, the threat will devise a report for all the customers, run queries on your behalf, on your data and give it to you. Within 24 hours, you'll have something to show your CEO or your executive team, your board, even this is where we got impacted or not impacted. >>This is what we did. Here's the mitigation thing. Step that we need to take from world class experts that you might not get access to for every single attack out there that really helps customers kind of feel like they they're, they're safe. There's someone there to help them. There's a big broader there. I call it sometimes the bad signal when we need the most. The other thing is on the day to day, a lot of a lot of solution will, will, will kind of talk about out of the box security. Now, the problem with out of the box security is keeping an up to date. That's what a lot of people miss. You have to think that you installed a year ago, but security doesn't stay put, you need to keep updating it. And you need to keep that updated pretty, pretty frequently to, to stay ahead of the curve. >>If you, if you're behind couple of months on your security updates, you know, what happens, same thing with your, your stock platform or your SIM rule base. What the reason that customers don't update is because if they usually do, then it might blow up the amount of alerts they're getting, cuz they need to tweak them with the approach that we take, that we tested on our customer's data transparently for them and make sure to release them without false positives. We're just allowing them to push the updates transparently directly to their account. They don't need to do anything. And one customer, one of our biggest accounts, they have dozens of subsidiaries and multiple songs. And, and one of the largest eCommerce companies in the world and the person running security. He said, if I had to do what hunters gives me out of the box myself, I have to hire 20 people and put them to work eight for 18 months for what you give me out of the box. So for me, it's a first, that's huge, kinda what we give customers and the kind of challenges that we're able to solve for them. >>Big challenges laal and over, thank you so much for joining us on the cube today. As part of this AWS startup showcase, talking about what hunters does, why the vision and the value in it for customers, we appreciate your time and your insights. Thank you so much for having us, my pleasure for my guests. I'm Lisa Martin. Thank you for watching this episode of the AWS startup showcase. We'll see us in.

>>Hey everyone. Welcome to the cubes presentation of the AWS startup showcase. This is season two, episode four of our ongoing series. That's featuring exciting startups within the AWS ecosystem. This theme, cybersecurity protect and detect against threats. I'm your host. Lisa Martin. I've got two guests here with me. Please. Welcome back to the program. Sam Kam, a COO and co-founder of security scorecard and bar Roth. Charri team lead solutions marketing at confluent guys. It's great to have you on the program talking about cybersecurity. >>Thanks for having us, Lisa, >>Sam, let's go ahead and kick off with you. You've been on the queue before, but give the audience just a little bit of context about security scorecard or SSC as they're gonna hear it referred to. >>Yeah. AB absolutely. Thank you for that. Well, the easiest way to, to put it is when people wanna know about their credit risk, they consult one of the major credit scoring companies. And when companies wanna know about their cybersecurity risk, they turn to security scorecard to get that holistic view of, of, of the security posture. And the way it works is SSC is continuously 24 7 collecting signals from across the entire internet. I entire IPV four space and they're doing it to identify vulnerable and misconfigured digital assets. And we were just looking back over like a three year period. We looked from 2019 to 2022. We, we, we assessed through our techniques over a million and a half organizations and found that over half of them had at least one open critical vulnerability exposed to the internet. What was even more shocking was 20% of those organizations had amassed over a thousand vulnerabilities each. >>So SSC we're in the business of really building solutions for customers. We mine the data from dozens of digital sources and help discover the risks and the flaws that are inherent to their business. And that becomes increasingly important as companies grow and find new sources of risk and new threat vectors that emerge on the internet for themselves and for their vendor and business partner ecosystem. The last thing I'll mention is the platform that we provide. It relies on data collection and processing to be done in an extremely accurate and real time way. That's a key for that's allowed us to scale. And in order to comp, in order for us to accomplish this security scorecard engineering teams, they used a really novel combination of confluent cloud and confluent platform to build a really, really robust data for streaming pipelines and the data streaming pipelines enabled by confluent allow us at security scorecard to collect the data from a lot of various sources for risk analysis. Then they get feer further analyzed and provided to customers as a easy to understand summary of analytics. >>Rob, let's bring you into the conversation, talk about confluent, give the audience that overview and then talk about what you're doing together with SSC. >>Yeah, and I wanted to say Sam did a great job of setting up the context about what confluent is. So, so appreciate that, but a really simple way to think about it. Lisa is confident as a data streaming platform that is pioneering a fundamentally new category of data infrastructure that is at the core of what SSE does. Like Sam said, the key is really collect data accurately at scale and in real time. And that's where our cloud native offering really empowers organizations like SSE to build great customer experiences for their customers. And the other thing we do is we also help organizations build a sophisticated real time backend operations. And so at a high level, that's the best way to think about comfort. >>Got it. But I'll talk about data streaming, how it's being used in cyber security and what the data streaming pipelines enable enabled by confluent allow SSE to do for its customers. >>Yeah, I think Sam can definitely share his thoughts on this, but one of the things I know we are all sort of experiencing is the, is the rise of cyber threats, whether it's online from a business B2B perspective or as consumers just be our data and, and the data that they're generating and the companies that have access to it. So as the, the need to protect the data really grows companies and organizations really need to effectively detect, respond and protect their environments. And the best way to do this is through three ways, scale, speed, and cost. And so going back to the points I brought up earlier with conference, you can really gain real time data ingestion and enable those analytics that Sam talked about previously while optimizing for cost scale. So those are so doing all of this at the same time, as you can imagine, is, is not easy and that's where we Excel. >>And so the entire premise of data streaming is built on the concepts. That data is not static, but constantly moving across your organization. And that's why we call it data streams. And so at its core, we we've sort of built or leveraged that open source foundation of APA sheet Kafka, but we have rearchitected it for the cloud with a totally new cloud native experience. And ultimately for customers like SSE, we have taken a away the need to manage a lot of those operational tasks when it comes to Apache Kafka. The other thing we've done is we've added a ton of proprietary IP, including security features like role based access control. I mean, some prognosis talking about, and that really allows you to securely connect to any data no matter where it resides at scale at speed. And it, >>Can you talk about bar sticking with you, but some of the improvements, and maybe this is a actually question for Sam, some of the improvements that have been achieved on the SSC side as a result of the confluent partnership, things are much faster and you're able to do much more understand, >>Can I, can Sam take it away? I can maybe kick us off and then breath feel, feel free to chime in Lisa. The, the, the, the problem that we're talking about has been for us, it was a longstanding challenge. We're about a nine year old company. We're a high growth startup and data collection has always been in, in our DNA. It's at it's at the core of what we do and getting, getting the insights, the, and analytics that we synthesize from that data into customer's hands as quickly as possible is the, is the name of the game because they're trying to make decisions and we're empowering them to make those decisions faster. We always had challenges in, in the arena because we, well partners like confluent didn't didn't exist when we started scorecard when, when we we're a customer. But we, we, we think of it as a partnership when we found confluent technology and you can hear it from Barth's description. >>Like we, we shared a common vision and they understood some of the pain points that we were experiencing on a very like visceral and intimate level. And for us, that was really exciting, right? Just to have partners that are there saying, we understand your problem. This is exactly the problem that we're solving. We're, we're here to help what the technology has done for us since then is it's not only allowed us to process the data faster and get the analytics to the customer, but it's also allowed us to create more value for customers, which, which I'll talk about in a bit, including new products and new modules that we didn't have the capabilities to deliver before. >>And we'll talk about those new products in a second exciting stuff coming out there from SSC, bro. Talk about the partnership from, from confluence perspective, how has it enabled confluence to actually probably enhance its technology as a result of seeing and learning what SSC is able to do with the technology? >>Yeah, first of all, I, I completely agree with Sam it's, it's more of a partnership because like Sam said, we sort of shared the same vision and that is to really make sure that organizations have access to the data. Like I said earlier, no matter where it resides so that you can scan and identify the, the potential security security threads. I think from, from our perspective, what's really helped us from the perspective of partnering with SSE is just looking at the data volumes that they're working with. So I know a stat that we talked about recently was around scanning billions of records, thousands of ports on a daily basis. And so that's where, like I, like I mentioned earlier, our technology really excels because you can really ingest and amplify the volumes of data that you're processing so that you can scan and, and detect those threats in real time. >>Because I mean, especially the amount of volume, the data volume that's increasing on a year by basis, that aspect in order to be able to respond quickly, that is paramount. And so what's really helped us is just seeing what SSE is doing in terms of scanning the, the web ports or the data systems that are at are at potential risk. Being able to support their use cases, whether it's data sharing between their different teams internally are being able to empower customers, to be able to detect and scan their data systems. And so the learning for us is really seeing how those millions and billions of records get processed. >>Got it sounds like a really synergistic partnership that you guys have had there for the last year or so, Sam, let's go back over to you. You mentioned some new products. I see SSC just released a tax surface intelligence product. That's detecting thousands of vulnerabilities per minute. Talk to us about that, the importance of that, and another release that you're making. >>There are some really exciting products that we have released recently and are releasing at security scorecard. When we think about, when we think about ratings and risk, we think about it not just for our companies or our third parties, but we think about it in a, in a broader sense of an, of an ecosystem, because it's important to have data on third parties, but we also want to have the data on their third parties as well. No, nobody's operating in a vacuum. Everybody's operating in this hyper connected ecosystem and the risk can live not just in the third parties, but they might be storing processing data in a myriad of other technological solutions, which we want to understand, but it's really hard to get that visibility because today the way it's done is companies ask their third parties. Hey, send me a list of your third parties, where my data is stored. >>It's very manual, it's very labor intensive, and it's a trust based exercise that makes it really difficult to validate. What we've done is we've developed a technology called a V D automatic vendor detection. And what a V D does is it goes out and for any company, your own company or another business partner that you work with, it will go detect all of the third party connections that we see that have a live network connection or data connection to an organization. So that's like an awareness and discovery tool because now we can see and pull the veil back and see what the bigger ecosystem and connectivity looks like. Thus allowing the customers to go hold accountable, not just the third parties, but their fourth parties, fifth parties really end parties. And they, and they can only do that by using scorecard. The attack surface intelligence tool is really exciting for us because well, be before security scorecard people thought what we were doing was fairly, I impossible. >>It was really hard to get instant visibility on any company and any business partner. And at the same time, it was of critical importance to have that instant visibility into the risk because companies are trying to make faster decisions and they need the risk data to steer those decisions. So when I think about, when I think about that problem in, in managing sort of this evolving landscape, what it requires is it requires insightful and actionable, real time security data. And that relies on a couple things, talent and tech on the talent side, it starts with people. We have an amazing R and D team. We invest heavily. It's the heartbeat of what we do. That team really excels in areas of data collection analysis and scaling large data sets. And then we know on the tech side, well, we figured out some breakthrough techniques and it also requires partners like confluent to help with the real time streaming. >>What we realized was those capabilities are very desired in the market. And we created a new product from it called the tech surface intelligence. A tech surface intelligence focuses less on the rating. There's, there's a persona on users that really value the rating. It's easy to understand. It's a bridge language between technical and non-technical stakeholders. That's on one end of the spectrum on the other end of the spectrum. There's customers and users, very technical customers and users that may not have as much interest in a layman's rating, but really want a deep dive into the strong threat Intel data and capabilities and insights that we're producing. So we produced ASI, which stands for attack surface intelligence that allows customers to look at the surface area of attack all of the digital assets for any organization and see all of the threats, vulnerabilities, bad actors, including sometimes discoveries of zero day vulnerabilities that are, that are out in the wild and being exploited by bad guys. So we have a really strong pulse on what's happening on the internet, good and bad. And we created that product to help service a market that was interested in, in going deep into the data. >>So it's >>So critical. Go >>Ahead to jump in there real quick, because I think the points that Sam brought up, we had a great, great discussion recently while we were building on the case study that I think brings this to life, going back to the AVD product that Sam talked about and, and Sam can probably do a better job of walking through the story, but the way I understand it, one of security scorecards customers approached them and told them that they had an issue to resolve and what they ended up. So this customer was using an AVD product at the time. And so they said that, Hey, the car SSE, they said, Hey, your product shows that we used, you were using HubSpot, but we stopped using that age server. And so I think when SSE investigated, they did find a very recent HubSpot ping being used by the marketing team in this instance. And as someone who comes from that marketing background, I can raise my hand and said, I've been there, done that. So, so yeah, I mean, Sam can probably share his thoughts on this, but that's, I think the great story that sort of brings this all to life in terms of how actually customers go about using SSCs products. >>And Sam, go ahead on that. It sounds like, and one of the things I'm hearing that is a benefit is reduction in shadow. It, I'm sure that happens so frequently with your customers about Mar like a great example that you gave of, of the, the it folks saying we don't use HubSpot, have it in years marketing initiates an instance. Talk about that as some of the benefits in it for customers reducing shadow it, there's gotta be many more benefits from a security perspective. >>Yeah, the, there's a, there's a big challenge today because the market moved to the cloud and that makes it really easy for anybody in an organization to go sign, sign up, put in a credit card, or get a free trial to, to any product. And that product can very easily connect into the corporate system and access the data. And because of the nature of how cloud products work and how easy they are to sign up a byproduct of that is they sort of circumvent a traditional risk assessment process that, that organizations go through and organizations invest a, a lot of money, right? So there's a lot of time and money and energy that are invested in having good procurement risk management life cycles, and making sure that contracts are buttoned up. So on one side you have companies investing loads of energy. And then on the other side, any employee can circumvent that process by just going and with a few clicks, signing up and purchasing a product. >>And that's, and, and, and then that causes a, a disparity and Delta between what the technology and security team's understanding is of the landscape and, and what reality is. And we're trying to close that gap, right? We wanna close and reduce any windows of time or opportunity where a hacker can go discover some misconfigured cloud asset that somebody signed up for and maybe forgot to turn off. I mean, it's a lot of it is just human error and it, and it happens the example that Barra gave, and this is why understanding the third parties are so important. A customer contacted us and said, Hey, you're a V D detection product has an error. It's showing we're using a product. I think it was HubSpot, but we stopped using that. Right. And we don't understand why you're still showing it. It has to be a false positive. >>So we investigated and found that there was a very recent live HubSpot connection, ping being made. Sure enough. When we went back to the customer said, we're very confident the data's accurate. They looked into it. They found that the marketing team had started experimenting with another instance of HubSpot on the side. They were putting in real customer data in that instance. And it, it, you know, it triggered a security assessment. So we, we see all sorts of permutations of it, large multinational companies spin up a satellite office and a contractor setting up the network equipment. They misconfigure it. And inadvertently leave an administrator portal to the Cisco router exposed on the public internet. And they forget to turn off the administrative default credentials. So if a hacker stumbles on that, they can ha they have direct access to the network. We're trying to catch those things and surface them to the client before the hackers find it. >>So we're giving 'em this, this hacker's eye view. And without the continuous data analysis, without the stream processing, the customer wouldn't have known about those risks. But if you can automatically know about the risks as they happen, what that does is that prevents a million shoulder taps because the customer doesn't have to go tap on the marketing team's shoulder and go tap on employees and manually interview them. They have the data already, and that can be for their company. That can be for any company they're doing business with where they're storing and processing data. That's a huge time savings and a huge risk reduction, >>Huge risk reduction. Like you're taking blinders off that they didn't even know were there. And I can imagine Sam tune in the last couple of years, as SAS skyrocketed the use of collaboration tools, just to keep the lights on for organizations to be able to communicate. There's probably a lot of opportunity in your customer base and perspective customer base to engage with you and get that really full 360 degree view of their entire organization. Third parties, fourth parties, et cetera. >>Absolutely. Absolutely. CU customers are more engaged than they've ever been because that challenge of the market moving to the cloud, it hasn't stopped. We've been talking about it for a long time, but there's still a lot of big organizations that are starting to dip their toe in the pool and starting to cut over from what was traditionally an in-house data center in the basement of the headquarters. They're, they're moving over to the cloud. And then on, on top of that cloud providers like Azure, AWS, especially make it so easy for any company to go sign up, get access, build a product, and launch that product to the market. We see more and more organizations sitting on AWS, launching products and software. The, the barrier to entry is very, very low. And the value in those products is very, very high. So that's drawing the attention of organizations to go sign up and engage. >>The challenge then becomes, we don't know who has control over this data, right? We don't have know who has control and visibility of our data. We're, we're bringing that to surface and for vendors themselves like, especially companies that sit in AWS, what we see them doing. And I think Lisa, this is what you're alluding to. When companies engage in their own scorecard, there's a bit of a social aspect to it. When they look good in our platform, other companies are following them, right? So now all of the sudden they can make one motion to go look good, make their scorecard buttoned up. And everybody who's looking at them now sees that they're doing the right things. We actually have a lot of vendors who are customers, they're winning more competitive bakeoffs and deals because they're proving to their clients faster that they can trust them to store the data. >>So it's a bit of, you know, we're in a, two-sided kind of market. You have folks that are assessing other folks. That's fun to look at others and see how they're doing and hold them accountable. But if you're on the receiving end, that can be stressful. So what we've done is we've taken the, that situation and we've turned it into a really positive and productive environment where companies, whether they're looking at someone else or they're looking at themselves to prove to their clients, to prove to the board, it turns into a very productive experience for them >>One. Oh >>Yeah. That validation. Go ahead, bro. >>Really. I was gonna ask Sam his thoughts on one particular aspect. So in terms of the industry, Sam, that you're seeing sort of really moving to the cloud and like this need for secure data, making sure that the data can be trusted. Are there specific like verticals that are doing that better than the others? Or do you see that across the board? >>I think some industries have it easier and some industries have it harder, definitely in industries that are, I think, health, healthcare, financial services, a absolutely. We see heavier activity there on, on both sides, right? They they're, they're certainly becoming more and more proactive in their investments, but the attacks are not stopping against those, especially healthcare because the data is so valuable and historically healthcare was under, was an underinvested space, right. Hospitals. And we're always strapped for it folks. Now, now they're starting to wake up and pay very close attention and make heavier investments. >>That's pretty interesting. >>Tremendous opportunity there guys. I'm sorry. We are out of time, but this is such an interesting conversation. You see, we keep going, wanna ask you both where can, can prospective interested customers go to learn more on the SSC side, on the confluence side, through the AWS marketplace? >>I let some go first. >>Sure. Oh, thank thank, thank you. Thank you for on the security scorecard side. Well look, security scorecard is with the help of Colu is, has made it possible to instantly rate the security posture of any company in the world. We have 12 million organizations rated today and, and that, and that's going up every day. We invite any company in the world to try security scorecard for free and experience how, how easy it is to get your rating and see the security rating of, of any company and any, any company can claim their score. There's no, there's no charge. They can go to security, scorecard.com and we have a special, actually a special URL security scorecard.com/free-account/aws marketplace. And even better if someone's already on AWS, you know, you can view our security posture with the AWS marketplace, vendor insights, plugin to quickly and securely procure your products. >>Awesome. Guys, this has been fantastic information. I'm sorry, bro. Did you wanna add one more thing? Yeah. >>I just wanted to give quick call out leads. So anyone who wants to learn more about data streaming can go to www confluent IO. There's also an upcoming event, which has a separate URL. That's coming up in October where you can learn all about data streaming and that URL is current event.io. So those are the two URLs I just wanted to quickly call out. >>Awesome guys. Thanks again so much for partnering with the cube on season two, episode four of our AWS startup showcase. We appreciate your insights and your time. And for those of you watching, thank you so much. Keep it right here for more action on the, for my guests. I am Lisa Martin. We'll see you next time.

(upbeat music) >> Hey everyone. Welcome to theCube's presentation of the AWS Startup Showcase. This is season two, episode four of our ongoing series featuring exciting startups in the AWS ecosystem. This theme is cyber security, detecting and protecting against threats. I'm your host, Lisa Martin and I'm pleased to be joined by Raghu Nadakumara the senior director of solutions marketing at Illumio. We're going to be talking about all things, cybersecurity, Raghu. it's great to have you on the program >> Lisa, it's fantastic to be here and the lovely to have the opportunity. Thank you >> Absolutely. So, so much changing in the threat landscape. We're seeing threat actors are booming, new threats customers having to solve really hard security problems across their organization. On-prem in the cloud, hybrid multi-cloud, et cetera. Talk to me about some of the ways in which Illumio is helping customers to address those massive challenges. >> Sure. I think like it's a sort of to pair off what you said to begin with. You said so much has changed, but equally and Kim Jetta made this point last week in her keynote at Black Hat and Chris Krebs former director of CISA also kind of reiterated this, so much has changed yet so much hasn't changed. And really from sort of Illumio's perspective the way we look at this is that as we are moving to a sort of a world of ever increasing connectivity I kind of almost pair off digital transformation which pretty much every organization talks about. They've got a digital transformation program. I really pair that off with what does that mean? It really means hyper connectivity because you've got your data center connecting into workloads, running in the cloud with users and user devices everywhere with a plethora of other connected devices. So we've got this massive hyper connected web. Well, what does that lead to? It leads to a massively increasing mushrooming attack surface. So from a threat actor perspective, just the the size of the opportunity is so much larger these days. But the problem then from a from a defender's perspective is that how do you even understand your, this complex very hybrid attack surface? So what we lack is the ability to get that consistent visibility of our actual exposure across the board, but, and then the ability to then deploy a consistent security control set across that estate to be able to manage that attack service and reduce that exposure risk. And these two problems, the challenge of consistent visibility and the challenge of consistent security from an Illumio perspective, we believe we solve both of those with our zero trust segmentation platform. So we are really looking at helping organizations helping our customers be resilient to the threats of today and the threats of tomorrow by giving them that consistent visibility and that consistent security through zero trust segmentation. >> Let's unpack zero trust segmentation. You know, when we look at some of the stats on ransom where it's been a while that it's a matter of when, not if for organizations so getting that visibility and consistent security policies across the estate, as you say is critical for businesses in every organization. How does zero trust segmentation, first of all define it and then tell us how that helps. >> Oh, happily. It's kind of one my favorite subjects to talk about. Right. So let start with zero trust segmentation and kind of, sort of to put it into a context that's probably more easy to understand, right? Is that we see sort of zero trust segmentation as being founded on two pillars, right? The first is an assumed breach mindset and I'll come onto what we mean by that in a second. And the second paired with that and what we see is kind of the natural progression from that is then the use of least privileged policies to go and control and protect your estate. So what does assume breach mean? Well, assume breach is really that approach that says work on the assumption that bad event that malicious actor, that anomalous action that unexpected behavior, and that could be intentional and the result of a malicious action or it could be completely unintentional. Think of that sort of someone, a misconfiguration in an application, for example, right? All of these things are essentially unexpected anomalous event. So start from that assumption that that's either happened or it's going to happen at some point, right? So when you make that assumption, right, and that assumption that that is happening on your internal network. So remember right. Assume that that thing is already happening on your internal network, not it's on outside of the perimeter and it's got to still find its way in. No, it's really about assuming that that initial sort of thing to get onto the network and some anomalous event has already happened. If you started from that premise then how would you design your security controls? Well, the natural reaction to that is, well if that's going to happen what I need to ensure is that the impact of that is as limited as possible is as restricted as possible. So how do I ensure that that is as limited as possible? Well, it's by ensuring that any access into the rest of my environment, the rest of the infrastructure and that could be that hybrid infrastructure, private cloud, public cloud, et cetera is built on a least privileged access model. And that way I can ensure that even if I have a compromise in one part of my environment or potentially there could be compromises in different parts of my environment that they're not going to impact the rest of the whole. So I'm containing the impact of that. And as a result I'm protecting the rest of the infrastructure and able to maintain my resilience for longer. So that's how zero trust segmentation, well, that's what zero trust segmentation is and how it delivers better security for an organization. >> So preventing that lateral spread is really critical especially as we've seen in the last couple of years this acceleration of cloud adoption, cloud migration for customers that are in transit, if you will, CTS why is it so fundamental? >> Well, I think you expressed it brilliantly, right? That if you look at any sort of malicious attack, right? Whether it's ransomware, whether it's an advanced attacker like APT style attack over the last sort of decade, right? A common part, a common tactic, those attackers used in order to proliferate and in order to move to either spread that attack as far and wide as possible in the case of ransomware or in the case of a very targeted attack to go and find that trophy target. One of the key tactics they leverage is lateral movement. So from a defender's perspective if you are able to better detect and ideally better prevent upfront that lateral movement and limit you are, you are defending yourself. You are proactively defending yourself from this threat. So what does that mean then from the perspective of organizations that are moving into cloud? So organizations that are say on that journey to transition into AWS, right? Whether from a right, I'm going all in an AWS and ultimately leaving my private data center behind or sort of more likely where my applications now in this hybrid deployment model where I have some on-prem some in the cloud. So there it's even more important because we know that things that are deployed in the cloud can very easily sort of get exposed to the internet. Right? We've seen that with a number of sort of different customers of cloud where a misconfigured security group suddenly gives access to all resources from the internet, right? Or gives access on high risk ports that you didn't want to have that you didn't want to be able to access. So here, zero trust segmentation is so important because if you come back to the fundamentals of it, it's around consistent visibility and consistent security policy. So what do we provide? Well, from an Illumio perspective and through our zero trust segmentation platform we ensure that as your application, as your key resources, as they transition from your private data center into the cloud, you can have exactly the same visibility and exactly the same granularity of visibility over those interactions between your resources as they move into the cloud. And the most important thing here is that it's not in cloud. We realize it's not just about adopting compute. It's not just infrastructure as a service organizations are now adopting the the more cloud native services whether that's managed databases or containers or serverless, et cetera, right. But all of these make up part of that new application and all of those need be included in that visibility, right? So visibility, isn't just about what your computer's doing where you've got this OS that you can manage but it's really about any component that is interacting as part of your organization as part of your applications. So we provide visibility across that and as it moves so that, that sort of, that granularity of visibility the ability to see those dependencies between applications we provide that consistently. And then naturally we then allow you to con consistently apply security policy as this application moves. So as you transition from on-prem where you have controls where you have your lateral movement controls your segmentation controls, and as you move resources into the cloud we allow you to maintain that security posture as you move into cloud, but not just that doesn't just stop there. So we spoke at the top about how least privileged is fundamental to zero trust from a policy perspective what we give you the ability to do give our customers the ability to do as they move into AWS is compare what they have configured on their security groups. So they way they think they've got the right security posture, we compare that to what the actual usage around those resources is. And we provide them recommendations to better secure those security groups. So essentially always tending them towards a more secure con configuration, such that they can maintain that least privileged access over the, around their critical resources. So this is the way our technology helps our customers move and migrate safely and securely from on-prem into AWS. >> That's a great description, very thorough in how you're talking about the benefits to organizations. You know, as we think about cloud adoption migration, cybersecurity these are clearly C-suite conversations. Are you seeing things like zero trust segmentation rise up to the C-suite and maybe even beyond to the board? Is this from a security perspective, a board level issue? >> Oh, absolutely. And, and Chris Krebs, former director of CISA last week set security must absolutely be a board level topic. It's not something that needs to be sort of in the weeds of IT or just sort of under the purview of what the chief security is doing. It needs to a board level issue. And what we see is while sort of talking about let's say zero trust segmentation or zero trust is very much a security function. What it typically ladders up to at the boardroom level is tying it into operational resilience, right? Because I think organizations now it's not just about the ability, given that sort of attacks are proliferating. And particularly the threat around ransomware is so high that the use of ransomware, not just as a way to steal data and extract money, but also ransomware as essentially a way to disrupt operations. And that is now what the concern is at that board level. Is that how is this attack going to impact me from a from a productivity perspective from an availability perspective, and depending on the type of organization, if it's, for example a financial organization there their worry is around their reputation because ultimately organizations are unable to trust that financial organization. We very quickly see that we have sort of that run on the bank, where customers, counterparties et cetera, quickly want to take their business elsewhere. If it's a manufacturing or healthcare provider, their concern is can we deliver our critical services? For example, healthcare can we deliver patient services? Manufacturing, can we continue to produce whatever it is we manufacture, even in the case of being under attack? So at the board level they're thinking about it from the perspective of resilience and operational resilience, and that then translates into cyber resilience when it comes to talking about where does zero trust segmentation fit in? Zero trust segmentation enables cyber resilience which ultimately enables operational resilience. So this is how we see it laddering up to boardroom issues. >> Got it. And of course, you know when you were talking about brand reputation, brand damage you think nobody wants to be the next headline where a breach is occurring. We've seen too many of those and we probably will see many more. So Raghu, when you're in customer conversations what are say the top three differentiators that you share with customers versus like CSPM tools what are those key core Illumio differentiators? >> Yeah. So like sort of CSPM tools, right? They're very focusing on assessing posture and sort of reporting on compliance in comparison to a baseline. So for example, it's okay here is what I think the security configuration should be. And here is how I'm actually configured in AWS. Here is the diff and here is where I'm out of compliance, right? That that's typically what, what CSPM products do, right? And there is a very important place for them in any organization's tool set. Now, what they don't do and where we provide the differentiation is that they're not set up to sort of monitor around lateral movement, right? They're not about providing you with that view about how your resources are interacting each other. They're not about providing guidance as to whether a security reconfiguration could be enhanced and could be tightened up. They also don't give you the view particularly around is this even relevant, right? And that that's really where we come in because the the visibility allows you to understand how resources are interacting with each other. That then allows you to determine whether those interactions are required or not. That then allows you to define a least privileged policy that controls access between these resources. But it also kind of as this sort of the feedback loop goes on is to ensure that least privileged policy is always tending towards what you actually need, right? So it's from what I think I need to what you actually need based on, based on usage. So this is how we differentiate what we do from what a CSPM type of technology does, right? We're always about providing visibility and maintaining least privileged access between your resources >> How many different security tools are you seeing that organizations have in place today? Those prospects that are coming to Illumio saying we've got challenges, we understand the threat landscape. The malicious actors are very incentivized, but what are the security tools in place and is Illumio able to replace, like, reduce that number replace some of those tools. So that simplification happens in this growingly complex environment. >> Yeah, I think that's a really good question. And I think that the answer to that is really, actually not so much about not necessarily about reducing though, of course, right. Organizations always, if they can reduce tools and replace one tool that does one thing with a tool that does multiple things, it's, it's always a it's always a benefit, but the the way we see it is that what is the value that we provide that complements existing tooling that an organization already has, right. Because what we think is important is that any technology that you bring in, shouldn't be just sit on its own island where it's value is kind of isolated from the value you are getting from everything else, right. It should be part of it should be able to be part of a sort of integrated ecosystem of complimentary technologies, right. And we believe that what we do firmly fits in to that type of technology ecosystem, right. So we in, so for example, to to give you examples, right, we enhance your asset discovery piece by providing a, the visibility that allows you to get the understanding of all your interactions. Why is that important? Because you can use that data to ensure that what you think is labeled or tagged in a particular way is in fact, that asset, right. And we benefit from that because we benefit from the asset information to allow us to build security policy that map those dependencies. We provide value to your detection and response capabilities, because we have that visibility around lateral movement. We are able to be reactive in terms of containing an attack. We can be used to proactively limit sort of pathways such that let's say things like common ransomware can't leverage things like open RDP and open SMB ports to spread. We can go and inform things like service maps. So if your organization is sort of heavily invested in like service mapping and feeding that back into sort of your IT tool sets. So ITSM tool sets, et cetera, right. We can provide data into that to enhance that particular experience. So there is lots of value beyond sort of what our own product value proposition is that we bring into your existing technology ecosystem. Which is why we think we kind of add value into any deployment over and beyond just sort of the things that we do around visibility and consistent security. >> Yeah. What you were just describing. So well with the first thought coming to my mind was value-add. There's a lot of synergy there. Synergies between other technologies. You mentioned that complimentary nature, that seems like a huge value impact for organizations across any industry. Last question from a go to market perspective where can prospects go to learn more? This is available in the AWS marketplace, but talk to us about where they can go to learn more. >> Yeah, sure, so you can, so if you're an AWS customer, right, you can purchase Illumio straight from the AWS marketplace. Just go and find it under sort of security products in, I think it's infrastructure software. So you can go and find that. You can obviously reach out to your AWS account team if you want sort of further information around Illumio and how to secure that through AWS. And of course you can come along to illumio.com where we have a whole raft of information about what we do, how we do it, the benefits that we provide to our customers and how it ladders up to some of the key sort of boardroom issues, right. Around whether it's around transformation or resilience or ransomware containment. So come along to our website and and find out all those things. And we're here to help >> Awesome Raghu. What a great conversation around such an important topic, cybersecurity, detecting and protecting against threats that we know is is an evolving landscape. We appreciate all of your insights. Great explanations into what Illumio is doing there. How you're helping organizations and where they can go to find more. Thank you so much for joining me today. >> It's been absolute, absolute pleasure, Lisa. Thank you very much for having me. >> All right. For Raghu Nadkumara. I'm Lisa Martin. We want to thank you for watching this episode of the AWS Startup Showcase. We'll see you soon. (soft music)

>>Hello, and welcome to the cubes presentation of the AWS startup showcase. This is season two, episode four, the ongoing series covering exciting startups from the AWS ecosystem to talk about cyber security. I'm your host, John feer. And today we're excited to join by a Mediatel Walker, CEO of Quin security and sub IER, vice president of product management of sequence security gentlemen, thanks for joining us today on this showcase. >>Thank you, John PRAs. >>So the title of this session is continuous API protection life cycle to discover, detect, and defend security. APIs are part of it. They're hardened, everyone's using them, but they're they're target for malicious behavior. This is the focus of this segment. You guys are in the leading edge of this. What are the biggest challenges for organizations right now in assessing their security risks? Because you're seeing APIs all over the place in the news, just even this week, Twitter had a whistleblower come out from the security group, talking about their security plans, misleading the FTC on the bots and some of the malicious behavior inside the API interface of Twitter. This is really a mainstream Washington post is reporting on it. New York times, all the global outlets are talking about this story. This is the risk. I mean, yeah, this is what you guys do protect against this. >>Yeah, this is absolutely top of mind for a lot of security folks today. So obviously in the media and the type of attack that that is being discussed with this whistleblower coming out is called reputation bombing. This is not new. This has been going on since I would say at least eight to 10 years where the, the bad actors are using bots or automation and ultimately using APIs on these large social media platforms, whether it's Facebook, whether it's Twitter or some other social media platform and messing with the reputation system of those large platforms. And what I mean by that is they will do fake likes, fake commenting, fake retweeting in the case of Twitter. And what that means is that things that are, should not be very popular, all of a sudden become popular. That that way they're able to influence things like elections, shopping habits, personnel. >>We, we work with similar profile companies and we see this all the time. We, we mostly work on some of the secondary platforms like dating and other sort of social media platforms around music sharing and things like video sharing. And we see this all the time. These, these bots are bad. Actors are using bots, but ultimately it's an API problem. It's not just a bot problem. And that's what we've been trying to sort of preach to the world, which is your bot problem is subset of your API security challenges that you deal as an organization. >>You know, IMIA, we talked about this in the past on a previous conversation, but this really is front and center mainstream for the whole world to see around the challenges. All companies face, every CSO, every CIO, every board member organizations out there looking at this security posture that spans not just information technology, but physical and now social engineering. You have all kinds of new payloads of malicious behavior that are being compromised through, through things like APIs. This is not just about CSO, chief information security officer. This is chief security officer issues. What's your reaction >>Very much so I think the, this is a security problem, but it's also a reputation problem. In some cases, it's a data governance problem. We work with several companies which have very restrictive data governance and data regulations or data residency regulations there to conform to those regulations. And they have to look at that. It's not just a CSO problem anymore. In case of the, the news of the day to day, this is a platform problem. This goes all the way to the, that time CTO of Twitter. And now the CEO of Twitter, who was in charge of dealing with these problems. We see as just to give you an example, we, we work, we work with a similar sort of social media platform that allows Oop based login to their platform that is using tokens. You can sort of sign in with Facebook, sign in with Twitter, sign in with Google. These are API keys that are generated and trusted by these social media platforms. When we saw that Facebook leaked about 50 million of these login credentials or API keys, this was about three, four years ago. I wrote a blog about it. We saw a huge spike in those API keys being used to log to other social media platforms. So although one social platform might be taking care of its, you know, API or what problem, if something else gets reached somewhere else, it has a cascading impact on a variety of platforms. >>You know, that's a really interesting dynamic. And if you think about just the token piece that you mentioned, that's kind of under the coverage, that's a technology challenge, but also you get in the business logic. So let's go back and, and unpack that, okay, they discontinue the tokens. Now they're being reused here. In the case of Twitter, I was talking to an executive here in Silicon valley and they said, yeah, it's a cautionary tale, for sure. Although Twitter's a unique situation, but they abstract out the business value and say, Hey, they had an M and a deal on the table. And so if someone wants to unwind that deal, all I gotta say is, Hey, there's a bot problem. And now you have essentially new kinds of risk in the business have nothing to do with some sign the technology, okay. They got a security breach, but here with Twitter, you have an, an, an M and a deal, an acquisition that's being contested because of the, the APIs. So, so if you're in business, you gotta think to yourself, what am I risking with my API? So every organization should be assessing their security risks, tied to their APIs. This is a huge awakening for them. Where should they start? And that's the, that's the core question. Okay. You got my attention risks with the API. What do I do? >>So when I talked to you in my previous interview, the start is basically knowing what to, in most cases, you see these that are hitting the wire much. Every now there is a major in cases you'll find these APIs are targeted, that are not poorly protected. They're absolutely just not protected at all, which means the security team or any sort of team that is responsible for protecting these APIs are just completely unaware of these APIs being there in the first place. And this is where we talk about the shadow it or shadow API problem. Large enterprises have teams that are geo distributed, and this problem is escalated after the pandemic even more because now you have teams that are completely distributed. They do M and a. So they acquire new companies and have no visibility into their API or security practices. And so there are a lot of driving factors why these APIs are just not protected and, and just unknown even more to the security team. So the first step has to be discover your API attack surface, and then prioritize which APIs you wanna target in terms of runtime protection. >>Yeah. I wanna dig into that API kind of attack surface area management, runtime monitoring capability in a second, but so I wanna get you in here too, because we're talking about APIs, we're talking about attacks. What does an API attack look like? >>Yeah, that's a very good question, John, there are really two different forms of attacks of APIs, one type of attack, exploits, APIs that have known vulnerabilities or some form of vulnerabilities. For instance, APIs that may use a weak form of authentication or are really built with no authentication at all, or have some sort of vulnerability that makes them very good targets for an attacker to target. And the second form of attack is a more subtle one. It's called business logic abuse. It's, it's utilizing APIs in completely legitimate manner manners, but exploiting those APIs to exfiltrate information or key sensitive information that was probably not thought through by the developer or the designers or those APIs. And really when we do API protection, we really need to be able to handle both of those scenarios, protect against abuse of APIs, such as broken authentication, or broken object level authorization APIs with that problem, as well as protecting APIs from business logic abuse. And that's really how we, you know, differentiate against other vendors in this >>Market. So just what are the, those key differentiated ways to identify the, in the malicious intents with APIs? Can you, can you just summarize that real quick, the three ways? >>Sure. Yeah, absolutely. There are three key ways that we differentiate against our competition. One is in the, we have built out a, in the ability to actually detect such traffic. We have built out a very sophisticated threat intelligence network built over the entire lifetime of the company where we have very well curated information about malicious infrastructures, malicious operators around the world, including not just it address ranges, but also which infrastructures do they operate on and stuff like that, which actually helps a lot in, in many environments in especially B2C environments, that alone accounts for a lot of efficacy for us in detecting our weed out bad traffic. The second aspect is in analyzing the request that are coming in the API traffic that is coming in and from the request itself, being able to tell if there is credential abuse going on or credential stuffing going on or known patterns that the traffic is exhibiting, that looks like it is clearly trying to attack the attack, the APM. >>And the third one is, is really more sophisticated as they go farther and farther. It gets more sophisticated where sequence actually has a lot of machine learning models built in which actually profile the traffic that is coming in and separate. So the legitimate or learns the legitimate traffic from the anomalous or suspicious traffic. So as the traffic, as the API requests are coming in, it automatically can tell that this traffic does not look like legitimate traffic does not look like the traffic that this API typically gets and automatically uses that to figure out, okay, where is this traffic coming from? And automatically takes action to prevent that attack? >>You know, it's interesting APIs have been part of the goodness of cloud and cloud scale. And it reminds me of the old Andy Grove quote, founder of, in one of the founders of Intel, you know, let chaos, let, let the chaos happen, then reign it in it's APIs. You know, a lot of people have been creating them and you've got a lot of different stakeholders involved in creating them. And so now securing them and now manage them. So a lot of creation now you're starting to secure them and now you gotta manage 'em. This all is now big focus. As you pointed out, what are some of the dynamics that customers who have to deal with on the product side and, and organization, let, let chaos rain, and then rain in the chaos, as, as the saying goes, what, what do companies do? >>Yeah. Typically companies start off with like, like a mayor talked about earlier. Discovery is really the key thing to start with, like figuring out what your API attack surfaces and really getting your arms around that problem. And typically we are finding customers start that off from the security organization, the CSO organization to really go after that problem. And in some cases, in some customers, we even find like dedicated centers of excellence that are created for API security, which go after that problem to be able to get their arms around the whole API attack surface and the API protection problem statement. So that's where usually that problem starts to get addressed. >>I mean, organizations and your customers have to stop the attacks. A lot of different techniques, you know, run time. You mentioned that earlier, the surface area monitoring, what's the choice. What's the, where are, where are, where is everybody? Is everyone in the, in the boiling water, like the frog and boiling water or they do, they know it's happening? Like what did they do? What's their opportunity to get in >>Position? Yeah. So I, I think let's take a step back a little bit, right? What has happened is if you draw the cloud security market, if you will, right. Which is the journey to the cloud, the security of these applications or APIs at a container level, in terms of vulnerabilities and, and other things that market grew with the journey to the cloud, pretty much locked in lockstep. What has happened in the API side is the API space has kind of lacked behind the growth and explosion in the API space. So what that means is APIs are getting published way faster than the security teams are able to sort of control and secure them. APIs are getting published in environments that the security completely unaware of. We talked about in the past about the parameter, the parameter, as we know, it doesn't exist anymore. It used to be the case that you hit a CDN, you terminate your SSL, you stop your layer three and four DDoS. >>And then you go into the application and do the business logic. That parameter is just gone because it's now could be living in multi-cloud environment. It could be living in the on-prem environment, which is PubNet is friendly. And so security teams that are used to protecting apps, using a perimeter defense plus changes, it's gone. You need to figure out where your perimeter is. And therefore we sort of recommend an approach, which is have a uniform view across all your APIs, wherever they could be distributed and have a single point of control across those with a solution like sequence. And there are others also in this space, which is giving you that uniform view, which is first giving you that, you know, outside and looking view of what APIs to protect. And then let's, you sort of take the journey of securing the API life cycle. >>So I would say that every company now hear me out on this indulges me for a second. Every company in the world will be non perimeter based, except for maybe 5% because of maybe unique reason, proprietary lockdown, information, whatever. But for most, most companies, everyone will be in the cloud or some cloud native, non perimeter based security posture. So the question is, how does your platform fit into that trajectory? And specifically, why are you guys in the position in your mind to help customers solve this API problem? Because again, APIs have been the greatest thing about the cloud, right? Yeah. So the goodness is there because of APS. Now you gotta reign it in reign in the chaos. Yeah. What, what about your platform share? What is it, why is it win? Why should customers care about this? >>Absolutely. So if you think about it, you're right, the parameter doesn't exist. People have APIs deployed in multiple environments, multicloud hybrid, you name it sequence is uniquely positioned in a way that we can work with your environment. No matter what that environment is. We're the only player in this space that can protect your APIs purely as a SA solution or purely as an on-prem deployment. And that could be a SaaS platform. It doesn't need to be RackN, but we also support that and we could be a hybrid deployment. We have some deployments which are on your prem and the rest of this solution is in our SA. If you think about it, customers have secured their APIs with sequence with 15 minutes, you know, going live from zero to life and getting that protection instantaneously. We have customers that are processing a billion API calls per day, across variety of different cloud environments in sort of six different brands. And so that scale, that flexibility of where we can plug into your infrastructure or be completely off of your infrastructure is something unique to sequence that we offer that nobody else is offering >>Today. Okay. So I'll be, I'll be a naysayer. Yeah, look, it, we are perfectly coded APIs. We are the best in the business. We're locked down. Our APIs are as tight as a drum. Why do I need you? >>So that goes back to who's answer. Of course, >>Everyone's say that that's, that's great, but that's my argument. >>There are two types of API attacks. One is a tactic problem, which is exploiting a vulnerability in an API, right? So what you're saying is my APIs are secure. It does not have any vulnerability I've taken care of all vulnerabilities. The second type of attack that targets APIs is the business logic. Use this stuff in the news this week, which is the whistleblower problem, which is, if you think APIs that Twitter is publishing for users are perfectly secure. They are taking care of all the vulnerabilities and patching them when they find new ones. But it's the business logic of, you know, REWE liking or commenting that the bots are targeting, which they have no against. Right. And then none of the other social networks too. Yeah. So there are many examples. Uber wrote a program to impersonate users in different geo locations to find lifts, pricing, and driver information and passenger information, completely legitimate use of APIs for illegitimate, illegitimate purpose using bots. So you don't need bots by the way, don't, don't make this about bot versus not. Yeah. You can use APIs sort of for the, the purpose that they're not designed for sort of exploiting their business logic, either using a human interacting, a human farm, interacting with those APIs or a bot form targeting those APIs, I think. But that's the problem when you have, even when you've secured all your problem, all your APIs, you still have to worry about these of challenges. >>I think that's the big one. I think the business logic one, certainly the Twitter highlights that the Uber example is a good one. That is basically almost the, the backlash of having a simplistic API, which people design to. Right. Yeah. You know, as you point out, Twitter is very simple API, hardened, very strong security, but they're using it to maliciously manipulate what's inside. So in a way that perimeter's dead too. Right. So how do you stop that business logic? What's the, what's the solution what's the customer do about that? Because their goal is to create simple, scalable APIs. >>Yeah. I'll, I'll give you a little bit, and then I think Subaru should maybe go into a little bit of the depth of the problem, but what I think that the answer lies in what Subaru spoke earlier, which is our ML. AI is, is good at profiling plus split between the API users, are these legitimate users, humans versus bots. That's the first split we do. The split second split we do is even when these, these are classified users as bots, we will say there are some good bots that are necessary for the business and bad bots. So we are able to split this across three types of users, legitimate humans, good bots and bad bots. And just to give you an example of good bots is there are in the financial work, there are aggregators that are scraping your data and aggregating for end users to consume, right? Your, your, and other type of financial aggregators FinTech companies like MX. These are good bots and you wanna allow them to, you know, use your APIs, whereas you wanna stop the bad bots from using your APIs super, if you wanna add so, >>So good bots versus bad bots, that's the focus. Go ahead. Weigh in, weigh in on your thought on this >>Really breaks down into three key areas that we talk about here, sequence, right? One is you start by discovering all your APIs. How many APIs do I have in my environment that ly immediately highlight and say, Hey, you have, you know, 10,000 APIs. And that usually is an eye opener to many customers where they go, wow. I thought we had a 10th of that number. That usually is an eyeopener for them to, to at least know where they're at. The second thing is to tell them detection information. So discover, detect, and defend detect will tell them, Hey, your APIs are getting traffic from. So and so it addresses so and so infrastructure. So and so countries and so on that usually is another eye opener for them. They then get to see where their API traffic is coming from. Let's say, if you are a, if you're running a pizza delivery service out of California and your traffic is coming from Eastern Europe to go, wait a minute, nobody's trying, I'm not, I'm not, I don't deliver pizzas in Eastern Europe. Why am I getting traffic from that part of the world? So that sort of traffic immediately comes up and it will tell you that it is hitting your unauthenticated API. It is hitting your API. That has, that is vulnerable to a broken object level, that authorization, vulnerable be and so on. >>Yeah, I think, and >>Then comes the different aspect. Yeah. The different aspect is where you can take action and say, I wanna block certain types of traffic, or I wanna rate limit certain types of traffic. If, if you're seeing spikes there or you could maybe insert header so that it passes on to the end application and the application team can use that bit to essentially take a, a conscious response. And so, so the platform is very flexible in allowing them to take an action that suits their needs. >>Yeah. And I think this is the big trend. This is why I like what you guys are doing. One APIs we're built for the goodness of cloud. They're now the plumbing, you know, anytime you see plumbing involved, connection points, you know, that's pretty important. People are building it out and it has made the cloud what it is. Now, you got a security challenge. You gotta add more intelligence, more smarts to it. This is where I think platform versus tools matter. Can you guys just quickly share your thoughts on that? Cuz a lot of your customers and, and future customers have dealt with the sprawls of all these different tools. Right? I got a tool for this. I got a tool for that, but people are gravitating towards platforms, but how many platforms can a customer have? So again, this brings up the point point around how you guys are engaging with customers. Can you share your thoughts on tooling platforms? Your customers are constantly inundated with the same tsunami. Isn't new thing. Why, what, how should they look at this? >>Yeah, I mean, we don't wanna be, we don't wanna add to that alert fatigue problem that affects much of the cybersecurity industry by generating a whole bunch of alerts and so on. So what we do is we actually integrate very well with S IEM systems or so systems and allow customers to integrate the information that we are detecting or mitigating and feed them onto enterprise systems like a Splunk or a Datadog where they may have sophisticated processes built in to monitor, you know, spikes in anomalous traffic or actions that are taken by sequence. And that can be their dashboard where a whole bunch of alerting and reporting actually happens. So we play in the security ecosystem very well by integrating with other products and integrate very tightly with them, right outta the box. >>Okay. Mia, this is a wrap up now for the showcase. Really appreciate you guys sharing your awesome technology and very relevant product for your customers and where we are right now in this we call Supercloud or now multi-cloud or hybrid world of cloud. Share a, a little bit about the company, how people can get involved in your solution, how they can consume it and things they should know about, about sequence security. >>Yeah, we've been on this journey, an exciting journey it's been for, for about eight years. We have very large fortune 100 global 500 customers that use our platform on a daily basis. We have some amazing logos, both in Europe and, and, and in us customers are, this is basically not the shelf product customers not only use it, but depend on sequence. Several retailers. We are sitting in front of them handling, you know, black Friday, cyber, Monday, Christmas shopping, or any sort of holiday seasonality shopping. And we have handled that the journey starts by, by just simply looking at your API attack surface, just to a discover call with sequence, figure out where your APIs are posted work with you to prioritize how to protect them in a sort of a particular order and take the whole life cycle with sequence. This is, this is an exciting phase exciting sort of stage in the company's life. We just raised a very sort of large CDC round of funding in December from Menlo ventures. And we are excited to see, you know, what's next in, in, in the next, you know, 12 to 18 months. It certainly is the, you know, one of the top two or three items on the CSOs, you know, budget list for next year. So we are extremely busy, but we are looking for, for what the next 12 to 18 months are, are in store for us. >>Well, congratulations to all the success. So will you run the roadmap? You know, APIs are the plumbing. If you will, you know, they connection points, you know, you want to kind of keep 'em simple, as they say, keep the pipes dumb and make the intelligence around it. You seem to see more and more intelligence coming around, not just securing it, but does, where does this go in your mind? Where, where do we go beyond once we secure everything and manage it properly, APRs, aren't going away, they're only gonna get better and smarter. Where's the intelligence coming share a little bit. >>Absolutely. Yeah. I mean, there's not a dull moment in the space. As digital transformation happens to most enterprise systems, many applications are getting transformed. We are seeing an absolute explosion in the volume of APIs and the types of APIs as well. So the applications that were predominantly limited to data centers sort of deployments are now splintered across multiple different cloud environments are completely microservices based APIs, deep inside a Kubernetes cluster, for instance, and so on. So very exciting stuff in terms of proliferation of volume of APIs, as well as types of APIs, there's nature of APIs. And we are building very sophisticated machine learning models that can analyze traffic patterns of such APIs and automatically tell legitimate behavior from anomalous or suspicious behavior and so on. So very exciting sort of breadth of capabilities that we are looking at. >>Okay. I mean, yeah. I'll give you the final words since you're the CEO for the CSOs out there, the chief information security officers and the chief security officers, what do you want to tell them? If you could give them a quick shout out? What would you say to them? >>My shout out is just do an assessment with sequence. I think this is a repeating thing here, but really get to know your APIs first, before you decide what and where to protect them. That's the one simple thing I can mention for thes >>Am. Thank you so much for, for joining me today. Really appreciate it. >>Thank you. >>Thank you. Okay. That is the end of this segment of the eight of his startup showcase. Season two, episode four, I'm John for your host and we're here with sequin security. Thanks for watching.

(upbeat music) >> Hello and welcome to theCUBE's presentation of the AWS Startup Showcase. This is season two, episode four of the ongoing series covering the exciting hot startups from the AWS ecosystem. Here we're talking about cybersecurity in this episode. I'm your host, John Furrier here we're excited to have CUBE alumni who's back Snehal Antani who's the CEO and co-founder of Horizon3.ai talking about exploitable weaknesses and vulnerabilities with autonomous pen testing. Snehal, it's great to see you. Thanks for coming back. >> Likewise, John. I think it's been about five years since you and I were on the stage together. And I've missed it, but I'm glad to see you again. >> Well, before we get into the showcase about your new startup, that's extremely successful, amazing margins, great product. You have a unique journey. We talked about this prior to you doing the journey, but you have a great story. You left the startup world to go into the startup, like world of self defense, public defense, NSA. What group did you go to in the public sector became a private partner. >> My background, I'm a software engineer by education and trade. I started my career at IBM. I was a CIO at GE Capital, and I think we met once when I was there and I became the CTO of Splunk. And we spent a lot of time together when I was at Splunk. And at the end of 2017, I decided to take a break from industry and really kind of solve problems that I cared deeply about and solve problems that mattered. So I left industry and joined the US Special Operations Community and spent about four years in US Special Operations, where I grew more personally and professionally than in anything I'd ever done in my career. And exited that time, met my co-founder in special ops. And then as he retired from the air force, we started Horizon3. >> So there's really, I want to bring that up one, 'cause it's fascinating that not a lot of people in Silicon Valley and tech would do that. So thanks for the service. And I know everyone who's out there in the public sector knows that this is a really important time for the tactical edge in our military, a lot of things going on around the world. So thanks for the service and a great journey. But there's a storyline with the company you're running now that you started. I know you get the jacket on there. I noticed get a little military vibe to it. Cybersecurity, I mean, every company's on their own now. They have to build their own militia. There is no government supporting companies anymore. There's no militia. No one's on the shores of our country defending the citizens and the companies, they got to offend for themselves. So every company has to have their own military. >> In many ways, you don't see anti-aircraft rocket launchers on top of the JP Morgan building in New York City because they rely on the government for air defense. But in cyber it's very different. Every company is on their own to defend for themselves. And what's interesting is this blend. If you look at the Ukraine, Russia war, as an example, a thousand companies have decided to withdraw from the Russian economy and those thousand companies we should expect to be in the ire of the Russian government and their proxies at some point. And so it's not just those companies, but their suppliers, their distributors. And it's no longer about cyber attack for extortion through ransomware, but rather cyber attack for punishment and retaliation for leaving. Those companies are on their own to defend themselves. There's no government that is dedicated to supporting them. So yeah, the reality is that cybersecurity, it's the burden of the organization. And also your attack surface has expanded to not just be your footprint, but if an adversary wants to punish you for leaving their economy, they can get, if you're in agriculture, they could disrupt your ability to farm or they could get all your fruit to spoil at the border 'cause they disrupted your distributors and so on. So I think the entire world is going to change over the next 18 to 24 months. And I think this idea of cybersecurity is going to become truly a national problem and a problem that breaks down any corporate barriers that we see in previously. >> What are some of the things that inspired you to start this company? And I loved your approach of thinking about the customer, your customer, as defending themselves in context to threats, really leaning into it, being ready and able to defend. Horizon3 has a lot of that kind of military thinking for the good of the company. What's the motivation? Why this company? Why now? What's the value proposition? >> So there's two parts to why the company and why now. The first part was what my observation, when I left industry realm or my military background is watching "Jack Ryan" and "Tropic Thunder" and I didn't come from the military world. And so when I entered the special operations community, step one was to keep my mouth shut, learn, listen, and really observe and understand what made that community so impressive. And obviously the people and it's not about them being fast runners or great shooters or awesome swimmers, but rather there are learn-it-alls that can solve any problem as a team under pressure, which is the exact culture you want to have in any startup, early stage companies are learn-it-alls that can solve any problem under pressure as a team. So I had this immediate advantage when we started Horizon3, where a third of Horizon3 employees came from that special operations community. So one is this awesome talent. But the second part that, I remember this quote from a special operations commander that said we use live rounds in training because if we used fake rounds or rubber bullets, everyone would act like metal of honor winners. And the whole idea there is you train like you fight, you build that muscle memory for crisis and response and so on upfront. So when you're in the thick of it, you already know how to react. And this aligns to a pain I had in industry. I had no idea I was secure until the bad guy showed up. I had no idea if I was fixing the right vulnerabilities, logging the right data in Splunk, or if my CrowdStrike EDR platform was configured correctly, I had to wait for the bad guys to show up. I didn't know if my people knew how to respond to an incident. So what I wanted to do was proactively verify my security posture, proactively harden my systems. I needed to do that by continuously pen testing myself or continuously testing my security posture. And there just wasn't any way to do that where an IT admin or a network engineer could in three clicks have the power of a 20 year pen testing expert. And that was really what we set out to do, not build a autonomous pen testing platform for security people, build it so that anybody can quickly test their security posture and then use the output to fix problems that truly matter. >> So the value preposition, if I get this right is, there's a lot of companies out there doing pen tests. And I know I hate pen tests. They're like, cause you do DevOps, it changes you got to do another pen test. So it makes sense to do autonomous pen testing. So congratulations on seeing that that's obvious to that, but a lot of other have consulting tied to it. Which seems like you need to train someone and you guys taking a different approach. >> Yeah, we actually, as a company have zero consulting, zero professional services. And the whole idea is that build a true software as a service offering where an intern, in fact, we've got a video of a nine year old that in three clicks can run pen tests against themselves. And because of that, you can wire pen tests into your DevOps tool chain. You can run multiple pen tests today. In fact, I've got customers running 40, 50 pen tests a month against their organization. And that what that does is completely lowers the barrier of entry for being able to verify your posture. If you have consulting on average, when I was a CIO, it was at least a three month lead time to schedule consultants to show up and then they'd show up, they'd embarrass the security team, they'd make everyone look bad, 'cause they're going to get in, leave behind a report. And that report was almost identical to what they found last year because the older that report, the one the date itself gets stale, the context changes and so on. And then eventually you just don't even bother fixing it. Or if you fix a problem, you don't have the skills to verify that has been fixed. So I think that consulting led model was acceptable when you viewed security as a compliance checkbox, where once a year was sufficient to meet your like PCI requirements. But if you're really operating with a wartime mindset and you actually need to harden and secure your environment, you've got to be running pen test regularly against your organization from different perspectives, inside, outside, from the cloud, from work, from home environments and everything in between. >> So for the CISOs out there, for the CSOs and the CXOs, what's the pitch to them because I see your jacket that says Horizon3 AI, trust but verify. But this trust is, but is canceled out, just as verify. What's the product that you guys are offering the service. Describe what it is and why they should look at it. >> Yeah, sure. So one, when I back when I was the CIO, don't tell me we're secure in PowerPoint. Show me we're secure right now. Show me we're secure again tomorrow. And then show me we're secure again next week because my environment is constantly changing and the adversary always has a vote and they're always evolving. And this whole idea of show me we're secure. Don't trust that your security tools are working, verify that they can detect and respond and stifle an attack and then verify tomorrow, verify next week. That's the big mind shift. Now what we do is-- >> John: How do they respond to that by the way? Like they don't believe you at first or what's the story. >> I think, there's actually a very bifurcated response. There are still a decent chunk of CIOs and CSOs that have a security is a compliance checkbox mindset. So my attitude with them is I'm not going to convince you. You believe it's a checkbox. I'll just wait for you to get breached and sell to your replacement, 'cause you'll get fired. And in the meantime, I spend all my energy with those that actually care about proactively securing and hardening their environments. >> That's true. People do get fired. Can you give an example of what you're saying about this environment being ready, proving that you're secure today, tomorrow and a few weeks out. Give me an example. >> Of, yeah, I'll give you actually a customer example. There was a healthcare organization and they had about 5,000 hosts in their environment and they did everything right. They had Fortinet as their EDR platform. They had user behavior analytics in place that they had purchased and tuned. And when they ran a pen test self-service, our product node zero immediately started to discover every host on the network. It then fingerprinted all those hosts and found it was able to get code execution on three machines. So it got code execution, dumped credentials, laterally maneuvered, and became a domain administrator, which in IT, if an attacker becomes a domain admin, they've got keys to the kingdom. So at first the question was, how did the node zero pen test become domain admin? How'd they get code execution, Fortinet should have detected and stopped it. Well, it turned out Fortinet was misconfigured on three boxes out of 5,000. And these guys had no idea and it's just automation that went wrong and so on. And now they would've only known they had misconfigured their EDR platform on three hosts if the attacker had showed up. The second question though was, why didn't they catch the lateral movement? Which all their marketing brochures say they're supposed to catch. And it turned out that that customer purchased the wrong Fortinet modules. One again, they had no idea. They thought they were doing the right thing. So don't trust just installing your tools is good enough. You've got to exercise and verify them. We've got tons of stories from patches that didn't actually apply to being able to find the AWS admin credentials on a local file system. And then using that to log in and take over the cloud. In fact, I gave this talk at Black Hat on war stories from running 10,000 pen tests. And that's just the reality is, you don't know that these tools and processes are working for you until the bad guys have shown. >> The velocities there. You can accelerate through logs, you know from the days you've been there. This is now the threat. Being, I won't say lazy, but just not careful or just not thinking. >> Well, I'll do an example. We have a lot of customers that are Horizon3 customers and Splunk customers. And what you'll see their behavior is, is they'll have Horizon3 up on one screen. And every single attacker command executed with its timestamp is up on that screen. And then look at Splunk and say, hey, we were able to dump vCenter credentials from VMware products at this time on this host, what did Splunk see or what didn't they see? Why were no logs generated? And it turns out that they had some logging blind spots. So what they'll actually do is run us to almost like stimulate the defensive tools and then see what did the tools catch? What did they miss? What are those blind spots and how do they fix it. >> So your price called node zero. You mentioned that. Is that specifically a suite, a tool, a platform. How do people consume and engage with you guys? >> So the way that we work, the whole product is designed to be self-service. So once again, while we have a sales team, the whole intent is you don't need to have to talk to a sales rep to start using the product, you can log in right now, go to Horizon3.ai, you can run a trial log in with your Google ID, your LinkedIn ID, start running pen test against your home or against your network against this organization right now, without talking to anybody. The whole idea is self-service, run a pen test in three clicks and give you the power of that 20 year pen testing expert. And then what'll happen is node zero will execute and then it'll provide to you a full report of here are all of the different paths or attack paths or sequences where we are able to become an admin in your environment. And then for every attack path, here is the path or the kill chain, the proof of exploitation for every step along the way. Here's exactly what you've got to do to fix it. And then once you've fixed it, here's how you verify that you've truly fixed the problem. And this whole aha moment is run us to find problems. You fix them, rerun us to verify that the problem has been fixed. >> Talk about the company, how many people do you have and get some stats? >> Yeah, so we started writing code in January of 2020, right before the pandemic hit. And then about 10 months later at the end of 2020, we launched the first version of the product. We've been in the market for now about two and a half years total from start of the company till present. We've got 130 employees. We've got more customers than we do employees, which is really cool. And instead our customers shift from running one pen test a year to 40, 50 pen test. >> John: And it's full SaaS. >> The whole product is full SaaS. So no consulting, no pro serve. You run as often as you-- >> Who's downloading, who's buying the product. >> What's amazing is, we have customers in almost every section or sector now. So we're not overly rotated towards like healthcare or financial services. We've got state and local education or K through 12 education, state and local government, a number of healthcare companies, financial services, manufacturing. We've got organizations that large enterprises. >> John: Security's diverse. >> It's very diverse. >> I mean, ransomware must be a big driver. I mean, is that something that you're seeing a lot. >> It is. And the thing about ransomware is, if you peel back the outcome of ransomware, which is extortion, at the end of the day, what ransomware organizations or criminals or APTs will do is they'll find out who all your employees are online. They will then figure out if you've got 7,000 employees, all it takes is one of them to have a bad password. And then attackers are going to credential spray to find that one person with a bad password or whose Netflix password that's on the dark web is also their same password to log in here, 'cause most people reuse. And then from there they're going to most likely in your organization, the domain user, when you log in, like you probably have local admin on your laptop. If you're a windows machine and I've got local admin on your laptop, I'm going to be able to dump credentials, get the admin credentials and then start to laterally maneuver. Attackers don't have to hack in using zero days like you see in the movies, often they're logging in with valid user IDs and passwords that they've found and collected from somewhere else. And then they make that, they maneuver by making a low plus a low equal a high. And the other thing in financial services, we spend all of our time fixing critical vulnerabilities, attackers know that. So they've adapted to finding ways to chain together, low priority vulnerabilities and misconfigurations and dangerous defaults to become admin. So while we've over rotated towards just fixing the highs and the criticals attackers have adapted. And once again they have a vote, they're always evolving their tactics. >> And how do you prevent that from happening? >> So we actually apply those same tactics. Rarely do we actually need a CVE to compromise your environment. We will harvest credentials, just like an attacker. We will find misconfigurations and dangerous defaults, just like an attacker. We will combine those together. We'll make use of exploitable vulnerabilities as appropriate and use that to compromise your environment. So the tactics that, in many ways we've built a digital weapon and the tactics we apply are the exact same tactics that are applied by the adversary. >> So you guys basically simulate hacking. >> We actually do the hacking. Simulate means there's a fakeness to it. >> So you guys do hack. >> We actually compromise. >> Like sneakers the movie, those sneakers movie for the old folks like me. >> And in fact that was my inspiration. I've had this idea for over a decade now, which is I want to be able to look at anything that laptop, this Wi-Fi network, gear in hospital or a truck driving by and know, I can figure out how to gain initial access, rip that environment apart and be able to opponent. >> Okay, Chuck, he's not allowed in the studio anymore. (laughs) No, seriously. Some people are exposed. I mean, some companies don't have anything. But there's always passwords or so most people have that argument. Well, there's nothing to protect here. Not a lot of sensitive data. How do you respond to that? Do you see that being kind of putting the head in the sand or? >> Yeah, it's actually, it's less, there's not sensitive data, but more we've installed or applied multifactor authentication, attackers can't get in now. Well MFA only applies or does not apply to lower level protocols. So I can find a user ID password, log in through SMB, which isn't protected by multifactor authentication and still upon your environment. So unfortunately I think as a security industry, we've become very good at giving a false sense of security to organizations. >> John: Compliance drives that behavior. >> Compliance drives that. And what we need. Back to don't tell me we're secure, show me, we've got to, I think, change that to a trust but verify, but get rid of the trust piece of it, just to verify. >> Okay, we got a lot of CISOs and CSOs watching this showcase, looking at the hot startups, what's the message to the executives there. Do they want to become more leaning in more hawkish if you will, to use the military term on security? I mean, I heard one CISO say, security first then compliance 'cause compliance can make you complacent and then you're unsecure at that point. >> I actually say that. I agree. One definitely security is different and more important than being compliant. I think there's another emerging concept, which is I'd rather be defensible than secure. What I mean by that is security is a point in time state. I am secure right now. I may not be secure tomorrow 'cause something's changed. But if I'm defensible, then what I have is that muscle memory to detect, respondent and stifle an attack. And that's what's more important. Can I detect you? How long did it take me to detect you? Can I stifle you from achieving your objective? How long did it take me to stifle you? What did you use to get in to gain access? How long did that sit in my environment? How long did it take me to fix it? So on and so forth. But I think it's being defensible and being able to rapidly adapt to changing tactics by the adversary is more important. >> This is the evolution of how the red line never moved. You got the adversaries in our networks and our banks. Now they hang out and they wait. So everyone thinks they're secure. But when they start getting hacked, they're not really in a position to defend, the alarms go off. Where's the playbook. Team springs into action. I mean, you kind of get the visual there, but this is really the issue being defensible means having your own essentially military for your company. >> Being defensible, I think has two pieces. One is you've got to have this culture and process in place of training like you fight because you want to build that incident response muscle memory ahead of time. You don't want to have to learn how to respond to an incident in the middle of the incident. So that is that proactively verifying your posture and continuous pen testing is critical there. The second part is the actual fundamentals in place so you can detect and stifle as appropriate. And also being able to do that. When you are continuously verifying your posture, you need to verify your entire posture, not just your test systems, which is what most people do. But you have to be able to safely pen test your production systems, your cloud environments, your perimeter. You've got to assume that the bad guys are going to get in, once they're in, what can they do? So don't just say that my perimeter's secure and I'm good to go. It's the soft squishy center that attackers are going to get into. And from there, can you detect them and can you stop them? >> Snehal, take me through the use. You got to be sold on this, I love this topic. Alright, pen test. Is it, what am I buying? Just pen test as a service. You mentioned dark web. Are you actually buying credentials online on behalf of the customer? What is the product? What am I buying if I'm the CISO from Horizon3? What's the service? What's the product, be specific. >> So very specifically and one just principles. The first principle is when I was a buyer, I hated being nickled and dimed buyer vendors, which was, I had to buy 15 different modules in order to achieve an objective. Just give me one line item, make it super easy to buy and don't nickel and dime me. Because I've spent time as a buyer that very much has permeated throughout the company. So there is a single skew from Horizon3. It is an annual subscription based on how big your environment is. And it is inclusive of on-prem internal pen tests, external pen tests, cloud attacks, work from home attacks, our ability to harvest credentials from the dark web and from open source sources. Being able to crack those credentials, compromise. All of that is included as a singles skew. All you get as a CISO is a singles skew, annual subscription, and you can run as many pen tests as you want. Some customers still stick to, maybe one pen test a quarter, but most customers shift when they realize there's no limit, we don't nickel and dime. They can run 10, 20, 30, 40 a month. >> Well, it's not nickel and dime in the sense that, it's more like dollars and hundreds because they know what to expect if it's classic cloud consumption. They kind of know what their environment, can people try it. Let's just say I have a huge environment, I have a cloud, I have an on-premise private cloud. Can I dabble and set parameters around pricing? >> Yes you can. So one is you can dabble and set perimeter around scope, which is like manufacturing does this, do not touch the production line that's on at the moment. We've got a hospital that says every time they run a pen test, any machine that's actually connected to a patient must be excluded. So you can actually set the parameters for what's in scope and what's out of scope up front, most again we're designed to be safe to run against production so you can set the parameters for scope. You can set the parameters for cost if you want. But our recommendation is I'd rather figure out what you can afford and let you test everything in your environment than try to squeeze every penny from you by only making you buy what can afford as a smaller-- >> So the variable ratio, if you will is, how much they spend is the size of their environment and usage. >> Just size of the environment. >> So it could be a big ticket item for a CISO then. >> It could, if you're really large, but for the most part-- >> What's large? >> I mean, if you were Walmart, well, let me back up. What I heard is global 10 companies spend anywhere from 50 to a hundred million dollars a year on security testing. So they're already spending a ton of money, but they're spending it on consultants that show up maybe a couple of times a year. They don't have, humans can't scale to test a million hosts in your environment. And so you're already spending that money, spend a fraction of that and use us and run as much as you want. And that's really what it comes down to. >> John: All right. So what's the response from customers? >> What's really interesting is there are three use cases. The first is that SOC manager that is using us to verify that their security tools are actually working. So their Splunk environment is logging the right data. It's integrating properly with CrowdStrike, it's integrating properly with their active directory services and their password policies. So the SOC manager is using us to verify the effectiveness of their security controls. The second use case is the IT director that is using us to proactively harden their systems. Did they install VMware correctly? Did they install their Cisco gear correctly? Are they patching right? And then the third are for the companies that are lucky to have their own internal pen test and red teams where they use us like a force multiplier. So if you've got 10 people on your red team and you still have a million IPs or hosts in your environment, you still don't have enough people for that coverage. So they'll use us to do recon at scale and attack at scale and let the humans focus on the really juicy hard stuff that humans are successful at. >> Love the product. Again, I'm trying to think about how I engage on the test. Is there pilots? Is there a demo version? >> There's a free trials. So we do 30 day free trials. The output can actually be used to meet your SOC 2 requirements. So in many ways you can just use us to get a free SOC 2 pen test report right now, if you want. Go to the website, log in for a free trial, you can log into your Google ID or your LinkedIn ID, run a pen test against your organization and use that to answer your PCI segmentation test requirements, your SOC 2 requirements, but you will be hooked. You will want to run us more often. And you'll get a Horizon3 tattoo. >> The first hits free as they say in the drug business. >> Yeah. >> I mean, so you're seeing that kind of response then, trial converts. >> It's exactly. In fact, we have a very well defined aha moment, which is you run us to find, you fix, you run us to verify, we have 100% technical win rate when our customers hit a find, fix, verify cycle, then it's about budget and urgency. But 100% technical win rate because of that aha moment, 'cause people realize, holy crap, I don't have to wait six months to verify that my problems have actually been fixed. I can just come in, click, verify, rerun the entire pen test or rerun a very specific part of it on what I just patched my environment. >> Congratulations, great stuff. You're here part of the AWS Startup Showcase. So I have to ask, what's the relationship with AWS, you're on their cloud. What kind of actions going on there? Is there secret sauce on there? What's going on? >> So one is we are AWS customers ourselves, our brains command and control infrastructure. All of our analytics are all running on AWS. It's amazing, when we run a pen test, we are able to use AWS and we'll spin up a virtual private cloud just for that pen test. It's completely ephemeral, it's all Lambda functions and graph analytics and other techniques. When the pen test ends, you can delete, there's a single use Docker container that gets deleted from your environment so you have nothing on-prem to deal with and the entire virtual private cloud tears itself down. So at any given moment, if we're running 50 pen tests or a hundred pen tests, self-service, there's a hundred virtual private clouds being managed in AWS that are spinning up, running and tearing down. It's an absolutely amazing underlying platform for us to make use of. Two is that many customers that have hybrid environments. So they've got a cloud infrastructure, an Office 365 infrastructure and an on-prem infrastructure. We are a single attack platform that can test all of that together. No one else can do it. And so the AWS customers that are especially AWS hybrid customers are the ones that we do really well targeting. >> Got it. And that's awesome. And that's the benefit of cloud? >> Absolutely. And the AWS marketplace. What's absolutely amazing is the competitive advantage being part of the marketplace has for us, because the simple thing is my customers, if they already have dedicated cloud spend, they can use their approved cloud spend to pay for Horizon3 through the marketplace. So you don't have to, if you already have that budget dedicated, you can use that through the marketplace. The other is you've already got the vendor processes in place, you can purchase through your existing AWS account. So what I love about the AWS company is one, the infrastructure we use for our own pen test, two, the marketplace, and then three, the customers that span that hybrid cloud environment. That's right in our strike zone. >> Awesome. Well, congratulations. And thanks for being part of the showcase and I'm sure your product is going to do very, very well. It's very built for what people want. Self-service get in, get the value quickly. >> No agents to install, no consultants to hire. safe to run against production. It's what I wanted. >> Great to see you and congratulations and what a great story. And we're going to keep following you. Thanks for coming on. >> Snehal: Phenomenal. Thank you, John. >> This is the AWS Startup Showcase. I'm John John Furrier, your host. This is season two, episode four on cybersecurity. Thanks for watching. (upbeat music)