>>Hello, everyone. Welcome to the cubes presentation of the 80, but startup showcases MarTech is the focus. And this is all about the emerging cloud scale customer experience. This is season two, episode three of the ongoing series covering the exciting, fast growing startups from the cloud AWS ecosystem to talk about the future and what's available now, where are the actions? I'm your host John fur. Today. We joined by Cub alumni, Jerry Chen partner at Greylock ventures. Jerry. Great to see you. Thanks for coming on, >>John. Thanks for having me back. I appreciate you welcome there for season two. Uh, as a, as a guest star, >><laugh>, you know, Hey, you know, season two, it's not a one and done it's continued coverage. We, we got the episodic, uh, cube flicks model going >>Here. Well, you know, congratulations, the, the coverage on this ecosystem around AWS has been impressive, right? I think you and I have talked a long time about AWS and the ecosystem building. It just continues to grow. And so the coverage you did last season, all the events of this season is, is pretty amazing from the data security to now marketing. So it's, it's great to >>Watch. And 12 years now, the cube been running. I remember 2013, when we first met you in the cube, we just left VMware just getting into the venture business. And we were just riffing the next 80. No one really kind of knew how big it would be. Um, but we were kinda riffing on. We kind of had a sense now it's happening. So now you start to see every vertical kind of explode with the right digital transformation and disruption where you see new incumbents. I mean, new Newton brands get replaced the incumbent old guard. And now in MarTech, it's ripe for, for disruption because web two has gone on to web 2.5, 3, 4, 5, um, cookies are going away. You've got more governance and privacy challenges. There's a slew of kind of ad tech baggage, but yet lots of new data opportunities. Jerry, this is a huge, uh, thing. What's your take on this whole MarTech cloud scale, uh, >>Market? I, I think, I think to your point, John, that first the trends are correct and the bad and the good or good old days, the battle days MarTech is really about your webpage. And then email right there. There's, there's the emails, the only channel and the webpage was only real estate and technology to care about fast forward, you know, 10 years you have webpages, mobile apps, VR experiences, car experiences, your, your, your Alexa home experiences. Let's not even get to web three web 18, whatever it is. Plus you got text messages, WhatsApp, messenger, email, still great, et cetera. So I think what we've seen is both, um, explosion and data, uh, explosion of channel. So sources of data have increases and the fruits of the data where you can reach your customers from text, email, phone calls, etcetera have exploded too. So the previous generation created big company responses, Equa, you know, that exact target that got acquired by Oracle or, or, um, Salesforce, and then companies like, um, you know, MailChimp that got acquired as well, but into it, you're seeing a new generation companies for this new stack. So I, I think it's exciting. >>Yeah. And you mentioned all those things about the different channels and stuff, but the key point is now the generation shifts going on, not just technical generation, uh, and platform and tools, it's the people they're younger. They don't do email. They have, you know, proton mail accounts, zillion Gmail accounts, just to get the freebie. Um, they're like, they're, they'll do subscriptions, but not a lot. So the generational piece on the human side is huge. Okay. And then you got the standards, bodies thrown away, things like cookies. Sure. So all this is makes it for a complicated, messy situation. Um, so out of this has to come a billion dollar startup in my mind, >>I, I think multiple billion dollars, but I think you're right in the sense that how we want engage with the company branch, either consumer brands or business brands, no one wants to pick a phone anymore. Right? Everybody wants to either chat or DM people on Twitter. So number one, the, the way we engage is different, both, um, where both, how like chat or phone, but where like mobile device, but also when it's the moment when we need to talk to a company or brand be it at the store, um, when I'm shopping in real life or in my car or at the airport, like we want to reach the brands, the brands wanna reach us at the point of decision, the point of support, the point of contact. And then you, you layer upon that the, the playing field, John of privacy security, right? All these data silos in the cloud, the, the, the, the game has changed and become even more complicated with the startup. So the startups are gonna win. Will do, you know, the collect, all the data, make us secure in private, but then reach your customers when and where they want and how they want it. >>So I gotta ask you, because you had a great podcast just this week, published and snowflake had their event going on the data cloud, there's a new kind of SAS platform vibe going on. You're starting to see it play out. Uh, and one of the things I, I noticed on your podcast with the president of Hashi Corp, who was on people should listen to that podcast. It's on gray matter, which is the Greylocks podcast, uh, plug for you guys. He mentioned he mentions the open source dynamic, right? Sure. And, and I like what he, things, he said, he said, software business has changed forever. It's my words. Now he said infrastructure, but I'm saying software in general, more broader infrastructure and software as a category is all open source. One game over no debate. Right. You agree? >>I, I think you said infrastructure specifically starts at open source, but I would say all open source is one more or less because open source is in every bit of software. Right? And so from your operating system to your car, to your mobile phone, open source, not necessarily as a business model or, or, or whatever, we can talk about that. But open source as a way to build software distribute, software consume software has one, right? It is everywhere. So regardless how you make money on it, how you build software, an open source community ha has >>One. Okay. So let's just agree. That's cool. I agree with that. Let's take it to the next level. I'm a company starting a company to sell to big companies who pay. I gotta have a proprietary advantage. There's gotta be a way. And there is, I know you've talked about it, but I have my opinion. There is needs to be a way to be proprietary in a way that allows for that growth, whether it's integration, it's not gonna be on software license or maybe support or new open source model. But how does startups in the MarTech this area in general, when they disrupt or change the category, they gotta get value creation going. What's your take on, on building. >>You can still build proprietary software on top of open source, right? So there's many companies out there, um, you know, in a company called rock set, they've heavily open source technology like Rock's DB under the hood, but they're running a cloud database. That's proprietary snowflake. You talk about them today. You know, it's not open source technology company, but they use open source software. I'm sure in the hoods, but then there's open source companies, data break. So let's not confus the two, you can still build proprietary software. There's just components of open source, wherever we go. So number one is you can still build proprietary IP. Number two, you can get proprietary data sources, right? So I think increasingly you're seeing companies fight. I call this systems intelligence, right, by getting proprietary data, to train your algorithms, to train your recommendations, to train your applications, you can still collect data, um, that other competitors don't have. >>And then it can use the data differently, right? The system of intelligence. And then when you apply the system intelligence to the end user, you can create value, right? And ultimately, especially marketing tech, the highest level, what we call the system of engagement, right? If, if the chat bot the mobile UI, the phone, the voice app, etcetera, if you own the system of engagement, be a slack, or be it, the operating system for a phone, you can also win. So still multiple levels to play John in multiple ways to build proprietary advantage. Um, just gotta own system record. Yeah. System intelligence, system engagement. Easy, right? Yeah. >>Oh, so easy. Well, the good news is the cloud scale and the CapEx funded there. I mean, look at Amazon, they've got a ton of open storage. You mentioned snowflake, but they're getting a proprietary value. P so I need to ask you MarTech in particular, that means it's a data business, which you, you pointed out and we agree. MarTech will be about the data of the workflows. How do you get those workflows what's changing and how these companies are gonna be building? What's your take on it? Because it's gonna be one of those things where it might be the innovation on a source of data, or how you handle two parties, ex handling encrypted data sets. I don't know. Maybe it's a special encryption tool, so we don't know what it is. What's your what's, what's your outlook on this area? >>I, I, I think that last point just said is super interesting, super genius. It's integration or multiple data sources. So I think either one, if it's a data business, do you have proprietary data? Um, one number two with the data you do have proprietary, not how do you enrich the data and do you enrich the data with, uh, a public data set or a party data set? So this could be cookies. It could be done in Brad street or zoom info information. How do you enrich the data? Number three, do you have machine learning models or some other IP that once you collected the data, enriched the data, you know, what do you do with the data? And then number four is once you have, um, you know, that model of the data, the customer or the business, what do you deal with it? Do you email, do you do a tax? >>Do you do a campaign? Do you upsell? Do you change the price dynamically in our customers? Do you serve a new content on your website? So I think that workflow to your point is you can start from the same place, what to do with the data in between and all the, on the out the side of this, this pipeline is where a MarTech company can have then. So like I said before, it was a website to an email go to website. You know, we have a cookie fill out a form. Yeah. I send you an email later. I think now you, you can't just do a website to email, it's a website plus mobile apps, plus, you know, in real world interaction to text message, chat, phone, call Twitter, a whatever, you know, it's >>Like, it's like, they're playing checkers in web two and you're talking 3d chess. <laugh>, I mean, there's a level, there's a huge gap between what's coming. And this is kind of interesting because now you mentioned, you know, uh, machine learning and data, and AI is gonna factor into all this. You mentioned, uh, you know, rock set. One of your portfolios has under the hood, you know, open source and then use proprietary data and cloud. Okay. That's a configuration, that's an architecture, right? So architecture will be important in terms of how companies posture in this market, cuz MarTech is ripe for innovation because it's based on these old technologies, but there's tons of workflows, but you gotta have the data. Right. And so if I have the best journey map from a client that goes to a website, but then they go and they do something in the organic or somewhere else. If I don't have that, what good is it? It's like a blind spot. >>Correct. So I think you're seeing folks with the data BS, snowflake or data bricks, or an Amazon that S three say, Hey, come to my data cloud. Right. Which, you know, Snowflake's advertising, Amazon will say the data cloud is S3 because all your data exists there anyway. So you just, you know, live on S3 data. Bricks will say, S3 is great, but only use Amazon tools use data bricks. Right. And then, but on top of that, but then you had our SaaS companies like Oracle, Salesforce, whoever, and say, you know, use our qua Marketo, exact target, you know, application as a system record. And so I think you're gonna have a battle between, do I just work my data in S3 or where my data exists or gonna work my data, some other application, like a Marketo Ella cloud Z target, um, or, you know, it could be a Twilio segment, right. Was combination. So you'll have this battle between these, these, these giants in the cloud, easy, the castles, right. Versus, uh, the, the, the, the contenders or the, or the challengers as we call >>'em. Well, great. Always chat with the other. We always talk about castles in the cloud, which is your work that you guys put out, just an update on. So check out greylock.com. They have castles on the cloud, which is a great thesis on and a map by the way ecosystem. So you guys do a really good job props to Jerry and the team over at Greylock. Um, okay. Now I gotta ask kind of like the VC private equity sure. Market question, you know, evaluations. Uh, first of all, I think it's a great time to do a startup. So it's a good time to be in the VC business. I think the next two years, you're gonna find some nice gems, but also you gotta have that cleansing period. You got a lot of overvaluation. So what happened with the markets? So there's gonna be a lot of M and a. So the question is what are some of the things that you see as challenges for product teams in particular that might have that killer answer in MarTech, or might not have the runway if there's no cash, um, how do people partner in this modern era, cuz scale's a big deal, right? Mm-hmm <affirmative> you can measure everything. So you get the combination of a, a new kind of M and a market coming, a potential growth market for the right solution. Again, value's gotta be be there. What's your take on this market? >>I, I, I think you're right. Either you need runway, so cash to make it through, through this next, you know, two, three years, whatever you think the market Turmo is or two, you need scale, right? So if you're at a company of scale and you have enough data, you can probably succeed on your own. If not, if you're kind of in between or early to your point, either one focus, a narrower wedge, John, just like we say, just reduce the surface area. And next two years focus on solving one problem. Very, very well, or number two in this MarTech space, especially there's a lot of partnership and integration opportunities to create a complete solution together, to compete against kind of the incumbents. Right? So I think they're folks with the data, they're folks doing data, privacy, security, they're post focusing their workflow or marketing workflows. You're gonna see either one, um, some M and a, but I definitely can see a lot of Coopers in partnership. And so in the past, maybe you would say, I'm just raise another a hundred million dollars and do what you're doing today. You might say, look, instead of raising more money let's partner together or, or merge or find a solution. So I think people are gonna get creative. Yeah. Like said scarcity often is good. Yeah. I think forces a lot more focus and a lot more creativity. >>Yeah. That's a great point. I'm glad you brought that up up. Cause I didn't think you were gonna go there. I was gonna ask that biz dev activity is going to be really fundamental because runway combined with the fact that, Hey, you know, if you know, get real or you're gonna go under is a real issue. So now people become friends. They're like, okay, if we partner, um, it's clearly a good way to go if you can get there. So what advice would you give companies? Um, even most experienced, uh, founders and operators. This is a different market, right? It's a different kind of velocity, obviously architectural data. You mentioned some of those key things. What's the posture to partner. What's your advice? What's the combat man manual to kind of compete in this new biz dev world where some it's a make or break time, either get the funding, get the customers, which is how you get funding or you get a biz dev deal where you combine forces, uh, go to market together or not. What's your advice? >>I, I think that the combat manual is either you're partnering for one or two things, either one technology or two customers or sometimes both. So it would say which partnerships, youre doing for technology EG solution completers. Like you have, you know, this puzzle piece, I have this puzzle piece data and data privacy and let's work together. Um, or number two is like, who can help you with customers? And that's either a, I, they can be channel for you or, or vice versa or can share customers and you can actually go to market together and find customers jointly. So ideally you're partner for one, if not the other, sometimes both. And just figure out where in your life cycle do you need? Um, friends. >>Yeah. Great. My final question, Jerry, first of all, thanks for coming on and sharing your in insight as usual. Always. Awesome final question for the folks watching that are gonna be partnering and buying product and services from these startups. Um, there's a select few great ones here and obviously every other episode as well, and you've got a bunch you're investing in this, it's actually a good market for the ones that are lean companies that are lean and mean have value. And the cloud scale does provide that. So a lot of companies are getting it right, they're gonna break through. So they're clearly gonna be getting customers the buyer side, how should they be looking through the lens right now and looking at companies, what should they look for? Um, and they like to take chances with seeing that. So it's not so much, they gotta be vetted, but you know, how do they know the winners from the pretenders? >>You know, I, I think the customers are always smart. I think in the, in the, in the past in market market tech, especially they often had a budget to experiment with. I think you're looking now the customers, the buyer technologies are looking for a hard ROI, like a return on investment. And before think they might experiment more, but now they're saying, Hey, are you gonna help me save money or increase revenue or some hardcore metric that they care about? So I think, um, the startups that actually have a strong ROI, like save money or increased revenue and can like point empirically how they do that will, will, you know, rise to the top of, of the MarTech landscape. And customers will see that they're they're, the customers are smart, right? They're savvy buyers. They, they, they, they, they can smell good from bad and they're gonna see the strong >>ROI. Yeah. And the other thing too, I like to point out, I'd love to get your reaction real quick is a lot of the companies have DNA, any open source or they have some community track record where communities now, part of the vetting. I mean, are they real good people? >>Yeah. I, I think open stores, like you said, in the community in general, like especially all these communities that move on slack or discord or something else. Right. I think for sure, just going through all those forums, slack communities or discord communities, you can see what's a good product versus next versus bad. Don't go to like the other sites. These communities would tell you who's working. >>Well, we got a discord channel on the cube now had 14,000 members. Now it's down to six, losing people left and right. We need a moderator, um, to get on. If you know anyone on discord, anyone watching wants to volunteer to be the cube discord, moderator. Uh, we could use some help there. Love discord. Uh, Jerry. Great to see you. Thanks for coming on. What's new at Greylock. What's some of the things happening. Give a quick plug for the firm. When you guys working on, I know there's been some cool things happening, new investments, people moving. >>Yeah. Look we're we're Greylock partners, seed series a firm. I focus at enterprise software. I have a team with me that also does consumer investing as well as crypto investing like all firms. So, but we're we're seed series a occasionally later stage growth. So if you're interested, uh, FA me@jkontwitterorjgreylock.com. Thank you, John. >>Great stuff, Jerry. Thanks for coming on. This is the Cube's presentation of the, a startup showcase. MarTech is the series this time, emerging cloud scale customer experience where the integration and the data matters. This is season two, episode three of the ongoing series covering the hottest cloud startups from the ADWS ecosystem. Um, John farrier, thanks for watching.

>> Hello, everyone. Welcome to theCUBE presentation of the AWS startup showcase, market MarTech, emerging cloud scale customer experience. This is season two, episode three of the ongoing series covering the exciting startups from the AWS ecosystem. Talking about the data analytics, all the news and all the hot stories. I'm John Furrier your host of theCUBE. And today we're excited to be joined by Rachel Ostler, VP of product at Heap, Heap.io. Here to talk about from what, to why the future of digital insights. Great to see you, thanks for joining us today. >> Thanks for having me, John. Thanks for having me back. >> Well, we had a great conversation prior to the event here, a lot going on, you guys had acquired Auryc in an acquisition. You kind of teased that out last time. Talk about this, the news here, and why is it important? And first give a little setup on Heap and then the acquisition with Auryc. >> Yeah. So heap is a digital insights platform. So as you mentioned, it's all about analytics and so Heap really excels at helping you understand what your users and customers are doing in your digital application at scale. So when it comes to Auryc, what we really saw was a broken workflow. Maybe, I would even call it a broken market. Where a lot of customers had an analytics tool like Heap. So they're using Heap on one hand to figure out what is happening at scale with their users. But on the other hand, they were also using, like a session replay tool separately, to look at individual sessions and see exactly what was happening. And no one was very effective at using these tools together. They didn't connect at all. And so as a result, neither one of them could really be fully leveraged. And so with this acquisition, we're able to put these two tools together, so that users can both understand the what at scale, and then really see the why, immediately together in one place. >> You know, that I love that word why, because there's always that, you know, that famous motivational video on the internet, "you got to know your why", you know, it's very a motivational thing, but now you're getting more practicality. What and why is the, is the lens you want, right? So, I totally see that. And again, you can teased that out in our last interview we did. But I want to understand what's under the covers, under the acquisition. What was the big thesis behind it? Why the joint forces? What does this all mean? Why is this so important to understand this new, what and why and the acquisition specifically? >> Yeah, so let me give you example of a couple used cases, that's really helpful for understanding this. So imagine that you are a product manager or a, maybe a growth marketer, but you're someone who owns a digital experience. And what you're trying to do, of course, is make that digital experience amazing for your users so that they get value and that may mean that they're using it more, it may mean that new features are easily discoverable, that you can upsell things on your own. There's all sorts of different things that that may mean, but it's all about making it easy to use, discoverable, understandable, and as self-service as possible too. And so most of these digital builders, we call 'em digital builders sometimes. They are trying to figure out when the application is not working the way that it should be working, where people are getting stuck, where they're not getting the value and figure out how to fix that. And so, one really great used case is, I just want to understand in mass, like, let's say I have a flow, where are people dropping off? Right, so I see that I have a four step funnel and between step three and four people are dropping off. Heap is great for getting very detailed on exactly what action they're taking, where they're dropping off. But then the second you find what that action is, quantitatively, you want to watch it, you want to see what they did exactly before it. You want to see what they did after it. You want to understand why they're getting stuck. What they're confused at, are they mouthing over two things, like you kind of want to watch their session. And so what this acquisition allows us to do, is to put those things together seamlessly, you find the point in friction, you watch a bunch of examples, very easily. In the past, this would take you at least hours, if you could do it at all. And then in other used cases, the other direction. So there's the kind of, I think of it as the max to the min, and then there's the other direction as well. Like you have the, or maybe it's the macro to micro. You have the micro to macro, which is you have one user that had a problem. Maybe they send in a support ticket. Well, you can validate the problem. You can watch it in the session, but then you want to know, did this only happen to them? Did this happen to a lot of users? And this is really worth fixing, because all these customers are having the same problem. That's the micro to macro flow that you can do as well. >> Yeah. That's like, that's like the quantitative qualitative, the what and the why. I truly see the value there and I liked the way you explained that, good call out. The question I have for you, because a lot of people have these tools. "I got someone who does that." "I got someone over here that does the quantitative." "I don't need to have one company do it, or do I?" So the question I have for you, what does having a single partner or vendor, providing both the quantitative and the qualitative nails mean for your customers? >> So it's all because now it's immediate. So today with the two tools being separate, you may find something quantitatively. But then to, then to find the sessions that you want to watch that are relevant to that quantitative data point is very difficult. At least it takes hours to do so. And a lot of times people just give up and they don't bother. The other way is also true, you can watch sessions, you can watch as many sessions as you want, you can spend hours doing it, you may never find anything of interest, right? So it just ends up being something that users don't do. And actually we've interviewed a lot of customers, they have a lot of guilt about this. A lot of product managers feel like they should be spending all this time, but they just don't have the time to spend. And so it not only brings them together, but it brings them together with immediacy. So you can immediately find the issue, find exactly where it is and watch it. And this is a big deal, because, if you think about, I guess, like today's economic conditions, you don't have a lot of money to waste. You don't have a lot of time to waste. You have to be very impactful with what you're doing and with your spending of development resources. >> Yeah. And totally, and I think one of the things that immediacy is key, because it allows you to connect dots faster. And we have the aha moments all the time. If you miss that, the consequences can be quantified in a bad product experience and lost customers. So, totally see that. Zooming out now, I want to get your thoughts on this, cause you're bringing, we're going down this road of essentially every company is digital now, right? So digitization, digital transformation. What do you want to call it? Data is digital. This video is an experience. It's also data as well. You're talking, we're going to share this and people are going to experience that. So every website that's kind of old school is now becoming essentially a digital native application or eCommerce platform. All the things that were once preserved for the big guys, the hyper-scalers and the categories, the big budgets, now are coming down to every company. Every company is a digital company. What challenges do they have to transition from? I got a website, I got a marketing team. Now I got to look like a world class, product, eCommerce, multifaceted, application with developers, with change, with agility? >> Well, so I think that last thing you said is a really important part of it, the agility. So, these products, when you're going from a, just a website to a product, they're a lot more complex. Right? And so maybe I can give an example. We have a customer, it's an insurance company. So they have this online workflow. And if you can imagine signing up for insurance online, it's a pretty long complicated workflow. I mean, Hey, better to do it online than to have to call someone and wait on, you know, on the phone. And so it's a good experience, but it's still fraught with like opportunities of people getting stuck and never coming back. And so one of the things that Heap allowed this customer to do was figure out something that wasn't working in their workflow. And so if you think about traditional analytics tools, typically what you're doing is you're writing tracking code and you're saying, "Hey, I'm going to track this funnel, this process." And so maybe it has, you know, five different forms or pages that you have to go through. And so what you're doing when you track it is you say, did you submit the first one? Did you submit the second one? Did you submit the third one? So you know, like where they're falling off. You know where they're falling off, but you don't know why, you don't know which thing got them stuck because each one of these pages has multiple inputs and it has maybe multiple steps that you need to do. And so you're completely blind to exactly what's happening. Well, it turned out because Heap collects all this data, that on one of these pages where users were dropping off, it was because they were clicking on a FAQ, there was a link to a FAQ, and because this was a big company, the FAQ took them to a completely different application. Didn't know how to get back from there and they just lost people. And imagine if you are doing this with traditional means today, right? You don't have any visibility into what's happening on that page, you just know that they fell off. You might think about what do I do to fix this? How do I make this flow work better? And you might come up with a bunch of ideas. One of your ideas could be, let's break it into multiple pages. Maybe there's too much stuff on this page. One of your ideas may have been, let's try a FAQ. They're getting stuck, let's give them some more help. That would be a very bad idea, right? Because that was actually the reason why they were leaving and never coming back. So, the point I'm making is that, if you don't know exactly where people are getting stuck and you can't see exactly what is happening, then you're going to make a lot of very bad decisions. You're going to waste a lot of resources, trying things that make no sense. It is hard enough as a digital builder and all the product managers and growth marketers and marketers out there can attest to this, it's hard enough when you know exactly what the problem is to figure out a good solution. Right? That's still hard. But if you don't know the problem, it's impossible. >> Okay, so let's just level up, the bumper sticker now for the challenges are what? Decision making, what's the, stack rank the top three challenges from that. So it's being agile, right? So being very fast, because you're competing with a lot of companies right now. It's about making really good decisions and driving impact, right? So you have to have all the data that you need. You have to have the, the specific information about what's going on. Cause if you don't have it, you're going to decide to invest in things and you're not going to drive the impact that you want. >> So now you got the acquisition of Auryc and Auryc and you have the, this visibility to the customers that are building, investing, you mentioned, okay. As they invest, whether it's the digital product or new technology in R and D, what feedback have you guys seen from these investments, from these customers, what results have come out of it? Could you share any specific answers to the problems and challenges you have outlined, because you know, there's growth hackers could be failing cause of stupid little product mistakes that could have been avoided in the feedback, you know what I'm saying? So it's like, where can you, where are these challenges addressed and what are some of the results? >> Yeah, so, what we've seen with our customers is that when they are applying this data and doing this analysis on say workflows or goals that they're trying to accomplish, they've been able to move the needle quite a bit. And so, whether it is, you know, increasing conversion rates or whether it is making sure that they don't have, you know, drop off of trial signups or making sure that their customers are more engaged than before, when they know exactly where they're failing, it is much easier to make an investment and move the needle. >> Awesome. Well, let's move on to the next big topic, which I love, it's about data science and data engineering. You guys are a data company and I want to ask you specifically, how Heap uniquely is positioned to help companies succeed, where in the old big tech world, they're tightening the ropes on secure cookies, privacy, data sharing. At the same time, there's been an explosion in cloud scale data opportunities and new technologies. So it seems like a new level of, capability, is going to replace the old cookies, privacy and data sharing, which seem to be constricting or going away. How do you, what's your reaction to that? Can you share how Heap fits into this next generation and the current situation going on with the cookies and this privacy stuff. >> Yep, so it is really important in this world to be collecting data compliantly, right? And so what that means is, you don't want to be reliant on third party cookies. You want to be reliant on just first party information. You want to make sure that you don't collect any PII. Heap is built to do that from the ground up. We by default will not collect information, like what do people put into forms, right? Because that's a obvious source of PII. The other thing is that, there's just so much data. So you kind of alluded to this, with this idea of data science. So first of all, you're collecting data compliantly, you're making sure that you have all the data of what your user actions are doing, compliantly, but then it's so much data that it like, how do you know where to start? Right? You want to know, you want to get to that specific point that users are dropping off, but there's so many different options out there. And so that's where Heap is applying data science, to automatically find those points of friction and automatically surface them to users, so that you don't have to guess and check and constantly guess at what the problem is, but you can see it in the product surface right for you. >> You know, Rachel, that's a great point. I want to call that out because I think a lot of companies don't underestimate, they may underestimate what you said earlier, capturing in compliance way means, you're opting in to say, not to get the data, to unwind it later, figure it out. You're capturing it in a compliant way, which actually reduces the risk and operational technical debt you might have to deploy to get it fixed on compliance. Okay, that's one thing, I love that. I want to make sure people understand that value. That's a huge value, especially for people that don't have huge teams and diverse platforms or other data sources. The other thing you mentioned is owning their own data. And that first party data is a strategic advantage, mainly around personalization and targeted customer interaction. So the question is, with the new data, I own the data, you got the comp- capture with compliance. How do you do personalization and targeted customer interactions, at the same time while being compliant? It just seems, it seems like compliance is restrictive and kind of forecloses value, but open means you can personalization and targeted interactions. How do you guys connect the dots there by being compliant, but yet being valuable on the personalization and targeted? >> Well, it all depends on how the customer is managing their information, but imagine that you have a logged in user, well, you know, who the logged in user is, right? And so all we really need is an ID. Doesn't have, we don't need to know any of the user information. We just need an ID and then we can serve up the information about like, what have they done, if they've done these three actions, maybe that means that this particular offer would be interested to them. And so that information is available within Heap, for our customers to use it as they want to, with their users. >> So you're saying you can enable companies to own their data, be compliant and then manage it end to end from a privacy standpoint. >> Yes. >> That's got to be a top seller right there. >> Well, it's not just a top seller, it's a necessity. >> It's a must have. I mean, think about it. I mean, what are people, what are the, what are people who don't do this? What do they face? What's the alternative? If you don't keep, get the Heap going immediately, what's the alternative? I'm going through logs, I got to have to get request to forget my data. All these things are all going on, right? Is, what's the consequence of not doing this? >> Well, there's a couple consequences. So one is, and I kind of alluded to it earlier that, you're just, you're blind to what your users are doing, which means that you're making investments that may not make sense, right? So you can, you can decide to add all the cool features in the world, but if the customers don't perceive them as being valuable or don't find them or don't understand them, it doesn't, it doesn't serve your business. And so, this is one of like the rule number one of being a product manager, is you're trying to balance what your customers need, with what is also good for your business. And both of those have to be in place. So that's basically where you are, is that you'll be making investments that just won't be hitting the mark and you won't be moving the needle. And as I mentioned, it's more important now in this economic climate than ever to make sure that the investments you're making are targeted and impactful. >> Yeah and I think the other thing to point out, is that's a big backlash against the whole, Facebook, you're the product, you're getting used, the users being used for product, but you're, you guys have a way to make that happen in a way that's safe for the user. >> Yes. Safe and compliant. So look, we're all about making sure that we certainly don't get our customers into trouble and we recommend that they follow all compliance rules, because the last thing you want to be is on the, on the wrong side of a compliance officer. >> Well, there's also the user satisfaction problem of, and the fines. So a lot going on there, great product. I got to ask you real quick before we kind of wrap up here. What's the reaction been to the acquisition? Quantitative, qualitative. What's been the vibe? What are some, what are people saying about it? >> We've got a lot of interest. So, I mentioned earlier that this is really a broken workflow in the market. And when users see the two products working together, they just love it because they have not been able to leverage them being separate before. And so it just makes it so much easier for these digital builders to figure out, what do I invest in because they know exactly where people are having trouble. So it's been really great, we've had a lot of reach outs already asking us how they can use it, try it, not quite available yet. So it's going to be available later this summer, but great, great response so far. >> Awesome. Well, I love the opportunity. Love the conversation, I have to ask you now, looking forward, what does the future look like for companies taking advantage of your platform and tool? What can they expect in terms of R and D investments, area moves you're making? You're the head of product, you get the keys to the kingdom. What's the future look like? What's coming next? >> Yeah, so other than pulling the qual and the quant together, you actually hinted at it earlier when you're asking me about data science, but continuing to automate as much of the analysis as we can. So, first of all, analysis, analytics, it should be easy for everyone. So we're continue to invest in making it easy, but part of making it easy is, like we can automate analysis. We can, we can see that your website has a login page on it and build a funnel for you automatically. So that's some of the stuff that we're working on, is how do we both automate getting up to speed and getting that initial analysis done easily, without any work. And then also, how do we automate more complex analysis? So you have, typically a lot of companies have a data science team and they end up doing a lot of analysis, it's a little bit more complex. I'm not saying data science teams will go away, they will be around forever. There's tons of very complex analysis that they're probably not even getting time to do. We're going to start chipping away at that, so we can help product managers do more and more of that self-service and then free up the data science team to do even more interesting things. >> I really like how you use the word product managers, product builders, digital builders, because while I got you, I want to get your thought on this, because it's a real industry shift. You're talking about it directly here, about websites going to eCommerce, CMOs, a C-suite, they generally observe that websites are old technology, but not going away, because the next level abstraction builds on top of it. What's the new capabilities because for the CMOs and the C-suites and the product folks out there, they're not building webpages, they're building applications. So what is it about this new world that's different from the old web architecture? How would you talk to a CMO or a leader? And to, when they ask what's this new opportunity to take my website, cause maybe it's not enough traffic. People are consuming out in the organic, what's this new expectation and how, what does a new product manager environment look like, if it's not the web, so to speak? >> Well, there's a couple things. So one is, and you alluded to it a bit, like the websites are also getting more complex and you need to start thinking of your website as a product. Now it's, it may not be the product that you sell, but it is, well for eCommerce it's the place that you get access to the product, for B2B SaaS, it is the window to the product. It's a place where you can learn about the product. And you need to think about, not just like, what pieces of content are being used, but you need to understand the user flow, through the application. So that's how it's a lot more like a product. >> Rachel, thanks so much for coming on theCUBE here for this presentation, final word, put a plugin for the company. What are you guys up to? What are you looking for? Take a minute to explain kind of that, what's going on. How do people contact you with a great value proposition? Put a plugin for the company. >> Yeah, well, if you want to up level your product experience or website experience, you want to be able to drive impact quickly, try Heap. You can go to Heap.io, you can try it for free. We have a free trial, we have a free product even. And yeah, and then if you have any questions, you want to talk to a live person, you can do that too, at sales@Heap.io. >> Rachel, thanks so much. Customer-scale experiences with the cloud house league. This is the season two, episode three of the ongoing series. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(gentle music) >> Hey everyone. Welcome to theCUBE's coverage of the AWS Startup Showcase. Season two, episode three. This is about MarTech, emerging cloud scale customer experience. This is our ongoing series that you know and love hopefully that feature a great number of AWS ecosystem partners. I'm your host, Lisa Martin. Got a great guest here from Branch. Michelle Lerner joins me, the senior director of business development. She's going to be talking about Branch but also about one of your favorite brands, Peet's, yep, the coffee place, and how they supercharged loyalty and app adoption with Branch. Michelle, it's great to have you on the program. >> Yeah. Great to be here. Thank you so much for having me. >> Tell us a little bit about Branch, what you guys do for the modern mobile marketer. >> Yeah, absolutely. So you can think about Branch as a mobile linking platform. So what that means is we offer a seamless deep linking experience and insightful campaign measurement across every single marketing channel and platform on mobile. We exist so that we can break down walled gardens to help our customers engage with their customers in the most optimal way across any device and from every marketing channel. Our products are specifically designed to help create an amazing user experience, but also provide full picture holistic downstream measurement across any paid, owned, and earned channels so that brands can actually see what's working. So what that really means is that we make it really easy to scale our links across every single marketing channel, which then route the users to the right place at any device through even past install so that they can get to the context that they expect for a seamless experience. We then provide that cross channel analytics back to the brand so that they could see what's working and they can make better business decisions. So kind of summing it up, our industry leading mobile linking actually powers those deep links, also supports that measurement so that brands can build a sophisticated experience that actually delight their users but also improve their metrics and conversion rates. >> Those two things that you said are key. We expected to be delighted with whatever experience we're having and we also want to make sure, and obviously, the brands want to make sure that they're doing that but also that from an attribution perspective, from a campaign conversion perspective, that they can really understand the right tactics and the right strategic elements that are driving those conversions. That's been a challenge for marketers for a long time. Speaking of challenges, we've all been living through significant challenges. There's no way to say it nicely. The last two years, every industry completely affected by the pandemic talk. We're going to talk about Peet's Coffee. And I want to understand some of the challenges that you saw in the quick service restaurant or QSR industry at large. Talk to me about those industry challenges and then we'll dig into the Peet's story. >> Yeah, absolutely. So obviously the pandemic changed so much in our lives whether it's going to work or commuting or taking our kids to school or even getting our morning coffee. So when you think about Peet's, specifically within the QSR industry, they knew that they needed to innovate in order to make sure that they could provide their customers with their daily cups of coffee in a really safe and effective way. So they thought really quickly on their feet, they engaged us at Branch to help launch their order ahead messaging across their online and offline channels. They really wanted to maintain their commitment to an excellent customer experience but in a way that obviously would be safe and effective. >> That was one of the things that I missed the very most in the very beginning of the pandemic was going to my local Peet's. I missed that experience. Talk me about, you mentioned the online and offline, I'm very familiar with the online as an app user, mobile app user, but what were some of the challenges that they were looking to Branch to resolve on the offline experiences? People were queuing outside or for those folks that were they trying to get folks to convert to using the mobile app that maybe weren't users already? What was that online and offline experience? What were some of the challenges they were looking to resolve? >> Yeah, absolutely. The modern marketer is really both, like you said, online and offline, there is a heavy focus within the app and Peet's kind of wanted to bridge those two by pushing users into the app to provide a better experience there. So what they ended up doing was they used our deep linking capabilities to seamlessly route their customers to their loyalty program and their rewards catalog and other menu offerings within the app so that they could actually get things done in real time, but also in real time was the ability to then measure across those different campaigns so that they had visibility, Peet's, into kind of the way that they could optimize that campaign performance but also still give that great experience to their users. And they actually saw higher loyalty adoption, order values, and attributed purchases when they were able to kind of see in real time where these users were converting. But another thing that we're actually seeing across the board and Peet's did a great job of this was leveraging Branch power QR codes where we are seeing like the rebirth of the QR code. They're back, they're here to stay. They actually used that across multiple channels. So they used it with their in-store signage. You might have even seen it on their to go cups, coffee cards that were handed out by baristas. They were all encouraging customers to go order ahead using the Peet's coffee app. But that was kind of just the beginning for them. The creation of unique links for those QR codes actually spread for them to create Branch links across everything from emails to ads on Instagram. So before long, most of Peet's retail marketing were actually Branch links just because of the ease of creation and reliability, but more so again, going back to that customer experience, it really provided that good experience for the customers to make sure that they were getting within the mobile app so that they can take action and order their coffee. Another way that Branch kind of bridges the different platforms is actually between mobile web and app. Peet used Branch Journeys and that's a product of ours. It's a way that they can convert their mobile web users into app users. So they used deferred deep links with the ultimate goal of then converting those users into high value app users. So the Peet's team actually tested different creative and interstitials across the mobile site which would then place those users into the key pages, like either the homepage or the store locator, or the menu pages within the app. So that also helped them kind of build up not just their mobile app order online but also their delivery business so they could hire new trials of seasonal beverages. They could pair them with a free delivery offering. So they knew that they were able to leverage that at scale across multiple initiatives. >> I love those kinds of stories where it's kind of like a land and expand where there was obviously a global massive problem. They saw that recognized our customers are still going to be is demanding. Maybe if not more than they were before with I want my coffee, I want it now, you mentioned real time. I think one of the things we learned during the pandemic is access to realtime data isn't a nice to have anymore. We expect it as consumers even in our business lives, but the ability to be able to measure, course correct, but then see, wow, this is driving average order value up, we're getting more folks using our mobile app, maybe using delivery. Let's expand the usage of Branch across what we're doing in marketing can really help transform our marketing organization and a business at the brand level. >> Absolutely. And it also helps predict that brand loyalty. Because like you said, we, as consumers expect that that brands are going to kind of follow us where we are in our life cycle as consumers and if you don't do that, then you're going to be left in the dust unfortunately. >> I think one of the memories that will always stick with me, Michelle, during the last couple years is that first cup of Peet's that I didn't have to make at home myself. Just finally getting the courage to go back in, use the app, go in there, but oh man, that was probably the best taste of coffee I probably will ever have. You mentioned some of the products, you mentioned Journeys, and that allows them to do AB testing, looking at different CTAs, being able to kind of course correct and adjust campaigns in real time. >> Yeah, absolutely. So Journeys, what it does is it's basically a banner or a full page interstitial that is populated on the mobile web. So if you go to let's say Peets.com, you could get served as a user, either different creative or depending on where you are, location wise, you could be in the store, maybe there's a promotion. So it's triggered by all these different targeting capabilities. And so what that does is it takes me as a user. I can click that and go into the app where, like we said before, we have higher order value, higher lifetime value of a customer. And all my credit card information is saved. It just makes it so much more seamless for me to convert as a user within the app. And obviously Peet's likes that as well because then their conversion rates are actually higher. There's also kind of fun ways to play around with it. So if I am already a loyal customer and I have the app, you probably would target different creative for me than you would for someone who doesn't have the app. So you could say, hey, download our app, get $5 off of your next mobile order. Things like that you could play around with and you can see really does help increase that loyalty. But actually they were able to take, they kind of are experimenting with the geotargeted journeys in different key markets with different Peet's. And actually it was helping ultimately get their reinstalls growing. So for customers who maybe had the app before but needed to reinstall it because now there's such a bigger focus, they saw it both on the acquisition and the re-engagement side as well. >> So Branch has been pretty transformative, not in my estimation to Peet's marketing, but to Peet's as a business I'm hearing absolutely customer loyalty, revenue obviously impacted, brand loyalty, brand reputation. These are things that really kind of boil up to the top of the organization. So we're not just talking about benefits to the marketing and the sales folks. This is the overall massive business outcomes that you guys are enabling organizations like Peet's to generate. >> Yeah, definitely. And that's kind of what we tell our customers when they come to Branch. We want them to think about what their overall business objectives are versus if you think just campaign by campaign, okay, that's fine. But ultimately what are we trying to achieve? How could we help the bottom line? And then how can we also kind of help integrate with other mobile marketing technology or the modern tech stack that they're using? How do we integrate into that and actually provide not just a seamless experience for their end user, but with their marketing orgs, their product orgs, whoever's kind of touching the business as well? >> Have you noticed along those lines in the last couple of years as things like customer delight, seamless experience, the ability to translate, if I start on my iPad and I go to my laptop and then I finish a transaction on my phone, have you noticed your customer conversations increasing up to the C-suite level? Is this much more of a broad organizational objective around we've got to make sure that we have a really strong digital user experience? >> Yeah, absolutely. Like we were talking about before, it really does help affect the bottom line when you're providing a great experience with Branch being a mobile linking platform, our links just work. We outperform everybody else in the space and it might sound like really simple, okay, a link is working getting me from point A to point B, but doing it the right way and being consistent actually will increase performance over time of all these campaigns. So it's just an addition to providing that experience, you're seeing those key business results every single time. >> Talk about attribution for a minute because I've been in marketing for a long time in the tech industry. And that's always one of the challenges is we want to know what lever did the customer pull that converted them from opportunity to a lead to whatnot? Talk about the ability for Branch from an attribution perspective to really tell those marketers and the organization exactly, tactically, down to the tactical level, this is what's working. This is what's not working. Even if it's a color combination for example. That science is critical. >> Yeah, absolutely. Because we are able to cover the entire marketing life cycle of that they're trying to reach their customers. We cover off on email. We have mobile web to app. We have organic, we have search. No matter what you can look at that purview under a Branch lens. So we are just providing not just the accurate attribution down to the post-install, what happens after that, but also a more holistic view of everything that's happening on mobile. So then you can stitch all that together and really look at which ones are actually performing so you could see exactly which campaigns attributed directly to what amount of spend or which campaigns helped us understand the true lifetime long term value of customers, let's say in this case who ordered delivery or pickup. So to the kind of customer persona, it really helped. And also they actually were able to see Peet's because of our attribution, they saw actually a four and a half time increase in attributed purchases at the peak of the pandemic. And even since then, they're still seeing a three times increase in monthly attributed purchases. So because they actually have the view across everything that they're doing, we're able to provide that insight. >> That insight is so critical these days, like we mentioned earlier talking about real time data. Well we expect the experiences to be real time. And I expect that when I go back on the app they're going to know what I ordered last time. Maybe I want that again. Maybe I want to be able to change that, but I want them to know enough about me in a non creepy way. Give me that seamless experience that I'm expecting because of course that drives me to come back over and over again and spend way too much money there which I'm guilty of, guilty as charged. >> Coffee is totally fine. >> Right? Thank you. Thank you so much for validating that. I appreciate that. But talk to me about, as we are kind of wrapping things up here, the brick and mortars, it was such a challenge globally, especially the mom and pops to be able to convert quickly and figure out how do we reach a digital audience? How do we get our customers to be loyal? What's some of the advice that you have for the brick and mortars or those quick service restaurants like Peet's who've been navigating this the last couple years now here we are in this interesting semi post pandemic I would like to believe world? >> Yeah, we're getting there slowly but surely, but yeah, it's really important for them to adapt as we kind of move into this semi post pandemic world, we're kind of in the middle of like a hybrid online, offline, are we in stores, are we ordering online? These brand and customer relationships are super complex. I think the mobile app is just one part of that. Customers really shouldn't have any problems getting from the content or item they're looking for, no matter if they're in the store, if they're in the app, if they're on the desktop, if they're checking their email, if they're perusing TikTok, the best customer relationships really are omnichannel in nature. So what I would say, the need for providing the stellar customer experience isn't going to go away. It's actually really key. Whether it's driving users from their mobile properties to the app, providing a great in-store experience, like the QR codes, customers are expecting a lot more than they did before the pandemic. So they're not really seeing these brand touch points as little silos. They're seeing one brand. So it really should feel like one brand you should speak to the customers as if it's one brand across every single device, channel, and platform, and really unify that experience for them. >> Absolutely. That's going to be I think for so many different brands, whether it's a brick and mortar QSR, that's going to be one of the defining competitive advantages. If they can give their end users a single brand experience across channels, and you mentioned TikTok, those channels are only going to grow. As are I think or expectations. I don't think anybody's going to go back to wanting less than they did two years ago, right? >> Absolutely. Absolutely. >> Well this has been great, Michelle, thank you so much for joining me, talking about Branch, what you guys are doing, mobile linking platform, mobile measurement platform, the deep links, what you were able to do with Peet's Coffee, a beloved brand since the 60s and so many others. We appreciate your insights, your time and the story that you shared. >> Thank you so much, Lisa. I hope you have a great rest of your day. >> You as well. For Michelle Lerner, I'm Lisa Martin. You're watching theCUBE's coverage of the AWS Showcase. Keep it right here. More great content coming up from theCUBE, the leader in live tech coverage. (gentle music)

(upbeat music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase. Topic is MarTech: Emerging Cloud-Scale Experience. This is season two, episode three of the ongoing series covering the exciting startups from the AWS ecosystem. Talk about their value proposition and their company and all the good stuff that's going on. I'm your host, John Furrier. And today we're excited to be joined by Manyam Mallela who's the co-founder and head of AI at Blueshift. Great to have you on here to talk about the Blueshift-Intelligent Customer Engagement, Made Simple. Thanks for joining us today. >> Thank you, John. Thank you for having me. >> So last time we did our intro video. We put it out in the web. Got great feedback. One of the things that we talked about, which is resonating out there in the viral Twitter sphere and in the thought leadership circles is this concept that you mentioned called 10X marketer. That idea that you have a solution that can provide 10X value. Kind of a riff on the 10X engineer in the DevOps cloud world. What does it mean? And how does someone get there? >> Yeah, fantastic. I think that's a great way to start our discussion. I think a lot of organizations, especially as of this current economic environment are looking to say, I have limited resources, limited budgets, how do I actually achieve digital and customer engagement that helps move the needle for my key metrics, whether it's average revenue per user, lifetime value of the user and frequent interactions. Above all, the more frequently a brand is able to interact with their customers, the better they understand them, the better they can actually engage them. And that usually leads to long term good outcomes for both customer and the brand and the organizations. So the way I see 10X marketer is that you need to have tools that give you that speed and agility without hindering your ability to activate any of the campaigns or experience that you want to create. And I see the roadblocks usually for many organizations, is that kind of threefold. One is your data silos. Usually data that is on your sites, does not talk to your app data, does not talk to your social data, does not talk to your CRM data and so forth. So how do I break those silos? The second is channel silos. I actually have customers who are only engaging on email or some are on email and mobile apps. Some are on email and mobile apps and maybe the OTT TV in a Roku or one of the connected TV experiences, or maybe in the future, another Web3 environments. How do I actually break those channel silos so that I get a comprehensive view of the customer and my marketing team can engage with all of them in respect to the channel? So break the channel silos. And the last part, what I call like some of the little talked about is I call the inside silo, which is that, not only do you need to have the data, but you also have to have a common language to share and talk about within your organizations. What are we learning from our customers? What do we translate our learning and insight on this common data platform or fabric into an action? And that requires the shared language of how do I actually know my customers and what do I do with them? Like either the inside silo as well. I think a lot of times organizations do get into this habit like each one speaks their own language, but they don't actually are talking the common language of what did we actually know about the real customer there. >> Yeah, and I think that's a great conversation because there's two, when you hear 10X marketer or 10X conversations, it implies a couple things. One is you're breaking an old way and bringing in something new. And the new is a force multiplier, in this case, 10X marketer. But this is the cloud scale so marketing executives, chiefs, staffs, chiefs of staffs of CMOs and their staffs. They want to get that scale. So marketing at scale is now the table stakes. Now budget constraints are there as well. So you're starting to see, okay, I need to do more with less. Now the big question comes up is ROI. So I want to have AI. I want to have all these force multipliers. What do I got to do with the old? How do I handle that? How do I bring the new in and operationalize it? And if that's the case, I'm making a change. So I have to ask you, what's your view on the ROI of AI marketing, because this is a key component 'cause you've got scale factor here. You've got to force multiplier opportunity. How do you get that ROI on the table? >> I think that as you rightly said, it's table stakes. And I think the ROI of AI marketing starts with one very key simple premise that today some of the tools allow you to do things one at a time. So I can actually say, "can I run this campaign today?" And you can scramble your team, hustle your way, get everybody involved and run that campaign. And then tomorrow I'd say like, Hey, I looked at the results. Can I do this again? And they're like, oh, we just asked for all of us to get that done. How do I do it tomorrow? How do I do it next week? How do I do it for every single week for the rest of the year? That's where I think the AI marketing is essentially taking your insight, taking your creativity, and creating a platform and a tool that allows you to run this every single day. And that's agility at scale. That is not only a scale of the customer base, but scale across time. And that AI-based automation is the key ROI piece for a lot of AI marketing practitioners. So Forrester, for example, did a comprehensive total economic impact study with our customers. And what they found out was actually the 781% ROI that they reported in that particular report is based on three key factors. One is being able to do experiences that are intelligent at scale, day in and day out. So do your targeting, do your recommendations. Not just one day, but do it every single day. And don't hold back yourself on being able to do that. >> I think they got to get the return. They got to get the sales too. This is the numbers. >> That's right. They actually have real dollars, real numbers attached to it. They have a calculator. You can actually go in and plug your own numbers and get what you might expect from your existing customer base. The second is that once you have a unified platform like ours, the 10X marketer that we're talking about is actually able to do more. It's sometimes actually, it's kind of counterintuitive to think that a smaller team does more. But in reality, what we have seen, that is the case. When you actually have the right tools, the smaller teams actually achieve more. And that's the redundant operations, conflicting insights that go away into something more coherent and comprehensive. And that's the second insight that they found. And the third is just having reporting and all of the things in one place means that you can amplify it. You can amplify it across your paid media channels. You can amplify it across your promotions programs and other partnerships that you're running. >> That's the key thing about platforms that people don't understand is that you have a platform and it enables a lot of value. In this case, force multiplier value. It enables more value than you pay for it. But the key is it enables customers to do things without a line of code, meaning it's a platform. They're innovating on top of it. And that's, I think, where the ROI comes in and this leads me where the next question is. I wanted to ask you is, not to throw a wet blanket on the MarTech industry, but I got to think of when I hear marketing automation, I kind of think old. I think old, inadequate antiquated technologies. I think email blasting and just some boring stuff that just gets siloed or it's bespoke from something else. Are marketing automation tools created equal? Does something like, what you guys are doing with SmartHub? Change that, and can you just talk about that 'cause it's not going to go away. It's just another level that's going to be abstracted away under the coverage. >> Yeah, great question. Certainly, email marketing has been practiced for two or three decades now and in some form or another. I think we went from essentially what people call list-based marketing. I have a list, let me keep blasting the same message to everybody and then hopefully something will come out of it. A little bit more of saying, then they can, okay, maybe now I have CRM database and can I do database marketing, which they will call like, "Hey, Hi John. Hi Manyam", which is the first name. And that's all they think will get the customer excited about because you'll call them by name, which is certainly helpful, but not enough. I think now what we call like, the new age that we live in is that we call it graph-based marketing. And the way we materialize that is that every single user is interacting with a brand with their offerings. So that this interaction graph that's happening across millions of customers, across thousands of content articles, videos, shows, products, items, and that graph actually has much richer knowledge of what the customer wants than the first names or list-based ones. So I think the next evolution of marketing automation, even though the industry has been there a while, there is a step change in what can actually be done at scale. And which is taking that interaction graph and making that a part of the experience for the customer, and that's what we enable. That's why we do think of that as a big step change from how people are being practicing list-based marketing. And within that, certainly there is a relation of curve as to how people approach AI marketing and they are in a different spectrum. Some people are still at list-based marketing. Some people are database marketing. And hopefully will move them to this new interaction graph-based marketing. >> Yeah and I think the context is key. I like how you bring up the graph angle on this because the graph databases imply there's a lot of different optionality around what's happened contextually both over time and currently and it adds to it. Makes it smarter. It's not just siloed, just one dimensional. It feels like it's got a lot there. This is clearly I'm a big fan of and I think this is the way to go. As you get more personalization, you get more data. Graphic database makes a lot of sense. So I have to ask you, this is a really cutting edge value proposition, who are the primary buyers and users in an organization that you guys are working with? >> Yeah, great question. So we typically have CMO organizations approaching us with this problem and they usually talk to their CIO organizations, their counterparts, and the chief information officers have been investing in data fabrics, data lakes, data warehouses for the better part of last decade or two, and have some very cutting edge technology that goes into organizing all this data. But that doesn't still solve the problem of how do I take this data and make a meaningful, relevant, authentic experience for the customer. That's the CMO problem. And CMO are now challenge with creating product level experience with every interaction and that's where we coming. So the CMO are the buyers of our SmartHub CDP platform. And we're looking for consolidating hundreds of tools that they had in the past and making that one or two channel marketers. Actually, the 10X marketer that we talk about. And you need the right tool on top of your data lakes and data warehouses to be able to do that. So CMO are also the real drivers of using this technology. >> I think that also place the ROI equation around ROI and having that unified platform. Great call out there. I got to ask you the question here 'cause this comes up a lot and when I hear you talking, I think, okay, all the great stuff you guys have there. But if I'm a company, I want to make my core competencies mine. I don't really want to outsource or buy something that's going to be core to my business. But at the same time as market shifts, the business changes. And sometimes people don't even know what business they're in at the end of the day. And as it gets more complicated too, by the way. So the question comes up with companies and I can see this clearly, do I buy it? Do I build it? When it comes to AI because that's a core competency. Wait a minute, AI. I'm going to maybe buy some chatbot technology. That's not really AI, but it feels like AI, but I'm a company, I want to buy it or build it. That's a choice. What do you see there? 'Cause you guys have a very comprehensive platform. It's hard to replicate, imitates, inimitable. So what's your customers doing with respect buy and build? And where do they get the core competency? What do they get to have as a core competency? >> Fantastic. I think certainly, AI as it applies to at the organization level, I've seen this at my previous organization that I was part of, and there will be product and financial applications that are using AI for the service of that organization. So we do see, depending upon the size of the organization having in-house AI and data science teams. They are focused on these long term problems that they are doing as part of their product itself. Adjacent to that, the CMO organization gets some resources, but not certainly a lot. I think the CMO organization is usually challenged with the task, but not given the hundred people data science and engineering team to be able to go solve that. So what we see among our customer base is that they need agile platform to do most of the things that they need to do on a day to day basis, but augmented with what our in-house data science they have. So we are an extensible platform. What we have seen is that half of our customers use us solely for the AI needs. The other half certainly uses both AI modules that we provide and are actually augmented with things that they've already built. And we do not have a fight in that ring. But we do acknowledge and we do provide the right hooks for getting the data out of our system and bringing their AI back into our system. And we think that at the end of the day, if you want agility for the CMO, there should not be any barriers. >> It's like they're in the data business and that's the focus. So I think with what I hear you saying is that with your technology and platform, you're enabling to get them to be in the data business as fast as possible. >> That's right. >> Versus algorithm business, which they could add to over time. >> Certainly they could add to. But I think the bulk of competencies for the CMO are on the creative side. And certainly wrangling with data pipelines day in and day out and wondering what actually happened to a pipeline in the middle of the night is not probably what they would want to focus on. >> Not their core confidence. Yeah, I got that. >> That's right. >> You can do all the heavy lifting. I love that. I got to ask you on the Blueshift side on customer experience consumption. how can someone experience the product before buying? Is there a trial or POC? What's the scale and scope of operationalizing and getting the Blueshift value proposition in them? >> Yeah, great. So we actually recently released a fantastic way to experience our product. So if you go to our website, there's only one call-to-action saying, explore Blueshift. And if you click on that, without asking, anything other than your business email address, you're shown the full product. You're given a guided tour of all the possibilities. So you can actually experience what your marketing team would be doing in the product. And they call it Project Rover. We launched it very recently and we are seeing fantastic reception to that. I think a lot of times, as you said, there is that question mark of like, I have a marketing team that is already doing X, Y, Z. Now you are asking me to implement Blueshift. How would they actually experience the product? And now they can go in and experience the product. It's a great way to get the gist of the product in 10 clicks. Much more than going through any number of videos or articles. I think people really want to say, let me do those 10 clicks. And I know what impression that I can get from platform. So we do think that's a great way to experience the product and it's easily available from the main website. >> It's in the value proposition. It isn't always a straight line. And you got that technology. And I got to ask from between your experience with the customers that you're talking to, prospects, and customers, where do you see yourself winning deals on Customer Engagement, Made Simple because the word customer engagement's been around for a while, and it's become, I won't say cliche, but there's been different generational evolutions of technology that made that possible. Obviously, we're living in an era of high velocity Omni-Channel, a lot of data, the graph databases you mentioned are in there, big part of it. Where are you winning deals? Where are customers pain points where you are solving that specifically? >> Yeah, great question. So the organizations that come to us usually have one of the dimensions of either they have offering complexity, which is what catalog of content or videos or items do they offer to the customers. And on the data complexity on the other side is to what the scale of customer base that I usually target. And that problem has not gone away. I think the customer engagement, even though has been around for a while, the problem of engaging those customers at scale hasn't gone away and it only is getting harder and harder and organizations that have, especially on what we call the business-to-consumer side where the bulk of what marketing organizations in a B2C segments are doing. I have tens to millions of customers and how do I engage them day in and day out. And I think that all that problem is only getting harder because consumer preferences keeps shifting all the time. >> And where's your sweet spot for your customer? What size? Can you just share the target organization? Is it medium enterprise, large B2C, B2B2C? What's the focus area? >> Yeah, great question. So we have seen like startups that are in Silicon Valley. I have now half a million monthly active users, how do I actually engage them to customers and clients like LendingTree and PayPal and Discovery and BBC who have been in the business for multiple decades, have tens of millions of customers that they're engaging with. So that's kind of our sweet spot. We are certainly not maybe for small shop with maybe a hundred plus customers. But as you reach the scale of tens of thousands of customers, you start seeing this problem. And then you start to look out for solutions that are beyond, especially list-based marketing and email blast. >> So as the scale, you can dial up and down, but you have to have some enough scale to get the data pattern. >> That's right. >> If I can connect the dots there. >> I would probably say, looking at a hundred thousand or more monthly active customer base, and then you're trying to ramp up your own growth based on what you're learning and to engage those customers. >> It's like a bulldozer. You need the heavy equipment. Great conversation. For the last minute we have here Manyam, give you a plug for the company. What's going on? What are you guys doing? What's new? Give some success stories, your latest achievements. Take a minute to give a plug for the company. >> Yeah, great. We have been recognized by Deloitte as the fastest growth startup two years in a row and continuing to be on that streak. We have released currently integrations with AWS partners and Snowflake partners and data lake partners that allow implementing Blueshift a much streamlined experience with bidirectional integrations. We have now hundred plus data connectors and data integrations in our system and that takes care of many of our needs. And now, I think organizations that have been budget constraint and are trying to achieve a lot with a small team are actually going to look at these solutions and say, "Can I get there?" and "Can I become that 10X marketing organization? And as you have said, agility at scale is very, very hard to achieve. Being able to take your marketing team and achieve 10X requires the right platform and the right solution. We are ready for it. >> And every company's in the data business that's the asset. You guys make that sing for them. It's good stuff. Love the 10X. Love the scale. Manyam Mallela, thanks for coming on. Co-founder, Head of AI at Blueshift. This is the AWS Startup Showcase season two, episode three of the ongoing series covering the exciting startups from the AWS ecosystem. I'm John Furrier, your host. Thanks for watching. >> Thank you, John. (upbeat music)

(upbeat music) >> Hello, everyone. Welcome to the theCUBE's presentation of the AWS Startup Showcase. The theme, this showcase is MarTech, the emerging cloud scale customer experiences. Season two of episode three, the ongoing series covering the startups, the hot startups, talking about analytics, data, all things MarTech. I'm your host, John Furrier, here joined by Christian Wiklund, founder and CEO of unitQ here, talk about harnessing the power of user feedback to empower marketing. Thanks for joining us today. >> Thank you so much, John. Happy to be here. >> In these new shifts in the market, when you got cloud scale, open source software is completely changing the software business. We know that. There's no longer a software category. It's cloud, integration, data. That's the new normal. That's the new category, right? So as companies are building their products, and want to do a good job, it used to be, you send out surveys, you try to get the product market fit. And if you were smart, you got it right the third, fourth, 10th time. If you were lucky, like some companies, you get it right the first time. But the holy grail is to get it right the first time. And now, this new data acquisition opportunities that you guys in the middle of that can tap customers or prospects or end users to get data before things are shipped, or built, or to iterate on products. This is the customer feedback loop or data, voice of the customer journey. It's a gold mine. And it's you guys, it's your secret weapon. Take us through what this is about now. I mean, it's not just surveys. What's different? >> So yeah, if we go back to why are we building unitQ? Which is we want to build a quality company. Which is basically, how do we enable other companies to build higher quality experiences by tapping into all of the existing data assets? And the one we are in particularly excited about is user feedback. So me and my co-founder, Nik, and we're doing now the second company together. We spent 14 years. So we're like an old married couple. We accept each other, and we don't fight anymore, which is great. We did a consumer company called Skout, which was sold five years ago. And Skout was kind of early in the whole mobile first. I guess, we were actually mobile first company. And when we launched this one, we immediately had the entire world as our marketplace, right? Like any modern company. We launch a product, we have support for many languages. It's multiple platforms. We have Android, iOS, web, big screens, small screens, and that brings some complexities as it relates to staying on top of the quality of the experience because how do I test everything? >> John: Yeah. >> Pre-production. How do I make sure that our Polish Android users are having a good day? And we found at Skout, personally, like I could discover million dollar bugs by just drinking coffee and reading feedback. And we're like, "Well, there's got to be a better way to actually harness the end user feedback. That they are leaving in so many different places." So, you know what, what unitQ does is that we basically aggregate all different sources of user feedback, which can be app store reviews, Reddit posts, Tweets, comments on your Facebook ads. It can be better Business Bureau Reports. We don't like to get to many of those, of course. But really, anything on the public domain that mentions or refers to your product, we want to ingest that data in this machine, and then all the private sources. So you probably have a support system deployed, a Zendesk, or an Intercom. You might have a chatbot like an Ada, or and so forth. And your end user is going to leave a lot of feedback there as well. So we take all of these channels, plug it into the machine, and then we're able to take this qualitative data. Which and I actually think like, when an end user leaves a piece of feedback, it's an act of love. They took time out of the day, and they're going to tell you, "Hey, this is not working for me," or, "Hey, this is working for me," and they're giving you feedback. But how do we package these very messy, multi-channel, multiple languages, all over the place data? How can we distill it into something that's quantifiable? Because I want to be able to monitor these different signals. So I want to turn user feedback into time series. 'Cause with time series, I can now treat this the same way as Datadog treats machine logs. I want to be able to see anomalies, and I want to know when something breaks. So what we do here is that we break down your data in something called quality monitors, which is basically machine learning models that can aggregate the same type of feedback data in this very fine grained and discrete buckets. And we deploy up to a thousand of these quality monitors per product. And so we can get down to the root cause. Let's say, passive reset link is not working. And it's in that root cause, the granularity that we see that companies take action on the data. And I think historically, there has been like the workflow between marketing and support, and engineering and product has been a bit broken. They've been siloed from a data perspective. They've been siloed from a workflow perspective, where support will get a bunch of tickets around some issue in production. And they're trained to copy and paste some examples, and throw it over the wall, file a Jira ticket, and then they don't know what happens. So what we see with the platform we built is that these teams are able to rally around the single source of troop or like, yes, passive recent link seems to have broken. This is not a user error. It's not a fix later, or I can't reproduce. We're looking at the data, and yes, something broke. We need to fix it. >> I mean, the data silos a huge issue. Different channels, omnichannel. Now, there's more and more channels that people are talking in. So that's huge. I want to get to that. But also, you said that it's a labor of love to leave a comment or a feedback. But also, I remember from my early days, breaking into the business at IBM and Hewlett-Packard, where I worked. People who complain are the most loyal customers, if you service them. So it's complaints. >> Christian: Yeah. >> It's leaving feedback. And then, there's also reading between the lines with app errors or potentially what's going on under the covers that people may not be complaining about, but they're leaving maybe gesture data or some sort of digital trail. >> Yeah. >> So this is the confluence of the multitude of data sources. And then you got the siloed locations. >> Siloed locations. >> It's complicated problem. >> It's very complicated. And when you think about, so I started, I came to Bay Area in 2005. My dream was to be a quant analyst on Wall Street, and I ended up in QA at VMware. So I started at VMware in Palo Alto, and didn't have a driver's license. I had to bike around, which was super exciting. And we were shipping box software, right? This was literally a box with a DVD that's been burned, and if that DVD had bugs in it, guess what it'll be very costly to then have to ship out, and everything. So I love the VMware example because the test cycles were long and brutal. It was like a six month deal to get through all these different cases, and they couldn't be any bugs. But then as the industry moved into the cloud, CI/CD, ship at will. And if you look at the modern company, you'll have at least 20 plus integrations into your product. Analytics, add that's the case, authentication, that's the case, and so forth. And these integrations, they morph, and they break. And you have connectivity issues. Is your product working as well on Caltrain, when you're driving up and down, versus wifi? You have language specific bugs that happen. Android is also quite a fragmented market. The binary may not perform as well on that device, or is that device. So how do we make sure that we test everything before we ship? The answer is, we can't. There's no company today that can test everything before the ship. In particular, in consumer. And the epiphany we had at our last company, Skout, was that, "Hey, wait a minute. The end user, they're testing every configuration." They're sitting on the latest device, the oldest device. They're sitting on Japanese language, on Swedish language. >> John: Yeah. >> They are in different code paths because our product executed differently, depending on if you were a paid user, or a freemium user, or if you were certain demographical data. There's so many ways that you would have to test. And PagerDuty actually had a study they came out with recently, where they said 51% of all end user impacting issues are discovered first by the end user, when they serve with a bunch of customers. And again, like the cool part is, they will tell you what's not working. So now, how do we tap into that? >> Yeah. >> So what I'd like to say is, "Hey, your end user is like your ultimate test group, and unitQ is the layer that converts them into your extended test team." Now, the signals they're producing, it's making it through to the different teams in the organization. >> I think that's the script that you guys are flipping. If I could just interject. Because to me, when I hear you talking, I hear, "Okay, you're letting the customers be an input into the product development process." And there's many different pipelines of that development. And that could be whether you're iterating, or geography, releases, all kinds of different pipelines to get to the market. But in the old days, it was like just customer satisfaction. Complain in a call center. >> Christian: Yeah. >> Or I'm complaining, how do I get support? Nothing made itself into the product improvement, except for slow moving, waterfall-based processes. And then, maybe six months later, a small tweak could be improved. >> Yes. >> Here, you're taking direct input from collective intelligence. Okay. >> Is that have input and on timing is very important here, right? So how do you know if the product is working as it should in all these different flavors and configurations right now? How do you know if it's working well? And how do you know if you're improving or not improving over time? And I think the industry, what can we look at, as far as when it relates to quality? So I can look at star ratings, right? So what's the star rating in the app store? Well, star ratings, that's an average over time. So that's something that you may have a lot of issues in production today, and you're going to get dinged on star ratings over the next few months. And then, it brings down the score. NPS is another one, where we're not going to run NPS surveys every day. We're going to run it once a quarter, maybe once a month, if we're really, really aggressive. That's also a snapshot in time. And we need to have the finger on the pulse of product quality today. I need to know if this release is good or not good. I need to know if anything broke. And I think that real time aspect, what we see as stuff sort of bubbles up the stack, and not into production, we see up to a 50% reduction in time to fix these end user impacting issues. And I think, we also need to appreciate when someone takes time out of the day to write an app review, or email support, or write that Reddit post, it's pretty serious. It's not going to be like, "Oh, I don't like the shade of blue on this button." It's going to be something like, "I got double billed," or "Hey, someone took over my account," or, "I can't reset my password anymore. The CAPTCHA, I'm solving it, but I can't get through to the next phase." And we see a lot of these trajectory impacting bugs and quality issues in these work, these flows in the product that you're not testing every day. So if you work at Snapchat, your employees probably going to use Snapchat every day. Are they going to sign up every day? No. Are they going to do passive reset every day? No. And these things are very hard to instrument, lower in the stack. >> Yeah, I think this is, and again, back to these big problems. It's smoke before fire, and you're essentially seeing it early with your process. Can you give an example of how this new focus or new mindset of user feedback data can help customers increase their experience? Can you give some examples, 'cause folks watching and be like, "Okay, I love this value. Sell me on this idea, I'm sold. Okay, I want to tap into my prospects, and my customers, my end users to help me improve my product." 'Cause again, we can measure everything now with data. >> Yeah. We can measure everything. we can even measure quality these days. So when we started this company, I went out to talk to a bunch of friends, who are entrepreneurs, and VCs, and board members, and I asked them this very simple question. So in your board meetings, or on all hands, how do you talk about quality of the product? Do you have a metric? And everyone said, no. Okay. So are you data driven company? Yes, we're very data driven. >> John: Yeah. Go data driven. >> But you're not really sure if quality, how do you compare against competition? Are you doing as good as them, worse, better? Are you improving over time, and how do you measure it? And they're like, "Well, it's kind of like a blind spot of the company." And then you ask, "Well, do you think quality of experience is important?" And they say, "Yeah." "Well, why?" "Well, top of fund and growth. Higher quality products going to spread faster organically, we're going to make better store ratings. We're going to have the storefronts going to look better." And of course, more importantly, they said the different conversion cycles in the product box itself. That if you have bugs and friction, or an interface that's hard to use, then the inputs, the signups, it's not going to convert as well. So you're going to get dinged on retention, engagement, conversion to paid, and so forth. And that's what we've seen with the companies we work with. It is that poor quality acts as a filter function for the entire business, if you're a product led company. So if you think about product led company, where the product is really the centerpiece. And if it performs really, really well, then it allows you to hire more engineers, you can spend more on marketing. Everything is fed by this product at them in the middle, and then quality can make that thing perform worse or better. And we developed a metric actually called the unitQ Score. So if you go to our website, unitq.com, we have indexed the 5,000 largest apps in the world. And we're able to then, on a daily basis, update the score. Because the score is not something you do once a month or once a quarter. It's something that changes continuously. So now, you can get a score between zero and 100. If you get the score 100, that means that our AI doesn't find any quality issues reported in that data set. And if your score is 90, that means that 10% will be a quality issue. So now you can do a lot of fun stuff. You can start benchmarking against competition. So you can see, "Well, I'm Spotify. How do I rank against Deezer, or SoundCloud, or others in my space?" And what we've seen is that as the score goes up, we see this real big impact on KPI, such as conversion, organic growth, retention, ultimately, revenue, right? And so that was very satisfying for us, when we launched it. quality actually still really, really matters. >> Yeah. >> And I think we all agree at test, but how do we make a science out of it? And that's so what we've done. And when we were very lucky early on to get some incredible brands that we work with. So Pinterest is a big customer of ours. We have Spotify. We just signed new bank, Chime. So like we even signed BetterHelp recently, and the world's largest Bible app. So when you look at the types of businesses that we work with, it's truly a universal, very broad field, where if you have a digital exhaust or feedback, I can guarantee you, there are insights in there that are being neglected. >> John: So Chris, I got to. >> So these manual workflows. Yeah, please go ahead. >> I got to ask you, because this is a really great example of this new shift, right? The new shift of leveraging data, flipping the script. Everything's flipping the script here, right? >> Yeah. >> So you're talking about, what the value proposition is? "Hey, board example's a good one. How do you measure quality? There's no KPI for that." So it's almost category creating in its own way. In that, this net new things, it's okay to be new, it's just new. So the question is, if I'm a customer, I buy it. I can see my product teams engaging with this. I can see how it can changes my marketing, and customer experience teams. How do I operationalize this? Okay. So what do I do? So do I reorganize my marketing team? So take me through the impact to the customer that you're seeing. What are they resonating towards? Obviously, getting that data is key, and that's holy gray, we all know that. But what do I got to do to change my environment? What's my operationalization piece of it? >> Yeah, and that's one of the coolest parts I think, and that is, let's start with your user base. We're not going to ask your users to ask your users to do something differently. They're already producing this data every day. They are tweeting about it. They're putting in app produce. They're emailing support. They're engaging with your support chatbot. They're already doing it. And every day that you're not leveraging that data, the data that was produced today is less valuable tomorrow. And in 30 days, I would argue, it's probably useless. >> John: Unless it's same guy commenting. >> Yeah. (Christian and John laughing) The first, we need to make everyone understand. Well, yeah, the data is there, and we don't need to do anything differently with the end user. And then, what we do is we ask the customer to tell us, "Where should we listen in the public domain? So do you want the Reddit post, the Trustpilot? What channels should we listen to?" And then, our machine basically starts ingesting that data. So we have integration with all these different sites. And then, to get access to private data, it'll be, if you're on Zendesk, you have to issue a Zendesk token, right? So you don't need any engineering hours, except your IT person will have to grant us access to the data source. And then, when we go live. We basically build up this taxonomy with the customers. So we don't we don't want to try and impose our view of the world, of how do you describe the product with these buckets, these quality monitors? So we work with the company to then build out this taxonomy. So it's almost like a bespoke solution that we can bootstrap with previous work we've done, where you don't have these very, very fine buckets of where stuff could go wrong. And then what we do is there are different ways to hook this into the workflow. So one is just to use our products. It's a SaaS product as anything else. So you log in, and you can then get this overview of how is quality trending in different markets, on different platforms, different languages, and what is impacting them? What is driving this unitQ Score that's not good enough? And all of these different signals, we can then hook into Jira for instance. We have a Jira integration. We have a PagerDuty integration. We can wake up engineers if certain things break. We also tag tickets in your support system, which is actually quite cool. Where, let's say, you have 200 people, who wrote into support, saying, "I got double billed on Android." It turns out, there are some bugs that double billed them. Well, now we can tag all of these users in Zendesk, and then the support team can then reach out to that segment of users and say, "Hey, we heard that you had this bug with double billing. We're so sorry. We're working on it." And then when we push fix, we can then email the same group again, and maybe give them a little gift card or something, for the thank you. So you can have, even big companies can have that small company experience. So, so it's groups that use us, like at Pinterest, we have 800 accounts. So it's really through marketing has vested interest because they want to know what is impacting the end user. Because brand and product, the lines are basically gone, right? >> John: Yeah. >> So if the product is not working, then my spend into this machine is going to be less efficient. The reputation of our company is going to be worse. And the challenge for marketers before unitQ was, how do I engage with engineering and product? I'm dealing with anecdotal data, and my own experience of like, "Hey, I've never seen these type of complaints before. I think something is going on." >> John: Yeah. >> And then engineering will be like, "Ah, you know, well, I have 5,000 bugs in Jira. Why does this one matter? When did it start? Is this a growing issue?" >> John: You have to replicate the problem, right? >> Replicate it then. >> And then it goes on and on and on. >> And a lot of times, reproducing bugs, it's really hard because it works on my device. Because you don't sit on that device that it happened on. >> Yup. >> So now, when marketing can come with indisputable data, and say, "Hey, something broke here." And we see the same with support. Product engineering, of course, for them, we talk about, "Hey, listen, you you've invested a lot in observability of your stack, haven't you?" "Yeah, yeah, yeah." "So you have a Datadog in the bottom?" "Absolutely." "And you have an APP D on the client?" "Absolutely." "Well, what about the last mile? How the product manifests itself? Shouldn't you monitor that as well using machines?" They're like, "Yeah, that'd be really cool." (John laughs) And we see this. There's no way to instrument everything, lowering the stack to capture these bugs that leak out. So it resonates really well there. And even for the engineers who's going to fix it. >> Yeah. >> I call it like empathy data. >> Yup. >> Where I get assigned a bug to fix. Well, now, I can read all the feedback. I can actually see, and I can see the feedback coming in. >> Yeah. >> Oh, there's users out there, suffering from this bug. And then when I fix it and I deploy the fix, and I see the trend go down to zero, and then I can celebrate it. So that whole feedback loop is (indistinct). >> And that's real time. It's usually missed too. This is the power of user feedback. You guys got a great product, unitQ. Great to have you on. Founder and CEO, Christian Wiklund. Thanks for coming on and sharing, and showcase. >> Thank you, John. For the last 30 seconds, the minute we have left, put a plug in for the company. What are you guys looking for? Give a quick pitch for the company, real quick, for the folks out there. Looking for more people, funding status, number of employees. Give a quick plug. >> Yes. So we raised our A Round from Google, and then we raised our B from Excel that we closed late last year. So we're not raising money. We are hiring across go-to-markets, engineering. And we love to work with people, who are passionate about quality and data. We're always, of course, looking for customers, who are interested in upping their game. And hey, listen, competing with features is really hard because you can copy features very quickly. Competing with content. Content is commodity. You're going to get the same movies more or less on all these different providers. And competing on price, we're not willing to do. You're going to pay 10 bucks a month for music. So how do you compete today? And if your competitor has a better fine tuned piano than your competitor will have better efficiencies, and they're going to retain customers and users better. And you don't want to lose on quality because it is actually a deterministic and fixable problem. So yeah, come talk to us if you want to up the game there. >> Great stuff. The iteration lean startup model, some say took craft out of building the product. But this is now bringing the craftsmanship into the product cycle, when you can get that data from customers and users. >> Yeah. >> Who are going to be happy that you fixed it, that you're listening. >> Yeah. >> And that the product got better. So it's a flywheel of loyalty, quality, brand, all off you can figure it out. It's the holy grail. >> I think it is. It's a gold mine. And every day you're not leveraging this assets, your use of feedback that's there, is a missed opportunity. >> Christian, thanks so much for coming on. Congratulations to you and your startup. You guys back together. The band is back together, up into the right, doing well. >> Yeah. We we'll check in with you later. Thanks for coming on this showcase. Appreciate it. >> Thank you, John. Appreciate it very much. >> Okay. AWS Startup Showcase. This is season two, episode three, the ongoing series. This one's about MarTech, cloud experiences are scaling. I'm John Furrier, your host. Thanks for watching. (upbeat music)

>> Hey everyone, welcome to theCUBE's coverage of the AWS startup showcase. This is season two, episode three of our ongoing series featuring AWS and its big ecosystem of partners. This particular season is focused on MarTech, emerging cloud scale customer experiences. I'm your host, Lisa Martin, and I'm pleased to be joined by James Fang, the VP of product marketing at mparticle. James, welcome to the program. Great to have you on. >> Thanks for having me. >> Tell us a little bit about mparticle, what is it that you guys do? >> Sure, so we're mparticle, we were founded in 2013, and essentially we are a customer data platform. What we do is we help brands collect and organize their data. And their data could be coming from web apps, mobile apps, existing data sources like data warehouses, data lakes, et cetera. And we help them help them organize it in a way where they're able to activate that data, whether it's to analyze it further, to gather insights or to target them with relevant messaging, relevant offers. >> What were some of the gaps in the market back then as you mentioned 2013, or even now, that mparticle is really resolving so that customers can really maximize the value of their customer's data. >> Yeah. So the idea of data has actually been around for a while, and you may have heard the buzzword 360 degree view of the customer. The problem is no one has really been actually been able to, to achieve it. And it's actually, some of the leading analysts have called it a myth. Like it's a forever ending kind of cycle. But where we've kind of gone is, first of all customer expectations have really just inflated over the years, right? And part of that was accelerated due to COVID, and the transformation we saw in the last two years, right. Everyone used to, you know, have maybe a digital footprint, as complimentary perhaps to their physical footprint. Nowadays brands are thinking digital first, for obvious reasons. And the data landscape has gotten a lot more complex, right? Brands have multiple experiences, on different screens, right? And, but from the consumer perspective, they want a complete end to end experience, no matter how you're engaging with the brand. And in order to, for a brand to deliver that experience they have to know, how the customers interacted before in each of those channels, and be able to respond in as real time as possible, to those experiences. >> So I can start an interaction on my iPad, maybe carry it through or continue it on my laptop, go to my phone. And you're right, as a, as a consumer, I want the experience across all of those different media to be seamless, to be the same, to be relevant. You talk about the customer 360, as a marketer I know that term well. It's something that so many companies use, interesting that you point out that it's really been, largely until companies like mparticle, a myth. It's one of those things though, that everybody wants to achieve. Whether we're talking about healthcare organization, a retailer, to be able to know everything about a customer so that they can deliver what's increasingly demanded that personalized, relevant experience. How does mparticle fill some of the gaps that have been there in customer 360? And do you say, Hey, we actually deliver a customer 360. >> Yeah, absolutely. So, so the reason it's been a myth is for the most part, data has been- exists either in silos, or it's kind of locked behind this black box that the central data engineering team or sometimes traditionally referred to as IT, has control over, right? So brands are collecting all sorts of data. They have really smart people working on and analyzing it. You know, being able to run data science models, predictive models on it, but the, the marketers and the people who want to draw insights on it are asking how do I get it in, in my hands? So I can use that data for relevant targeting messaging. And that's exactly what mparticle does. We democratize access to that data, by making it accessible in the very tools that the actual business users are are working in. And we do that in real time, you don't have to wait for days to get access to data. And the marketers can even self-service, they're able to for example, build audiences or build computed insights, such as, you know, average order value of a customer within the tool themselves. The other main, the other main thing that mparticle does, is we ensure the quality of that data. We know that activation is only as as good, when you can trust that data, right? When there's no mismatching, you know, first name last names, identities that are duplicated. And so we put a lot of effort, not only in the identity resolution component of our product but also being able to ensure that the consistency of that data when it's being collected meets the standard that you need. >> So give us a, a picture, kind of a topology of a, of a customer data platform. And what are some of the key components that it contains, then I kind of want to get into some of the use cases. >> Yeah. So at, at a core, a lot of customer data platforms look similar. They're responsible first of all for the collection of data, right? And again, that could be from web mobile sources, as well as existing data sources, as well as third party apps, right? For example, you may have e-commerce data in a Shopify, right. Or you may have, you know, a computer model from a, from a warehouse. And then the next thing is to kind of organize it somehow, right? And the most common way to do that is to unify it, using identity resolution into this idea of customer profiles, right. So I can look up everything that Lisa or James has done, their whole historical record. And then the third thing is to be able to kind of be able to draw some insights from that, whether to be able to build an audience membership on top of that, build a predictive model, such as the churn risk model or lifetime value of that customer. And finally is being able to activate that data, so you'll be able to push that data again, to those relevant downstream systems where the business users are actually using that data to, to do their targeting, or to do more interesting things with it. >> So for example, if I go to the next Warrior's game, which I predict they're going to win, and I have like a mobile app of the stadium or the team, how, and I and I'm a season ticket holder, how can a customer data platform give me that personalized experience and help to, yeah, I'd love to kind of get it in that perspective. >> Yeah. So first of all, again, in this modern day and age consumers are engaging with brands from multiple devices, and their attention span, frankly, isn't that long. So I may start off my day, you know, downloading the official warriors app, right. And I may be, you know browsing from my mobile phone, but I could get distracted. I've got to go join a meeting at work, drop off my kids or whatever, right? But later in the day I had in my mind, I may be interested in purchasing tickets or buying that warriors Jersey. So I may return to the website, or even the physical store, right. If, if I happen to be in the area and what the customer data platform is doing in the background, is associating and connecting all those online and offline touchpoints, to that user profile. And then now, I have a mar- so let's say I'm a marker for the golden state warriors. And I see that, you know, this particular user has looked at my website even added to their cart, you know, warriors Jersey. I'm now able to say, Hey, here's a $5 promotional coupon. Also, here's a special, limited edition. We just won, you know, the, the Western conference finals. And you can pre-book, you know, the, you know the warriors championships Jersey, cross your fingers, and target that particular user with that promotion. And it's much more likely because we have that contextual data that that user's going to convert, than just blasting them on a Facebook or something like that. >> Right. Which all of us these days are getting less and less patient with, Is those, those broad blasts through social media and things like that. That was, I love that example. That was a great example. You talked about timing. One of the things I think that we've learned that's in very short supply, in the last couple of years is people's patience and tolerance. We now want things in nanoseconds. So, the ability to glean insights from data and act on it in real time is no longer really a nice to have that's really table stakes for any type of organization. Talk to us about how mparticle facilitates that real time data, from an insights perspective and from an activation standpoint. >> Yeah. You bring up a good point. And this is actually one of the core differentiators of mparticle compared to the other CDPs is that, our architecture from the ground up is built for real time. And the way we do that is, we use essentially a real time streaming architecture backend. Essentially all the data points that we collect and send to those downstream destinations, that happens in milliseconds, right? So the moment that that user, again, like clicks a button or adds something to their shopping cart, or even abandons that shopping cart, that downstream tool, whether it's a marketer, whether it's a business analyst looking at that data for intelligence, they get that data within milliseconds. And our audience computations also happens within seconds. So again, if you're, if you have a targeted list for a targeted campaign, those updates happen in real time. >> You gave an- you ran with the Warrior's example that I threw at you, which I love, absolutely. Talk to me. You must have though, a favorite cu- real world customer example of mparticle's that you think really articulates the value to organizations, whether it's to marketers operators and has some nice, tangible business outcomes. Share with me if you will, a favorite customer story. >> Yeah, definitely one of mine and probably one of the- our most well known's is we were actually behind the scenes of the Whopper jr campaign. So a couple of years ago, Burger King ran this really creative ad where the, effectively their goal was to get their mobile app out, as well as to train, you know, all of us back before COVID days, how to order on our mobile devices and to do things like curbside checkout. None of us really knew how to do that, right. And there was a challenge of course that, no one wants to download another app, right? And most apps get downloaded and get deleted right out away. So they ran this really creative promotion where, if you drove towards a McDonald's, they would actually fire off a text message saying, Hey, how about a Whopper for 99 cents instead? And you would, you would, you would receive a text message personalized just for you. And you'd be able to redeem that at any burger king location. So we were kind of the core infrastructure plumbing the geofencing location data, to partner of ours called radar, which handles you geofencing, and then send it back to a marketing orchestration vendor to be able to fire that targeted message. >> Very cool. I, I, now I'm hungry. You, but there's a fine line there between knowing that, okay, Lisa's driving towards McDonald's let's, you know, target her with an ad for a whopper, in privacy. How do you guys help organizations in any industry balance that? Cause we're seeing more and more privacy regulations popping up all over the world, trying to give consumers the ability to protect either the right to forget about me or don't use my data. >> Yeah. Great question. So the first way I want to respond to that is, mparticle's really at the core of helping brands build their own first party data foundation. And what we mean by that is traditionally, the way that brands have approached marketing is reliant very heavily on second and third party data, right? And most that second-third party data is from the large walled gardens, such as like a Facebook or a TikTok or a Snapchat, right? They're they're literally just saying, Hey find someone that is going to, you know fit our target profile. And that data is from people, all their activity on those apps. But with the first party data strategy, because the brand owns that data, we- we can guarantee that or the brands can guarantee to their customers it's ethically sourced, meaning it's from their consent. And we also help brands have governance policies. So for example, if the user has said, Hey you're allowed to collect my data, because obviously you want to run your business better, but I don't want any my information sold, right? That's something that California recently passed, with CPRA. Then brands can use mparticle data privacy controls to say, Hey, you can pass this data on to their warehouses and analytics platforms, but don't pass it to a platform like Facebook, which potentially could resell that data. >> Got it, Okay. So you really help put sort of the, the reigns on and allow those customers to make those decisions, which I know the mass community appreciates. I do want to talk about data quality. You talked about that a little bit, you know, and and data is the lifeblood of an organization, if it can really extract value from it and act on it. But how do you help organizations maintain the quality of data so that what they can do, is actually deliver what the end user customer, whether it's a somebody buying something on a, on a eCommerce site or or, a patient at a hospital, get what they need. >> Yeah. So on the data quality front, first of all I want to highlight kind of our strengths and differentiation in identity resolution. So we, we run a completely deterministic algorithm, but it's actually fully customizable by the customer depending on their needs. So for a lot of other customer data providers, platform providers out there, they do offer identity resolution, but it's almost like a black box. You don't know what happens. And they could be doing a lot of fuzzy matching, right. Which is, you know, probabilistic or predictive. And the problem with that is, let's say, you know, Lisa your email changed over the years and CDP platform may match you with someone that's completely not you. And now all of a sudden you're getting ads that completely don't fit you, or worse yet that brand is violating privacy laws, because your personal data is is being used to target another user, which which obviously should not, should not happen, right? So because we're giving our customers complete control, it's not a black box, it's transparent. And they have the ability to customize it, such as they can specify what identifiers matter more to them, whether they want to match on email address first. They might've drawn on a more high confidence identifier like a, a hash credit card number or even a customer ID. They have that choice. The second part about ensuring data quality is we act actually built in schema management. So as those events are being collected you could say that, for example, when when it's a add to cart event, I require the item color. I require the size. Let's say it's a fashion apparel. I require the size of it and the type of apparel, right? And if, if data comes in with missing fields, or perhaps with fields that don't match the expectation, let's say you're expecting small, medium, large and you get a Q, you know Q is meaningless data, right? We can then enforce that and flag that as a data quality violation and brands can complete correct that mistake to make sure again, all the data that's flowing through is, is of value to them. >> That's the most important part is, is to make sure that the data has value to the organization, and of course value to whoever it is on the other side, the, the end user side. Where should customers start, in terms of working with you guys, do you recommend customers buy an all in one marketing suite? The best, you know, build a tech stack of best of breed? What are some of those things that you recommend for folks who are going, all right, We, maybe we have a CDP it's been under delivering. We can't really deliver that customer 360, mparticle, help us out. >> Yeah, absolutely. Well, the best part about mparticle is you can kind of deploy it in phases, right. So if you're coming from a world where you've deployed a, all in one marketing suite, like a sales force in Adobe, but you're looking to maybe modernize pieces of a platform mparticle can absolutely help with that initial step. So let again, let's say all you want to do is modernize your event collection. Well, we can absolutely, as a first step, for example, you can instrument us. You can collect all your data from your web and mobile apps in real time, and we can pipe to your existing, you know Adobe campaign manager, Salesforce, marketing cloud. And later down the line, let's say, you say I want to, you know, modernize my analytics platform. I'm tired of using Adobe analytics. You can swap that out, right again with an mparticle place, a marketer can or essentially any business user can flip the switch. And within the mparticle interface, simply disconnect their existing tool and connect a new tool with a couple of button clicks and bam, the data's now flowing into the new tool. So it mparticle really, because we kind of sit in the middle of all these tools and we have over 300 productized prebuilt integrations allows you to move away from kind of a locked in, you know a strategy where you're committed to a vendor a hundred percent to more of a best of breed, agile strategy. >> And where can customers that are interested, go what's your good and market strategy? How does that involve AWS? Where can folks go and actually get and test out this technology? >> Yeah. So first of all, we are we are AWS, a preferred partner. and we have a couple of productized integrations with AWS. The most obvious one is for example, being able to just export data to AWS, whether it's Redshift or an S3 or a kinesis stream, but we also have productized integrations with AWS, personalized. For example, you can take events, feed em to personalize and personalize will come up with the next best kind of content recommendation or the next best offer available for the customer. And mparticle can ingest that data back and you can use that for personalized targeting. In fact, Amazon personalize is what amazon.com themselves use to populate the recommended for use section on their page. So brands could essentially do the same. They could have a recommended for you carousel using Amazon technology but using mparticle to move the data back and forth to, to populate that. And then on top of that very, very soon we'll be also launching a marketplace kind of entry. So if you are a AWS customer and you have credits left over or you just want to transact through AWS, then you'll have that option available as well. >> Coming soon to the AWS marketplace. James, thank you so much for joining me talking about mparticle, how you guys are really revolutionizing the customer data platform and allowing organizations and many industries to really extract value from customer data and use it wisely. We appreciate your insights and your time. >> Thank you very much, Lisa >> For James Fang, I'm Lisa Martin. You're watching theCube's coverage of the AWS startup showcase season three, season two episode three, leave it right here for more great coverage on theCube, the leader in live tech coverage.

(introductory riff) >> Hey everyone. Welcome to theCUBE's presentation of the "AWS Startup Showcase." This is Season 2, Episode 3 of our ongoing series that features great partners in the massive AWS partner ecosystem. This series is focused on, "MarTech, Emerging Cloud-Scale Customer Experiences." I'm Lisa Martin, and I've got two guests here with me to talk about this. Please welcome Daisy Urfer, Cloud Alliance Sales Director at Algolia, and Jason Lang, the Head of Product for Apply Digital. These folks are here to talk with us today about how Algolia's Search and Discovery enables customers to create dynamic realtime user experiences for those oh so demanding customers. Daisy and Jason, it's great to have you on the program. >> Great to be here. >> Thanks for having us. >> Daisy, we're going to go ahead and start with you. Give the audience an overview of Algolia, what you guys do, when you were founded, what some of the gaps were in the market that your founders saw and fixed? >> Sure. It's actually a really fun story. We were founded in 2012. We are an API first SaaS solution for Search and Discovery, but our founders actually started off with a search tool for mobile platforms, so just for your phone and it quickly expanded, we recognize the need across the market. It's been a really fun place to grow the business. And we have 11,000 customers today and growing every day, with 30 billion searches a week. So we do a lot of business, it's fun. >> Lisa: 30 billion searches a week and I saw some great customer brands, Locost, NBC Universal, you mentioned over 11,000. Talk to me a little bit about some of the technologies, I see that you have a search product, you have a recommendation product. What are some of those key capabilities that the products deliver? 'Cause as we know, as users, when we're searching for something, we expect it to be incredibly fast. >> Sure. Yeah. What's fun about Algolia is we are actually the second largest search engine on the internet today to Google. So we are right below the guy who's made search of their verb. So we really provide an overall search strategy. We provide a dashboard for our end users so they can provide the best results to their customers and what their customers see. Customers want to see everything from Recommend, which is our recommended engine. So when you search for that dress, it shows you the frequently bought together shoes that match, things like that, to things like promoted items and what's missing in the search results. So we do that with a different algorithm today. Most in the industry rank and they'll stack what you would want to see. We do kind of a pair for pair ranking system. So we really compare what you're looking for and it gives a much better result. >> And that's incredibly critical for users these days who want results in milliseconds. Jason, you, Apply Digital as a partner of Algolia, talk to us about Apply Digital, what it is that you guys do, and then give us a little bit of insight on that partnership. >> Sure. So Apply Digital was originally founded in 2016 in Vancouver, Canada. And we have offices in Vancouver, Toronto, New York, LA, San Francisco, Mexico city, Sao Paulo and Amsterdam. And we are a digital experiences agency. So brands and companies, and startups, and all the way from startups to major global conglomerates who have this desire to truly create these amazing digital experiences, it could be a website, it could be an app, it could be a full blown marketing platform, just whatever it is. And they lack either the experience or the internal resources, or what have you, then they come to us. And and we are end-to-end, we strategy, design, product, development, all the way through the execution side. And to help us out, we partner with organizations like Algolia to offer certain solutions, like an Algolia's case, like search recommendation, things like that, to our various clients and customers who are like, "Hey, I want to create this experience and it's going to require search, or it's going to require some sort of recommendation." And we're like, "Well, we highly recommend that you use Algolia. They're a partner of ours, they've been absolutely amazing over the time that we've had the partnership. And that's what we do." And honestly, for digital experiences, search is the essence of the internet, it just is. So, I cannot think of a single digital experience that doesn't require some sort of search or recommendation engine attached to it. So, and Algolia has just knocked it out of the park with their experience, not only from a customer experience, but also from a development experience. So that's why they're just an amazing, amazing partner to have. >> Sounds like a great partnership. Daisy, let's point it back over to you. Talk about some of those main challenges, Jason alluded to them, that businesses are facing, whether it's e-commerce, SaaS, a startup or whatnot, where search and recommendations are concerned. 'Cause we all, I think I've had that experience, where we're searching for something, and Daisy, you were describing how the recommendation engine works. And when we are searching for something, if I've already bought a tent, don't show me more tent, show me things that would go with it. What are some of those main challenges that Algolia solution just eliminates? >> Sure. So I think, one of the main challenges we have to focus on is, most of our customers are fighting against the big guides out there that have hundreds of engineers on staff, custom building a search solution. And our consumers expect that response. You expect the same search response that you get when you're streaming video content looking for a movie, from your big retailer shopping experiences. So what we want to provide is the ability to deliver that result with much less work and hassle and have it all show up. And we do that by really focusing on the results that the customers need and what that view needs to look like. We see a lot of our customers just experiencing a huge loss in revenue by only providing basic search. And because as Jason put it, search is so fundamental to the internet, we all think it's easy, we all think it's just basic. And when you provide basic, you don't get the shoes with the dress, you get just the text response results back. And so we want to make sure that we're providing that back to our customers. What we see average is even, and everybody's going mobile. A lot of times I know I do all my shopping on my phone a lot of the time, and 40%-50% better relevancy results for our customers for mobile users. That's a huge impact to their use case. >> That is huge. And when we talked about patients wearing quite thin the last couple of years. But we have this expectation in our consumer lives and in our business lives if we're looking for SaaS or software, or whatnot, that we're going to be able to find what we want that's relevant to what we're looking for. And you mentioned revenue impact, customer churn, brand reputation, those are all things that if search isn't done well, to your point, Daisy, if it's done in a basic fashion, those are some of the things that customers are going to experience. Jason, talk to us about why Algolia, what was it specifically about that technology that really led Apply Digital to say, "This is the right partner to help eliminate some of those challenges that our customers could face?" >> Sure. So I'm in the product world. So I have the wonderful advantage of not worrying about how something's built, that is left, unfortunately, to the poor, poor engineers that have to work with us, mad scientist, product people, who are like, "I want, make it do this. I don't know how, but make it do this." And one of the big things is, with Algolia is the lift to implement is really, really light. Working closely with our engineering team, and even with our customers/users and everything like that, you kind of alluded to it a little earlier, it's like, at the end of the day, if it's bad search, it's bad search. It just is. It's terrible. And people's attention span can now be measured in nanoseconds, but they don't care how it works, they just want it to work. I push a button, I want something to happen, period. There's an entire universe that is behind that button, and that's what Algolia has really focused on, that universe behind that button. So there's two ways that we use them, on a web experience, there's the embedded Search widget, which is really, really easy to implement, documentation, and I cannot speak high enough about documentation, is amazing. And then from the web aspect, I'm sorry, from the mobile aspect, it's very API fort. And any type of API implementation where you can customize the UI, which obviously you can imagine our clients are like, "No we want to have our own front end. We want to have our own custom experience." We use Algolia as that engine. Again, the documentation and the light lift of implementation is huge. That is a massive, massive bonus for why we partnered with them. Before product, I was an engineer a very long time ago. I've seen bad documentation. And it's like, (Lisa laughing) "I don't know how to imple-- I don't know what this is. I don't know how to implement this, I don't even know what I'm looking at." But with Algolia and everything, it's so simple. And I know I can just hear the Apply Digital technology team, just grinding sometimes, "Why is a product guy saying that (mumbles)? He should do it." But it is, it just the lift, it's the documentation, it's the support. And it's a full blown partnership. And that's why we went with it, and that's what we tell our clients. It's like, listen, this is why we chose Algolia, because eventually this experience we're creating for them is theirs, ultimately it's theirs. And then they are going to have to pick it up after a certain amount of time once it's theirs. And having that transition of, "Look this is how easy it is to implement, here is all the documentation, here's all the support that you get." It just makes that transition from us to them beautifully seamless. >> And that's huge. We often talk about hard metrics, but ease of use, ease of implementation, the documentation, the support, those are all absolutely business critical for the organization who's implementing the software, the fastest time to value they can get, can be table stakes, and it can be on also a massive competitive differentiator. Daisy, I want to go back to you in terms of hard numbers. Algolia has a recent force or Total Economic Impact, or TEI study that really has some compelling stats. Can you share some of those insights with us? >> Yeah. Absolutely. I think that this is the one of the most fun numbers to share. We have a recent report that came out, it shared that there's a 382% Return on Investment across three years by implementing Algolia. So that's increase to revenue, increased conversion rate, increased time on your site, 382% Return on Investment for the purchase. So we know our pricing's right, we know we're providing for our customers. We know that we're giving them the results that we need. I've been in the search industry for long enough to know that those are some amazing stats, and I'm really proud to work for them and be behind them. >> That can be transformative for a business. I think we've all had that experience of trying to search on a website and not finding anything of relevance. And sometimes I scratch my head, "Why is this experience still like this? If I could churn, I would." So having that ability to easily implement, have the documentation that makes sense, and get such high ROI in a short time period is hugely differentiated for businesses. And I think we all know, as Jason said, we measure response time in nanoseconds, that's how much patience and tolerance we all have on the business side, on the consumer side. So having that, not just this fast search, but the contextual search is table stakes for organizations these days. I'd love for you guys, and on either one of you can take this, to share a customer example or two, that really shows the value of the Algolia product, and then also maybe the partnership. >> So I'll go. We have a couple of partners in two vastly different industries, but both use Algolia as a solution for search. One of them is a, best way to put this, multinational biotech health company that has this-- We built for them this internal portal for all of their healthcare practitioners, their HCPs, so that they could access information, data, reports, wikis, the whole thing. And it's basically, almost their version of Wikipedia, but it's all internal, and you can imagine the level of of data security that it has to be, because this is biotech and healthcare. So we implemented Algolia as an internal search engine for them. And the three main reasons why we recommended Algolia, and we implemented Algolia was one, HIPAA compliance. That's the first one, it's like, if that's a no, we're not playing. So HIPAA compliance, again, the ease of search, the whole contextual search, and then the recommendations and things like that. It was a true, it didn't-- It wasn't just like a a halfhearted implementation of an internal search engine to look for files thing, it is a full blown search engine, specifically for the data that they want. And I think we're averaging, if I remember the numbers correctly, it's north of 200,000 searches a month, just on this internal portal specifically for their employees in their company. And it's amazing, it's absolutely amazing. And then conversely, we work with a pretty high level adventure clothing brand, standard, traditional e-commerce, stable mobile application, Lisa, what you were saying earlier. It's like, "I buy everything on my phone," thing. And so that's what we did. We built and we support their mobile application. And they wanted to use for search, they wanted to do a couple of things which was really interesting. They wanted do traditional search, search catalog, search skews, recommendations, so forth and so on, but they also wanted to do a store finder, which was kind of interesting. So, we'd said, all right, we're going to be implementing Algolia because the lift is going to be so much easier than trying to do everything like that. And we did, and they're using it, and massively successful. They are so happy with it, where it's like, they've got this really contextual experience where it's like, I'm looking for a store near me. "Hey, I've been looking for these items. You know, I've been looking for this puffy vest, and I'm looking for a store near me." It's like, "Well, there's a store near me but it doesn't have it, but there's a store closer to me and it does have it." And all of that wraps around what it is. And all of it was, again, using Algolia, because like I said earlier, it's like, if I'm searching for something, I want it to be correct. And I don't just want it to be correct, I want it to be relevant. >> Lisa: Yes. >> And I want it to feel personalized. >> Yes. >> I'm asking to find something, give me something that I am looking for. So yeah. >> Yeah. That personalization and that relevance is critical. I keep saying that word "critical," I'm overusing it, but it is, we have that expectation that whether it's an internal portal, as you talked about Jason, or it's an adventure clothing brand, or a grocery store, or an e-commerce site, that what they're going to be showing me is exactly what I'm looking for, that magic behind there that's almost border lines on creepy, but we want it. We want it to be able to make our lives easier whether we are on the consumer side, whether we on the business side. And I do wonder what the Go To Market is. Daisy, can you talk a little bit about, where do customers go that are saying, "Oh, I need to Algolia, and I want to be able to do that." Now, what's the GTM between both of these companies? >> So where to find us, you can find us on AWS Marketplace which another favorite place. You can quickly click through and find, but you can connect us through Apply Digital as well. I think, we try to be pretty available and meet our customers where they are. So we're open to any options, and we love exploring with them. I think, what is fun and I'd love to talk about as well, in the customer cases, is not just the e-commerce space, but also the content space. We have a lot of content customers, things about news, organizations, things like that. And since that's a struggle to deliver results on, it's really a challenge. And also you want it to be relevant, so up-to-date content. So it's not just about e-commerce, it's about all of your solution overall, but we hope that you'll find us on AWS Marketplace or anywhere else. >> Got it. And that's a great point, that it's not just e-commerce, it's content. And that's really critical for some industry, businesses across industries. Jason and Daisy, thank you so much for joining me talking about Algolia, Apply Digital, what you guys are doing together, and the huge impact that you're making to the customer user experience that we all appreciate and know, and come to expect these days is going to be awesome. We appreciate your insights. >> Thank you. >> Thank you >> For Daisy and Jason, I'm Lisa Martin. You're watching "theCUBE," our "AWS Startup Showcase, MarTech Emerging Cloud-Scale Customer Experiences." Keep it right here on "theCUBE" for more great content. We're the leader in live tech coverage. (ending riff)

>>Hello, everyone. Welcome to the cubes presentation of the 80 startup showcase marketing technology, emerging cloud scale customer experiences. This is season two, episode three of the ongoing series covering the exciting startups from the, a AWS ecosystem to talk about all the top trends and also featuring the key customers. I'm your host, John ER, today we're joined by Louis Fernando, Denise vice president of peak pay social and John Kim, the CEO of Sandberg to learn about the future of what's going on in fostering deeper customer relationships. Gentlemen, thanks for joining us in the cube showcase, >>Excited to be here. >>So John talk about Sendbird real quick set the table for us. What you guys do, you got a customer here to highlight some of the key things you're doing with customers, the value proposition what's Sendbird and what's the showcase about, >>Yeah, I'm really excited to be here. Uh, I'm John founder, C of Sandberg. So Sandberg is the worst leading conversations platform for mobile applications. We can power user to user conversations in mobile applications, as well as the brand to user conversations such as marketing sales and support. So, uh, today we power over quarter billion users on a monthly basis. Uh, we have, you know, through over 300 employees across seven different countries around the world, we work with some of the world's leading, uh, uh, customers such as big pay that we are going to showcase today, along with other, uh, wonderful customers like DoorDash, Reddit, <inaudible> sports and so forth. We have collectively raised over 200 million in funding. Um, so that's kind of where we are today. >>Well, it's always great to have, uh, one great success. Uh, good funding, more important is the customers. And I love showcases where the customers do the talking, because that means you've got some success stories. Louise, talk about, um, are you happy customer? What's it like working with Sandberg? Give us the, give us the scoop. >>So sandbar is being a great partner with us. So pick pay is a Brazilian payment app. We're at a FinTech here with more than 30 million active users using everyday pick pay to pay everything. So the, the, the majority of the payments are between peers, between people. So sandbar is, is helping us to improve a lot this journey to make it more pleasant between every everyone who are using big, big. So we are here, let's talk and it's a >>Pleasure. Yeah, it's awesome. Well, I great to have you guys on great, great relationship. And one of the things we've been talking about on the cube, if the folks watching that know our audience, no we've been banging the, the drum hard on this new world and this new patterns of user expectations and building relationships in this new digital world is not about the old way, the old MarTech way. There are new new use cases, new expectations by the consumers, John, that are, that are bringing up new opportunities, but also expectations. It's not about, I mean, I mean, if someone's using discord, for example, cuz they're gamers, they're done discord. If they want to communicate with, with slack, they, I do slack, SMS, kind of old hat. You got WhatsApp, you've got all these now peer to peer organic connections, multiple channels. This is all the new world. What's your vision on this new relationship building digital communication world. >>Yeah. So I, I think you brought a really good point there. One of the most frequently used applications in the world today are messaging applications across any countries, any region, any culture, if you look at the most frequently used and most longest used applications are usually some form of a, a messaging application. Now the end users or the customers in the world are so used to using, uh, uh, such a, you know, frictionless ver very responsive, modern experience on those messaging applications. What we want to help with the business around the world, the 99.9% of the business around the world don't have those really te knowledge or user experience expertise in messaging. So we want to help our businesses, help our customers be able to harness the power of modern messaging capabilities and then be able to embed it in their own business so that they can retain their users on their platform, engage with them in the con context that their, uh, what their business is about so that they can not only, uh, control or provide a better user experience, but also be able to, uh, understand their users better, uh, understand what they're doing on their businesses, be able to own and, uh, control the data in a more secure and safe way. >>So really it's uh, we're like the Robin hood of the world trying to keep superpower yeah. Back to the businesses. >>Yeah. Deal from the rich idea, the messaging scale. Bring that to everybody else. I love that. Uh, and you got kind of this double int Robin hood kind of new for the new generation finance. This is about taking the advantage of scalable platforms, monopolies, right. And giving the entrepreneur an opportunity to have that same capability feature, rich Louise PPE. You guys used Sendbird together. You have to level up, you gotta compete with those big monopolies to pride, scalable conversations. Okay. How did you engage this? What was your success path look? What was it look like? >>Yeah. When we look to this majority, the bigger chat apps that we have nowadays in the market, we are looking to them and then Brazilians are using for their daily course, but Brazilians are paying every day millions and millions of payments. And these chat apps are not, uh, able to, to, to deal with these payments. So what we are doing here is that, uh, providing a solution where every conversation that are going to happen before, during, or after a payment between the, the people, they would, uh, uh, have a nice platform that could afford all, all of their emotions and discussions that they have to do before or after the payment. So we are putting together the chat platform and we with the payment platform. So that's, that's what we are doing now. >>Okay. So just so I get this right. You're using Sandberg essentially integrated your mobile payment experience. Okay. Which is your app you're Sandberg to bring that scalability into the, into the social app application into the app itself. Is that right? >>Yes. Perfect. Integrated with the payment journey. So everybody who is going to pay, they need to find the one, the, the one they want to pay and then they can chat and conclude the payment through the platform. Yeah. I >>Mean, why not have it right there at point of, uh, transaction. Right. Um, why did you, um, decide to, um, to use conversations in your mobile wallet? Just curious. >>So it's important to say that we were born social. We born in 2012. So when our main main product was peer to peer payments, so everybody were sending money to a friend requesting or charging their family. So a service provider. And once we, we started as a social platform in that period. In that moment, we are just focusing in likes comments and like public interactions and the word become more private. And as soon we under understood this situation, we decided to move from a public feed to a private, to a private interaction. So that's, uh, that then the, the conversational space was the solution for that moving from a public interaction to a private interaction. So between the peers, which are involved in the, the transaction. So that's why we are providing the chat solution integrated with payments. >>That's a great call. John, just give some context here, again, for the folks watching this is now expected, this integrated experience. What's your, how would you talk to folks out there? I mean, first of all, I, I, I see it clearly, you've got an app, you gotta have all this integration and you need it scaling to reach features. Talk about your view on that. Is that the, is that what's happening here? What's, what's the real dynamic here. What's the, the big trend. >>Yeah. One thing that's, uh, super interesting about, uh, uh, like messaging experience in general, if you think about any kind of conversations that's happening, uh, digitally between human beings, more and more conversations, just like what Louis mentioned earlier are happening between in a private setting, even on applications, whether it be slack or other forms of communication, uh, more hap uh, more conversations happen through either one-on-one conversations or in a private small group settings. And because people feel more secure, uh, safe to have, uh, more intimate conversations. So even when you're making transactions is more, you know, there's a higher trust and, uh, people tend to engage, uh, far better on platforms through these kind of private conversations. That's where we kind of come in, whether it be, you want to set a one-on-one conversations or with a group conversation. And then ultimately if you want to take it public in a large group setting, you can also support, you know, thousands, if not, you know, hundreds of thousands of people, uh, engaging a public forum as well. So all of those capabilities can be implemented using something Ember, but again, the world is, uh, right now the businesses and how the user are, are interacting with this with each other is all happening through digital conversations. And we're seeing more and more of that happening, uh, throughout the life cycle of our company. >>Yeah, just as a sidebar, I was just talking to a venture in San Francisco the other day, and we're talking about the future of security and SAS and cloud scale. And, you know, the conversation went to more of, is it SAS? Is it platform as a service Louis? I wanna get your thoughts because, you know, you're seeing more and more needs for customization, low code, no code. You're seeing these trends. You gotta built in security. So, you know, the different, the old SAS model was softwares a service, but now that's everything in the cloud is softwares a service. So, but you need to have that platform kind of vibe for scale customization, maybe some developer integration, cuz apps are becoming the, the touchpoint. So can you walk us through what your vision was when you decided to integrate, chat into your app and how did you see that chat, changing the customer experience for payments and across your user journey? Cause, I mean, it's obvious now looking at it, but it might not have been for some. What was your, what was your vision? And when you had to do that, >>When you looked to Brazilian reality, we can see those in, uh, payment apps. All of them are focused on the transactional moment. And as soon as we started to think, how could be, how could our journey be better, more pleased than the others and make people want to be here and to use and to open our app every day is just about making the interaction with the peers easier, even with a merchant or even with my friend. So the main point that our first step was just to connect all, all the users between themselves to payments. The second step we are providing now is using the chat platform, the send bird platform as a platform for peak pay. So we are going to provide more best information. We're going to provide a better customer experience through the support and everything. So, um, this, this, this interaction or this connection, this partnership with Sandberg are going to unlock a new level of service for our users. And at the same time, a much more pleasant or a more pleasant journey for them while they are using the, the app for a, a simple payment, or if they are going to look for a group objective or maybe a crowdfund in the future or a group to decide, or just to pay something. So we are then locking a new level of interaction between the peers between the people and the users that are, that are involved into this, this payment or this simple transaction, we are making it more conversational. >>Yeah. You're making the application more valuable. We're gonna get to that in the next segment about, you know, the future of apps one and done, you see a lot of sports apps, oh, this big tournament, you know, and then you use it and then you never use it again until next year. You know, you have very time specific apps, but now you guys are smart to kind of build this in, but I gotta ask you a question because a lot of developers and companies out there always have this buy versus build decision. Why did you decide to use Sendbird versus building it in house? It's always kind of like the big trade off. >>Yeah. First of all, it will take a long, long time for us to achieve a major platform as Sandberg. And we are not a chat platform. So we are going to use this social interaction to improve the payment platform that we have. So when we look to the market and we found Sandberg, then we thought, okay, this guys, they are a real platform. And through the conversations, we are seeing that they are roadmap working in synergy with our roadmap. And then we can, we could start to deliver value to our, to our users in a fastest way. Could you imagine it spending 2, 3, 4 years to develop something like sand? And even when we achieve this point, probably our solution will be, would be weaker than, than Sandberg. So it was like no brainer to do that. Yeah. Because we want to improve the payment journey, not to do a chat, only a chat platform. So that's why we are working together to prove it's >>Really, you start to see these plugins, these, you know, look at Stripe for payments, for instance, right. And here in the success they've had, you know, people want to plug in for services. So John, I gotta ask you about, um, about the, the complexity that goes into it. The trust required that they have for you, you have to do this heavy lifting, you gotta provide the confidence that your service is gonna have to scale the compliance. Talk about that. What do you guys do under the covers that make this easy again, great business model, heavy lifting done by you. Seamless integration provide that value. That's why business is good, but there's a lot going on share what's happening under the, under the covers. >>Yeah. Um, before going to like the technical, like intricacy of what we do just to provide a little bit of background context on why we even started this business is we, uh, this is my second startup. My first company was a gaming company. We had built like chat three, four times just for our own game. So we were basically, we felt like we were reinventing the wheel. And then we actually went on a buyer's journey when we were building a social application, uh, uh, for, for, uh, uh, building our own community. We tried to actually be a buyer to see if we can actually find a solution. We want to use turns out that there weren't a lot of like sophisticated, you know, top notch, modern, uh, uh, chat experience that we can build using some other third party solutions. So we had to build all of that ourselves, which became the foundation for se today. >>And what we realized is that for most companies like using a building, the most sophisticated chat is probably not going to be their highest priority in case a pick pay will be, you know, financial transactions and all the other business that can be built on and hosted by platform like pick pay. But, you know, building the most topnotch chat experience would be a priority for a company like let's say WhatsApp or, or telegram, but it will probably not be the priority for, you know, major gaming companies, food delivery companies, finance companies, chat is not the highest priority. That's kind of where we come in, cuz chat is the highest priority for us. And we also have a privilege of working with some of the other, uh, world industry, uh, industry leaders. So by, uh, having this collective experience, working with the industry leaders, we get, uh, uh, technological superiority, being able to, uh, scale to, you know, hundreds of millions of users on a monthly basis. Also the security and the compliances by working with some of the largest commercial banks on some of the largest FinTech applications across the globe. So we have, you know, security, compliances, all the industry, best practices that are built in and all the new topnotch user experience that we are, uh, building with other customers can be also be, uh, utilized by a customer like pick pay. So you get this collective almost like evolutionary benefit. Yeah. By, uh, working with a company like us, >>You get a lot of economies of scale. Could you mind just sharing the URL for the company? So folks watching can go get, do a deep dive. Cause I'm you guys got a lot of, lot of, um, certifications under the covers, a lot of things you guys do. So you mind just sharing URL real quick. >>Yeah. So our company, uh, you can find everything about our company on sandberg.com like carrot pigeon. So, uh, you're sending a bird to send a message. So, uh, yeah. send.com >>All so let's get it to the application, cuz this is really interesting cuz Chad is table stakes now, but things are evolving beyond Chad. You gotta integrate that user experience. It's data. Now you gotta have scale. I mean, you know, people who wanna roll their own chat will find out there's a lot of client side and backend scale issues. Right. You can have a tsunami river like on Twitch, you know, you chat. I mean that, could you got client side issues, data scale. <laugh> right. You got backend. Um, Louis, talk about that dynamic because you know, as you start to scale, you want to rely on that. Talk about this dynamic, how apps now are integrating all these new features. So is it, are apps gonna go like more multifunctional? Do you see apps one and done? What's the, how do you guys see this app world playing out and where does, does the Sendbird fit in? And >>Just, just let me know better John, about the performance or about the, just, just let me >>Oh, slow with performance. Uh, performance is huge, right? You gotta have no one wants to have lag on, on chat. >>Okay. So, um, big pay when we look to the payments have millions, thousands of, of, of payments happen happening every second. So what we are doing now is moving all the payments through a conversation. So it always happened inside the conversation. So since from the first moment, um, every second counts to convert this client. And since from the first moment we never saw in, on Sandberg, any issue about that. And even when we have a question or something that we need to improve the team we're working together. So that that's, those are the points that are making us to work together and to make things going pretty fast. When we look to the users who are going to use chat, they are, their intention is three times better than the users who are not using payments through the chat. They are average. Average spent is three times higher too. >>So they, they are making more connections. They are chatting with their friends. They are friends are here. So the network effect is stronger. So if they're going to pay and they need to wait one more second, two seconds to conclude the payment, probably they will not go into choose paying through the, again, they will use only the wallet, only the code, only the Alliance of the user. So that's is so important for us to perform really, really fast. And then this is what we are finding. And this is what is happening with the integration with Sandberg. >>And what's interesting is, is that the by build chat with conversation, we just had a minute ago kind of plays in here. You get the benefits of Sandberg, but now your transactional fidelity is in the chat <laugh> that you don't build that you rely on them on. So again, that's an interesting dynamic. This is the future of apps, John, this is where it matters. The engagement. This is what you talk about is the new, the new digital experience who would've thought that five, 10 years ago. I mean, chat was just like, Hey, what's going around direct message. Now it's integral part of the app. What's your reading. >>Yeah. I mean, we're seeing that across, uh, uh, to Lewis's point, not just transactions, but like marketing messages are now being sent through chat. So the marketing is no longer just about like giving discount calls, but you can actually reengage with the brand. Uh, also support is becoming more real time through chat. So you're actually building a relationship. The support agents have a better context about the previous conversations and the transactions, the sales conversations, even like building, uh, building alerts, notification, all those things are now, uh, happening through conversations. And that's a better way for customers to engage with the brand cuz you actually, you're actually building a better relationship and also, uh, being able to trust the brand more because there is a channel for you to communicate and, and, and be seen and be heard, uh, by the brand. So we do believe that that's the future of the business and how more and more, uh, brands will be building relationships with their customers. >>Yeah. I love, I love your business model. I think it's really critical. And I think that stickiness is a real, uh, call out point there and the brand, the co-branding and the branding capability, but also really quickly in the last minute we have John and Luis, if you don't mind talking about security, I mean, I can't go a day now without getting an SMS scam, uh, text, uh, you seeing it now on WhatsApp. I mean, I don't even use telegram anymore. I mean, come on. So like, like this is now a problem. The old way has been infiltrated with spam and security issues. Security has to be there. The trust and security real quick, John, we'll start with you and we all Louis go, go ahead. >>No, no. Just, just to, to say how important is that we are not only a chatting platform. We are a payment platform, so we have money now, the transaction. So here in Brazil, we have all this safe, the, the, the layers, the security layers that we have in, on our app. And then we have the security layers provided from Sandburg. So, and when we look to the features, Sandberg are providing to us a lot of features that help users to feel safer like per refined profiles, like announcements, where it's a profile from peak pay, where the users can recognize. So this is peak pay talking with me. It's not a user trying to pass, trying to use big Bay's name to talk with me. So these issues is something that we are really, really, we really care about here because we are not only a chat platform. As I said before, we are a payment platform. We are a FinTech, we're at a digital bank. So we need to take care a lot and we don't have any complaint about it because Sandberg understood it. And then they, they, they are providing since the first moment with the perfect solutions and the user interface to make it simpler for the users to recognize that we speak, pay who is chatting with them, not a user with, with bad, bad intentions. >>Great, great insight, Louis. Thanks for sharing that, John really appreciate you guys coming on. Great showcase. Real final word. John will give you the final word folks watching out there. How do they engage with Sendbird? I want to integrate, I want to use your chat service. What do I do? Do I have to connect in as it managed service is the line of code. What do I do to get Sendbird? >>Yeah. So if you're a developer building a mobile application, simply come visit our website, we have a open documentation and SDK you can download and simply plug into your application. You can have a chat experience up and running matter of minutes, if not ours using our UI kit. So we want to make it as easy as possible for all the builders in the world to be able to harness the superpower of digital conversations. >>All right, great. Congratulations, John, on your success and all the growth and Louis, thanks for coming in, sharing the customer perspective and great insight. Thanks for coming on the showcase. Really appreciate it. Thanks for your time. >>Yeah. Thank you for having me. >>Okay. The a of us startup showcase season two, episode three here I'm John for your host. Thanks for watching.

(upbeat music) >> Hello, everyone, welcome to theCUBE's presentation of the AWS Startup Showcase. We're in Season two, Episode three, and this is the topic of MarTech and the Emerging Cloud-Scale Customer Experiences, the ongoing coverage of AWS's ecosystem of large scale growth and new companies and growing companies. I'm your host, John Furrier. We're excited to have Tim Barnes, Global Director, General Manager of Advertiser and Marketing at AWS here doing the keynote cloud-scale customer experience. Tim, thanks for coming on. >> Oh, great to be here and thank you for having me. >> You've seen many cycles of innovation, certainly in the ad tech platform space around data, serving consumers and a lot of big, big scale advertisers over the years as the Web 1.0, 2.0, now 3.0 coming, cloud-scale, roll of data, all big conversations changing the game. We see things like cookies going away. What does this all mean? Silos, walled gardens, a lot of new things are impacting the applications and expectations of consumers, which is also impacting the folks trying to reach the consumers. And this is kind of creating a kind of a current situation, which is challenging, but also an opportunity. Can you share your perspective of what this current situation is, as the emerging MarTech landscape emerges? >> Yeah, sure, John, it's funny in this industry, the only constant has changed and it's an ever-changing industry and never more so than right now. I mean, we're seeing with whether it's the rise of privacy legislation or just breach of security of data or changes in how the top tech providers and browser controllers are changing their process for reaching customers. This is an inflection point in the history of both ad tech and MarTech. You hit the nail on the head with cookie deprecation, with Apple removing IDFA, changes to browsers, et cetera, we're at an interesting point. And by the way, we're also seeing an explosion of content sources and ability to reach customers that's unmatched in the history of advertising. So those two things are somewhat at odds. So whether we see the rise of connected television or digital out of home, you mentioned Web 3.0 and the opportunities that may present in metaverse, et cetera, it's an explosion of opportunity, but how do we continue to connect brands with customers and do so in a privacy compliant way? And that's really the big challenge we're facing. One of the things that I see is the rise of modeling or machine learning as a mechanism to help remove some of these barriers. If you think about the idea of one-to-one targeting, well, that's going to be less and less possible as we progress. So how am I still as a brand advertiser or as a targeted advertiser, how am I going to still reach the right audience with the right message in a world where I don't necessarily know who they are. And modeling is a really key way of achieving that goal and we're seeing that across a number of different angles. >> We've always talked about on the ad tech business for years, it's the behemoth of contextual and behavioral, those dynamics. And if you look at the content side of the business, you have now this new, massive source of new sources, blogging has been around for a long time, you got video, you got newsletters, you got all kinds of people, self-publishing, that's been around for a while, right? So you're seeing all these new sources. Trust is a big factor, but everyone wants to control their data. So this walled garden perpetuation of value, I got to control my data, but machine learning works best when you expose data, so this is kind of a paradox. Can you talk about the current challenge here and how to overcome it because you can't fight fashion, as they say, and we see people kind of going down this road as saying, data's a competitive advantage, but I got to figure out a way to keep it, own it, but also share it for the machine learning. What's your take on that? >> Yeah, I think first and foremost, if I may, I would just start with, it's super important to make that connection with the consumer in the first place. So you hit the nail on the head for advertisers and marketers today, the importance of gaining first party access to your customer and with permission and consent is paramount. And so just how you establish that connection point with trust and with very clear directive on how you're going to use the data has never been more important. So I would start there if I was a brand advertiser or a marketer, trying to figure out how I'm going to better connect with my consumers and get more first party data that I could leverage. So that's just building the scale of first party data to enable you to actually perform some of the types of approaches we'll discuss. The second thing I would say is that increasingly, the challenge exists with the exchange of the data itself. So if I'm a data control, if I own a set of first party data that I have consent with consumers to use, and I'm passing that data over to a third party, and that data is leaked, I'm still responsible for that data. Or if somebody wants to opt out of a communication and that opt out signal doesn't flow to the third party, I'm still liable, or at least from the consumer's perspective, I've provided a poor customer experience. And that's where we see the rise of the next generation, I call it of data clean rooms, the approaches that you're seeing, a number of customers take in terms of how they connect data without actually moving the data between two sources. And we're seeing that as certainly a mechanism by which you can preserve accessibility data, we call that federated data exchange or federated data clean rooms and I think you're seeing that from a number of different parties in the industry. >> That's awesome, I want to get into the data interoperability because we have a lot of startups presenting in this episode around that area, but why I got you here, you mentioned data clean room. Could you define for us, what is a federated data clean room, what is that about? >> Yeah, I would simply describe it as zero data movement in a privacy and secure environment. To be a little bit more explicit and detailed, it really is the idea that if I'm a party A and I want to exchange data with party B, how can I run a query for analytics or other purposes without actually moving data anywhere? Can I run a query that has accessibility to both parties, that has the security and the levels of aggregation that both parties agree to and then run the query and get those results sets back in a way that it actually facilitates business between the two parties. And we're seeing that expand with partners like Snowflake and InfoSum, even within Amazon itself, AWS, we have data sharing capabilities within Redshift and some of our other data-led capabilities. And we're just seeing explosion of demand and need for customers to be able to share data, but do it in a way where they still control the data and don't ever hand it over to a third party for execution. >> So if I understand this correctly, this is kind of an evolution to kind of take away the middleman, if you will, between parties that used to be historically the case, is that right? >> Yeah, I'd say this, the middleman still exists in many cases. If you think about joining two parties' data together, you still have the problem of the match key. How do I make sure that I get the broadest set of data to match up with the broadest set of data on the other side? So we have a number of partners that provide these types of services from LiveRamp, TransUnion, Experian, et cetera. So there's still a place for that so-called middleman in terms of helping to facilitate the transaction, but as a clean room itself, I think that term is becoming outdated in terms of a physical third party location, where you push data for analysis, that's controlled by a third party. >> Yeah, great clarification there. I want to get into this data interoperability because the benefits of AWS and cloud scales we've seen over the past decade and looking forward is, it's an API based economy. So APIs and microservices, cloud native stuff is going to be the key to integration. And so connecting people together is kind of what we're seeing as the trend. People are connecting their data, they're sharing code in open source. So there's an opportunity to connect the ecosystem of companies out there with their data. Can you share your view on this interoperability trend, why it's important and what's the impact to customers who want to go down this either automated or programmatic connection oriented way of connecting data. >> Never more important than it has been right now. I mean, if you think about the way we transact it and still too today do to a certain extent through cookie swaps and all sorts of crazy exchanges of data, those are going away at some point in the future; it could be a year from now, it could be later, but they're going away. And I think that that puts a great amount of pressure on the broad ecosystem of customers who transact for marketers, on behalf of marketers, both for advertising and marketing. And so data interoperability to me is how we think about providing that transactional layer between multiple parties so that they can continue to transact in a way that's meaningful and seamless, and frankly at lower cost and at greater scale than we've done in the past with less complexity. And so, we're seeing a number of changes in that regard, whether that's data sharing and data clean rooms or federated clean rooms, as we described earlier, whether that's the rise of next generation identity solutions, for example, the UID 2.0 Consortium, which is an effort to use hashed email addresses and other forms of identifiers to facilitate data exchange for the programmatic ecosystem. These are sort of evolutions based on this notion that the old world is going away, the new world is coming, and part of that is how do we connect data sources in a more seamless and frankly, efficient manner. >> It's almost interesting, it's almost flipped upside down, you had this walled garden mentality, I got to control my data, but now I have data interoperability. So you got to own and collect the data, but also share it. This is going to kind of change the paradigm around my identity platforms, attributions, audience, as audiences move around, and with cookies going away, this is going to require a new abstraction, a new way to do it. So you mentioned some of those standards. Is there a path in this evolution that changes it for the better? What's your view on this? What do you see happening? What's going to come out of this new wave? >> Yeah, my father was always fond of telling me, "The customer, my customers is my customer." And I like to put myself in the shoes of the Marc Pritchards of the world at Procter & Gamble and think, what do they want? And frankly, their requirements for data and for marketing have not changed over the last 20 years. It's, I want to reach the right customer at the right time, with the right message and I want to be able to measure it. In other words, summarizing, I want omnichannel execution with omnichannel measurement, and that's become increasingly difficult as you highlighted with the rise of the walled gardens and increasingly data living in silos. And so I think it's important that we, as an industry start to think about what's in the best interest of the one customer who brings virtually 100% of the dollars to this marketplace, which is the CMO and the CMO office. And how do we think about returning value to them in a way that is meaningful and actually drives its industry forward. And I think that's where the data operability piece becomes really important. How do we think about connecting the omnichannel channels of execution? How do we connect that with partners who run attribution offerings with machine learning or partners who provide augmentation or enrichment data such as third party data providers, or even connecting the buy side with the sell side in a more efficient manner? How do I make that connection between the CMO and the publisher in a more efficient and effective way? And these are all challenges facing us today. And I think at the foundational layer of that is how do we think about first of all, what data does the marketer have, what is the first party data? How do we help them ethically source and collect more of that data with proper consent? And then how do we help them join that data into a variety of data sources in a way that they can gain value from it. And that's where machine learning really comes into play. So whether that's the notion of audience expansion, whether that's looking for some sort of cohort analysis that helps with contextual advertising, whether that's the notion of a more of a modeled approach to attribution versus a one-to-one approach, all of those things I think are in play, as we think about returning value back to that customer of our customer. >> That's interesting, you broke down the customer needs in three areas; CMO office and staff, partners ISV software developers, and then third party services. Kind of all different needs, if you will, kind of tiered, kind of at the center of that's the user, the consumer who have the expectations. So it's interesting, you have the stakeholders, you laid out kind of those three areas as to customers, but the end user, the consumer, they have a preference, they kind of don't want to be locked into one thing. They want to move around, they want to download apps, they want to play on Reddit, they want to be on LinkedIn, they want to be all over the place, they don't want to get locked in. So you have now kind of this high velocity user behavior. How do you see that factoring in, because with cookies going away and kind of the convergence of offline-online, really becoming predominant, how do you know someone's paying attention to what and when attention and reputation. All these things seem complex. How do you make sense of it? >> Yeah, it's a great question. I think that the consumer as you said, finds a creepiness factor with a message that follows them around their various sources of engagement with content. So I think at first and foremost, there's the recognition by the brand that we need to be a little bit more thoughtful about how we interact with our customer and how we build that trust and that relationship with the customer. And that all starts with of course, opt-in process consent management center but it also includes how we communicate with them. What message are we actually putting in front of them? Is it meaningful, is it impactful? Does it drive value for the customer? I think we've seen a lot of studies, I won't recite them that state that most consumers do find value in targeted messaging, but I think they want it done correctly and there in lies the problem. So what does that mean by channel, especially when we lose the ability to look at that consumer interaction across those channels. And I think that's where we have to be a little bit more thoughtful with frankly, kind of going back to the beginning with contextual advertising, with advertising that perhaps has meaning, or has empathy with the consumer, perhaps resonates with the consumer in a different way than just a targeted message. And we're seeing that trend, we're seeing that trend both in television, connected television as those converge, but also as we see about connectivity with gaming and other sort of more nuanced channels. The other thing I would say is, I think there's a movement towards less interruptive advertising as well, which kind of removes a little bit of those barriers for the consumer and the brand to interact. And whether that be dynamic product placement, content optimization, or whether that be sponsorship type opportunities within digital. I think we're seeing an increased movement towards those types of executions, which I think will also provide value to both parties. >> Yeah, I think you nailed it there. I totally agree with you on the contextual targeting, I think that's a huge deal and that's proven over the years of providing benefit. People, they're trying to find what they're looking for, whether it's data to consume or a solution they want to buy. So I think that all kind of ties together. The question is these three stakeholders, the CMO office and staff you mentioned, and the software developers, apps, or walled gardens, and then like ad servers as they come together, have to have standards. And so, I think to me, I'm trying to squint through all the movement and the shifting plates that are going on in the industry and trying to figure out where are the dots connecting? And you've seen many cycles of innovation at the end of the day, it comes down to who can perform best for the end user, as well as the marketers and advertisers, so that balance. What's your view on this shift? It's going to land somewhere, it has to land in the right area, and the market's very efficient. I mean, this ad market's very efficient. >> Yeah, I mean, in some way, so from a standards perspective, I support and we interact extensively with the IB and other industry associations on privacy enhancing technologies and how we think about these next generations of connection points or identifiers to connect with consumers. But I'd say this, with respect to the CMO, and I mentioned the publisher earlier, I think over the last 10 years with the rise of programmatic, certainly we saw the power reside mostly with the CMO who was able to amass a large pool of cookies or purchase a large sort of cohort of customers with cookie based attributes and then execute against that. And so almost a blind fashion to the publisher, the publisher was sort of left to say, "Hey, here's an opportunity, do you want to buy it or not?" With no real reason why the marketer might be buying that customer? And I think that we're seeing a shift backwards towards the publisher and perhaps a healthy balance between the two. And so, I do believe that over time, that we're going to see publishers provide a lot more, what I might almost describe as mini walled gardens. So the ability, great publisher or a set of publishers to create a cohort of customers that can be targeted through programmatic or perhaps through programmatic guaranteed in a way that it's a balance between the two. And frankly thinking about that notion of federated data clean rooms, you can see an approach where publishers are able to share their first party data with a marketer's first party data, without either party feeling like they're giving up something or passing all their value over to the other. And I do believe we're going to see some significant technology changes over the next three to four years. That really rely on that interplay between the marketer and the publisher in a way that it helps both sides achieve their goals, and that is, increasing value back to the publisher in terms of higher CPMs, and of course, better reach and frequency controls for the marketer. >> I think you really brought up a big point there we can maybe follow up on, but I think this idea of publishers getting more control and power and value is an example of the market filling a void and the power log at the long tail, it's kind of a straight line. Then it's got the niche kind of communities, it's growing in the middle there, and I think the middle of the torso of that power law is the publishers because they have all the technology to measure the journeys and the click throughs and all this traffic going on their platform, but they just need to connect to someone else. >> Correct. >> That brings in the interoperability. So, as a publisher ourselves, we see that long tail getting really kind of fat in the middle where new brands are going to emerge, if they have audience. I mean, some podcasts have millions of users and some blogs are attracting massive audience, niche audiences that are growing. >> I would say, just look at the rise of what we might not have considered publishers in the past, but are certainly growing as publishers today. Customers like Instacart or Uber who are creating ad platforms or gaming, which of course has been an ad supported platform for some time, but is growing immensely. Retail as a platform, of course, amazon.com being one of the biggest retail platforms with advertising supported models, but we're seeing that growth across the board for retail customers. And I think that again, there's never been more opportunities to reach customers. We just have to do it the right way, in the way that it's not offensive to customers, not creepy, if you want to call it that, and also maximizes value for both parties and that be both the buy and the sell side. >> Yeah, everyone's a publisher and everyone's a media company. Everyone has their own news network, everyone has their own retail, it's a completely new world. Tim, thanks for coming on and sharing your perspective and insights on this key note, Tim Barnes, Global Director, General Manager of Advertiser and Market at AWS here with the Episode three of Season two of the AWS Startup Showcase. I'm John Furrier, thanks for watching. (upbeat music)

(mellow music) >> Hello everyone, welcome to theCUBE's coverage of AWS Startup Showcase. This is the closing panel session on AI machine learning, the top startups generating generative AI on AWS. It's a great panel. This is going to be the experts talking about riding the wave in generative AI. We got Ankur Mehrotra, who's the director and general manager of AI and machine learning at AWS, and Clem Delangue, co-founder and CEO of Hugging Face, and Ori Goshen, who's the co-founder and CEO of AI21 Labs. Ori from Tel Aviv dialing in, and rest coming in here on theCUBE. Appreciate you coming on for this closing session for the Startup Showcase. >> Thanks for having us. >> Thank you for having us. >> Thank you. >> I'm super excited to have you all on. Hugging Face was recently in the news with the AWS relationship, so congratulations. Open source, open science, really driving the machine learning. And we got the AI21 Labs access to the LLMs, generating huge scale live applications, commercial applications, coming to the market, all powered by AWS. So everyone, congratulations on all your success, and thank you for headlining this panel. Let's get right into it. AWS is powering this wave here. We're seeing a lot of push here from applications. Ankur, set the table for us on the AI machine learning. It's not new, it's been goin' on for a while. Past three years have been significant advancements, but there's been a lot of work done in AI machine learning. Now it's released to the public. Everybody's super excited and now says, "Oh, the future's here!" It's kind of been going on for a while and baking. Now it's kind of coming out. What's your view here? Let's get it started. >> Yes, thank you. So, yeah, as you may be aware, Amazon has been in investing in machine learning research and development since quite some time now. And we've used machine learning to innovate and improve user experiences across different Amazon products, whether it's Alexa or Amazon.com. But we've also brought in our expertise to extend what we are doing in the space and add more generative AI technology to our AWS products and services, starting with CodeWhisperer, which is an AWS service that we announced a few months ago, which is, you can think of it as a coding companion as a service, which uses generative AI models underneath. And so this is a service that customers who have no machine learning expertise can just use. And we also are talking to customers, and we see a lot of excitement about generative AI, and customers who want to build these models themselves, who have the talent and the expertise and resources. For them, AWS has a number of different options and capabilities they can leverage, such as our custom silicon, such as Trainium and Inferentia, as well as distributed machine learning capabilities that we offer as part of SageMaker, which is an end-to-end machine learning development service. At the same time, many of our customers tell us that they're interested in not training and building these generative AI models from scratch, given they can be expensive and can require specialized talent and skills to build. And so for those customers, we are also making it super easy to bring in existing generative AI models into their machine learning development environment within SageMaker for them to use. So we recently announced our partnership with Hugging Face, where we are making it super easy for customers to bring in those models into their SageMaker development environment for fine tuning and deployment. And then we are also partnering with other proprietary model providers such as AI21 and others, where we making these generative AI models available within SageMaker for our customers to use. So our approach here is to really provide customers options and choices and help them accelerate their generative AI journey. >> Ankur, thank you for setting the table there. Clem and Ori, I want to get your take, because the riding the waves, the theme of this session, and to me being in California, I imagine the big surf, the big waves, the big talent out there. This is like alpha geeks, alpha coders, developers are really leaning into this. You're seeing massive uptake from the smartest people. Whether they're young or around, they're coming in with their kind of surfboards, (chuckles) if you will. These early adopters, they've been on this for a while; Now the waves are hitting. This is a big wave, everyone sees it. What are some of those early adopter devs doing? What are some of the use cases you're seeing right out of the gate? And what does this mean for the folks that are going to come in and get on this wave? Can you guys share your perspective on this? Because you're seeing the best talent now leaning into this. >> Yeah, absolutely. I mean, from Hugging Face vantage points, it's not even a a wave, it's a tidal wave, or maybe even the tide itself. Because actually what we are seeing is that AI and machine learning is not something that you add to your products. It's very much a new paradigm to do all technology. It's this idea that we had in the past 15, 20 years, one way to build software and to build technology, which was writing a million lines of code, very rule-based, and then you get your product. Now what we are seeing is that every single product, every single feature, every single company is starting to adopt AI to build the next generation of technology. And that works both to make the existing use cases better, if you think of search, if you think of social network, if you think of SaaS, but also it's creating completely new capabilities that weren't possible with the previous paradigm. Now AI can generate text, it can generate image, it can describe your image, it can do so many new things that weren't possible before. >> It's going to really make the developers really productive, right? I mean, you're seeing the developer uptake strong, right? >> Yes, we have over 15,000 companies using Hugging Face now, and it keeps accelerating. I really think that maybe in like three, five years, there's not going to be any company not using AI. It's going to be really kind of the default to build all technology. >> Ori, weigh in on this. APIs, the cloud. Now I'm a developer, I want to have live applications, I want the commercial applications on this. What's your take? Weigh in here. >> Yeah, first, I absolutely agree. I mean, we're in the midst of a technology shift here. I think not a lot of people realize how big this is going to be. Just the number of possibilities is endless, and I think hard to imagine. And I don't think it's just the use cases. I think we can think of it as two separate categories. We'll see companies and products enhancing their offerings with these new AI capabilities, but we'll also see new companies that are AI first, that kind of reimagine certain experiences. They build something that wasn't possible before. And that's why I think it's actually extremely exciting times. And maybe more philosophically, I think now these large language models and large transformer based models are helping us people to express our thoughts and kind of making the bridge from our thinking to a creative digital asset in a speed we've never imagined before. I can write something down and get a piece of text, or an image, or a code. So I'll start by saying it's hard to imagine all the possibilities right now, but it's certainly big. And if I had to bet, I would say it's probably at least as big as the mobile revolution we've seen in the last 20 years. >> Yeah, this is the biggest. I mean, it's been compared to the Enlightenment Age. I saw the Wall Street Journal had a recent story on this. We've been saying that this is probably going to be bigger than all inflection points combined in the tech industry, given what transformation is coming. I guess I want to ask you guys, on the early adopters, we've been hearing on these interviews and throughout the industry that there's already a set of big companies, a set of companies out there that have a lot of data and they're already there, they're kind of tinkering. Kind of reminds me of the old hyper scaler days where they were building their own scale, and they're eatin' glass, spittin' nails out, you know, they're hardcore. Then you got everybody else kind of saying board level, "Hey team, how do I leverage this?" How do you see those two things coming together? You got the fast followers coming in behind the early adopters. What's it like for the second wave coming in? What are those conversations for those developers like? >> I mean, I think for me, the important switch for companies is to change their mindset from being kind of like a traditional software company to being an AI or machine learning company. And that means investing, hiring machine learning engineers, machine learning scientists, infrastructure in members who are working on how to put these models in production, team members who are able to optimize models, specialized models, customized models for the company's specific use cases. So it's really changing this mindset of how you build technology and optimize your company building around that. Things are moving so fast that I think now it's kind of like too late for low hanging fruits or small, small adjustments. I think it's important to realize that if you want to be good at that, and if you really want to surf this wave, you need massive investments. If there are like some surfers listening with this analogy of the wave, right, when there are waves, it's not enough just to stand and make a little bit of adjustments. You need to position yourself aggressively, paddle like crazy, and that's how you get into the waves. So that's what companies, in my opinion, need to do right now. >> Ori, what's your take on the generative models out there? We hear a lot about foundation models. What's your experience running end-to-end applications for large foundation models? Any insights you can share with the app developers out there who are looking to get in? >> Yeah, I think first of all, it's start create an economy, where it probably doesn't make sense for every company to create their own foundation models. You can basically start by using an existing foundation model, either open source or a proprietary one, and start deploying it for your needs. And then comes the second round when you are starting the optimization process. You bootstrap, whether it's a demo, or a small feature, or introducing new capability within your product, and then start collecting data. That data, and particularly the human feedback data, helps you to constantly improve the model, so you create this data flywheel. And I think we're now entering an era where customers have a lot of different choice of how they want to start their generative AI endeavor. And it's a good thing that there's a variety of choices. And the really amazing thing here is that every industry, any company you speak with, it could be something very traditional like industrial or financial, medical, really any company. I think peoples now start to imagine what are the possibilities, and seriously think what's their strategy for adopting this generative AI technology. And I think in that sense, the foundation model actually enabled this to become scalable. So the barrier to entry became lower; Now the adoption could actually accelerate. >> There's a lot of integration aspects here in this new wave that's a little bit different. Before it was like very monolithic, hardcore, very brittle. A lot more integration, you see a lot more data coming together. I have to ask you guys, as developers come in and grow, I mean, when I went to college and you were a software engineer, I mean, I got a degree in computer science, and software engineering, that's all you did was code, (chuckles) you coded. Now, isn't it like everyone's a machine learning engineer at this point? Because that will be ultimately the science. So, (chuckles) you got open source, you got open software, you got the communities. Swami called you guys the GitHub of machine learning, Hugging Face is the GitHub of machine learning, mainly because that's where people are going to code. So this is essentially, machine learning is computer science. What's your reaction to that? >> Yes, my co-founder Julien at Hugging Face have been having this thing for quite a while now, for over three years, which was saying that actually software engineering as we know it today is a subset of machine learning, instead of the other way around. People would call us crazy a few years ago when we're seeing that. But now we are realizing that you can actually code with machine learning. So machine learning is generating code. And we are starting to see that every software engineer can leverage machine learning through open models, through APIs, through different technology stack. So yeah, it's not crazy anymore to think that maybe in a few years, there's going to be more people doing AI and machine learning. However you call it, right? Maybe you'll still call them software engineers, maybe you'll call them machine learning engineers. But there might be more of these people in a couple of years than there is software engineers today. >> I bring this up as more tongue in cheek as well, because Ankur, infrastructure's co is what made Cloud great, right? That's kind of the DevOps movement. But here the shift is so massive, there will be a game-changing philosophy around coding. Machine learning as code, you're starting to see CodeWhisperer, you guys have had coding companions for a while on AWS. So this is a paradigm shift. How is the cloud playing into this for you guys? Because to me, I've been riffing on some interviews where it's like, okay, you got the cloud going next level. This is an example of that, where there is a DevOps-like moment happening with machine learning, whether you call it coding or whatever. It's writing code on its own. Can you guys comment on what this means on top of the cloud? What comes out of the scale? What comes out of the benefit here? >> Absolutely, so- >> Well first- >> Oh, go ahead. >> Yeah, so I think as far as scale is concerned, I think customers are really relying on cloud to make sure that the applications that they build can scale along with the needs of their business. But there's another aspect to it, which is that until a few years ago, John, what we saw was that machine learning was a data scientist heavy activity. They were data scientists who were taking the data and training models. And then as machine learning found its way more and more into production and actual usage, we saw the MLOps become a thing, and MLOps engineers become more involved into the process. And then we now are seeing, as machine learning is being used to solve more business critical problems, we're seeing even legal and compliance teams get involved. We are seeing business stakeholders more engaged. So, more and more machine learning is becoming an activity that's not just performed by data scientists, but is performed by a team and a group of people with different skills. And for them, we as AWS are focused on providing the best tools and services for these different personas to be able to do their job and really complete that end-to-end machine learning story. So that's where, whether it's tools related to MLOps or even for folks who cannot code or don't know any machine learning. For example, we launched SageMaker Canvas as a tool last year, which is a UI-based tool which data analysts and business analysts can use to build machine learning models. So overall, the spectrum in terms of persona and who can get involved in the machine learning process is expanding, and the cloud is playing a big role in that process. >> Ori, Clem, can you guys weigh in too? 'Cause this is just another abstraction layer of scale. What's it mean for you guys as you look forward to your customers and the use cases that you're enabling? >> Yes, I think what's important is that the AI companies and providers and the cloud kind of work together. That's how you make a seamless experience and you actually reduce the barrier to entry for this technology. So that's what we've been super happy to do with AWS for the past few years. We actually announced not too long ago that we are doubling down on our partnership with AWS. We're excited to have many, many customers on our shared product, the Hugging Face deep learning container on SageMaker. And we are working really closely with the Inferentia team and the Trainium team to release some more exciting stuff in the coming weeks and coming months. So I think when you have an ecosystem and a system where the AWS and the AI providers, AI startups can work hand in hand, it's to the benefit of the customers and the companies, because it makes it orders of magnitude easier for them to adopt this new paradigm to build technology AI. >> Ori, this is a scale on reasoning too. The data's out there and making sense out of it, making it reason, getting comprehension, having it make decisions is next, isn't it? And you need scale for that. >> Yes. Just a comment about the infrastructure side. So I think really the purpose is to streamline and make these technologies much more accessible. And I think we'll see, I predict that we'll see in the next few years more and more tooling that make this technology much more simple to consume. And I think it plays a very important role. There's so many aspects, like the monitoring the models and their kind of outputs they produce, and kind of containing and running them in a production environment. There's so much there to build on, the infrastructure side will play a very significant role. >> All right, that's awesome stuff. I'd love to change gears a little bit and get a little philosophy here around AI and how it's going to transform, if you guys don't mind. There's been a lot of conversations around, on theCUBE here as well as in some industry areas, where it's like, okay, all the heavy lifting is automated away with machine learning and AI, the complexity, there's some efficiencies, it's horizontal and scalable across all industries. Ankur, good point there. Everyone's going to use it for something. And a lot of stuff gets brought to the table with large language models and other things. But the key ingredient will be proprietary data or human input, or some sort of AI whisperer kind of role, or prompt engineering, people are saying. So with that being said, some are saying it's automating intelligence. And that creativity will be unleashed from this. If the heavy lifting goes away and AI can fill the void, that shifts the value to the intellect or the input. And so that means data's got to come together, interact, fuse, and understand each other. This is kind of new. I mean, old school AI was, okay, got a big model, I provisioned it long time, very expensive. Now it's all free flowing. Can you guys comment on where you see this going with this freeform, data flowing everywhere, heavy lifting, and then specialization? >> Yeah, I think- >> Go ahead. >> Yeah, I think, so what we are seeing with these large language models or generative models is that they're really good at creating stuff. But I think it's also important to recognize their limitations. They're not as good at reasoning and logic. And I think now we're seeing great enthusiasm, I think, which is justified. And the next phase would be how to make these systems more reliable. How to inject more reasoning capabilities into these models, or augment with other mechanisms that actually perform more reasoning so we can achieve more reliable results. And we can count on these models to perform for critical tasks, whether it's medical tasks, legal tasks. We really want to kind of offload a lot of the intelligence to these systems. And then we'll have to get back, we'll have to make sure these are reliable, we'll have to make sure we get some sort of explainability that we can understand the process behind the generated results that we received. So I think this is kind of the next phase of systems that are based on these generated models. >> Clem, what's your view on this? Obviously you're at open community, open source has been around, it's been a great track record, proven model. I'm assuming creativity's going to come out of the woodwork, and if we can automate open source contribution, and relationships, and onboarding more developers, there's going to be unleashing of creativity. >> Yes, it's been so exciting on the open source front. We all know Bert, Bloom, GPT-J, T5, Stable Diffusion, that work up. The previous or the current generation of open source models that are on Hugging Face. It has been accelerating in the past few months. So I'm super excited about ControlNet right now that is really having a lot of impact, which is kind of like a way to control the generation of images. Super excited about Flan UL2, which is like a new model that has been recently released and is open source. So yeah, it's really fun to see the ecosystem coming together. Open source has been the basis for traditional software, with like open source programming languages, of course, but also all the great open source that we've gotten over the years. So we're happy to see that the same thing is happening for machine learning and AI, and hopefully can help a lot of companies reduce a little bit the barrier to entry. So yeah, it's going to be exciting to see how it evolves in the next few years in that respect. >> I think the developer productivity angle that's been talked about a lot in the industry will be accelerated significantly. I think security will be enhanced by this. I think in general, applications are going to transform at a radical rate, accelerated, incredible rate. So I think it's not a big wave, it's the water, right? I mean, (chuckles) it's the new thing. My final question for you guys, if you don't mind, I'd love to get each of you to answer the question I'm going to ask you, which is, a lot of conversations around data. Data infrastructure's obviously involved in this. And the common thread that I'm hearing is that every company that looks at this is asking themselves, if we don't rebuild our company, start thinking about rebuilding our business model around AI, we might be dinosaurs, we might be extinct. And it reminds me that scene in Moneyball when, at the end, it's like, if we're not building the model around your model, every company will be out of business. What's your advice to companies out there that are having those kind of moments where it's like, okay, this is real, this is next gen, this is happening. I better start thinking and putting into motion plans to refactor my business, 'cause it's happening, business transformation is happening on the cloud. This kind of puts an exclamation point on, with the AI, as a next step function. Big increase in value. So it's an opportunity for leaders. Ankur, we'll start with you. What's your advice for folks out there thinking about this? Do they put their toe in the water? Do they jump right into the deep end? What's your advice? >> Yeah, John, so we talk to a lot of customers, and customers are excited about what's happening in the space, but they often ask us like, "Hey, where do we start?" So we always advise our customers to do a lot of proof of concepts, understand where they can drive the biggest ROI. And then also leverage existing tools and services to move fast and scale, and try and not reinvent the wheel where it doesn't need to be. That's basically our advice to customers. >> Get it. Ori, what's your advice to folks who are scratching their head going, "I better jump in here. "How do I get started?" What's your advice? >> So I actually think that need to think about it really economically. Both on the opportunity side and the challenges. So there's a lot of opportunities for many companies to actually gain revenue upside by building these new generative features and capabilities. On the other hand, of course, this would probably affect the cogs, and incorporating these capabilities could probably affect the cogs. So I think we really need to think carefully about both of these sides, and also understand clearly if this is a project or an F word towards cost reduction, then the ROI is pretty clear, or revenue amplifier, where there's, again, a lot of different opportunities. So I think once you think about this in a structured way, I think, and map the different initiatives, then it's probably a good way to start and a good way to start thinking about these endeavors. >> Awesome. Clem, what's your take on this? What's your advice, folks out there? >> Yes, all of these are very good advice already. Something that you said before, John, that I disagreed a little bit, a lot of people are talking about the data mode and proprietary data. Actually, when you look at some of the organizations that have been building the best models, they don't have specialized or unique access to data. So I'm not sure that's so important today. I think what's important for companies, and it's been the same for the previous generation of technology, is their ability to build better technology faster than others. And in this new paradigm, that means being able to build machine learning faster than others, and better. So that's how, in my opinion, you should approach this. And kind of like how can you evolve your company, your teams, your products, so that you are able in the long run to build machine learning better and faster than your competitors. And if you manage to put yourself in that situation, then that's when you'll be able to differentiate yourself to really kind of be impactful and get results. That's really hard to do. It's something really different, because machine learning and AI is a different paradigm than traditional software. So this is going to be challenging, but I think if you manage to nail that, then the future is going to be very interesting for your company. >> That's a great point. Thanks for calling that out. I think this all reminds me of the cloud days early on. If you went to the cloud early, you took advantage of it when the pandemic hit. If you weren't native in the cloud, you got hamstrung by that, you were flatfooted. So just get in there. (laughs) Get in the cloud, get into AI, you're going to be good. Thanks for for calling that. Final parting comments, what's your most exciting thing going on right now for you guys? Ori, Clem, what's the most exciting thing on your plate right now that you'd like to share with folks? >> I mean, for me it's just the diversity of use cases and really creative ways of companies leveraging this technology. Every day I speak with about two, three customers, and I'm continuously being surprised by the creative ideas. And the future is really exciting of what can be achieved here. And also I'm amazed by the pace that things move in this industry. It's just, there's not at dull moment. So, definitely exciting times. >> Clem, what are you most excited about right now? >> For me, it's all the new open source models that have been released in the past few weeks, and that they'll keep being released in the next few weeks. I'm also super excited about more and more companies getting into this capability of chaining different models and different APIs. I think that's a very, very interesting development, because it creates new capabilities, new possibilities, new functionalities that weren't possible before. You can plug an API with an open source embedding model, with like a no-geo transcription model. So that's also very exciting. This capability of having more interoperable machine learning will also, I think, open a lot of interesting things in the future. >> Clem, congratulations on your success at Hugging Face. Please pass that on to your team. Ori, congratulations on your success, and continue to, just day one. I mean, it's just the beginning. It's not even scratching the service. Ankur, I'll give you the last word. What are you excited for at AWS? More cloud goodness coming here with AI. Give you the final word. >> Yeah, so as both Clem and Ori said, I think the research in the space is moving really, really fast, so we are excited about that. But we are also excited to see the speed at which enterprises and other AWS customers are applying machine learning to solve real business problems, and the kind of results they're seeing. So when they come back to us and tell us the kind of improvement in their business metrics and overall customer experience that they're driving and they're seeing real business results, that's what keeps us going and inspires us to continue inventing on their behalf. >> Gentlemen, thank you so much for this awesome high impact panel. Ankur, Clem, Ori, congratulations on all your success. We'll see you around. Thanks for coming on. Generative AI, riding the wave, it's a tidal wave, it's the water, it's all happening. All great stuff. This is season three, episode one of AWS Startup Showcase closing panel. This is the AI ML episode, the top startups building generative AI on AWS. I'm John Furrier, your host. Thanks for watching. (mellow music)

(chill electronic music) >> Hello everyone, welcome to theCUBE's presentation of the AWS Startups Showcase, AI and machine learning, the top startups building generative AI on AWS. This is the season three, episode one of the ongoing series covering the exciting startups from the AWS ecosystem, talk about AI and machine learning. Can't believe it's three years and season one. I'm your host, John Furrier. Got a great guest today, we're joined by Joseph Nelson, the co-founder and CEO of Roboflow, doing some cutting edge stuff around computer vision and really at the front end of this massive wave coming around, large language models, computer vision. The next gen AI is here, and it's just getting started. We haven't even scratched a service. Thanks for joining us today. >> Thanks for having me. >> So you got to love the large language model, foundation models, really educating the mainstream world. ChatGPT has got everyone in the frenzy. This is educating the world around this next gen AI capabilities, enterprise, image and video data, all a big part of it. I mean the edge of the network, Mobile World Conference is happening right now, this month, and it's just ending up, it's just continue to explode. Video is huge. So take us through the company, do a quick explanation of what you guys are doing, when you were founded. Talk about what the company's mission is, and what's your North Star, why do you exist? >> Yeah, Roboflow exists to really kind of make the world programmable. I like to say make the world be read and write access. And our North Star is enabling developers, predominantly, to build that future. If you look around, anything that you see will have software related to it, and can kind of be turned into software. The limiting reactant though, is how to enable computers and machines to understand things as well as people can. And in a lot of ways, computer vision is that missing element that enables anything that you see to become software. So in the virtue of, if software is eating the world, computer vision kind of makes the aperture infinitely wide. It's something that I kind of like, the way I like to frame it. And the capabilities are there, the open source models are there, the amount of data is there, the computer capabilities are only improving annually, but there's a pretty big dearth of tooling, and an early but promising sign of the explosion of use cases, models, and data sets that companies, developers, hobbyists alike will need to bring these capabilities to bear. So Roboflow is in the game of building the community around that capability, building the use cases that allow developers and enterprises to use computer vision, and providing the tooling for companies and developers to be able to add computer vision, create better data sets, and deploy to production, quickly, easily, safely, invaluably. >> You know, Joseph, the word in production is actually real now. You're seeing a lot more people doing in production activities. That's a real hot one and usually it's slower, but it's gone faster, and I think that's going to be more the same. And I think the parallel between what we're seeing on the large language models coming into computer vision, and as you mentioned, video's data, right? I mean we're doing video right now, we're transcribing it into a transcript, linking up to your linguistics, times and the timestamp, I mean everything's data and that really kind of feeds. So this connection between what we're seeing, the large language and computer vision are coming together kind of cousins, brothers. I mean, how would you compare, how would you explain to someone, because everyone's like on this wave of watching people bang out their homework assignments, and you know, write some hacks on code with some of the open AI technologies, there is a corollary directly related to to the vision side. Can you explain? >> Yeah, the rise of large language models are showing what's possible, especially with text, and I think increasingly will get multimodal as the images and video become ingested. Though there's kind of this still core missing element of basically like understanding. So the rise of large language models kind of create this new area of generative AI, and generative AI in the context of computer vision is a lot of, you know, creating video and image assets and content. There's also this whole surface area to understanding what's already created. Basically digitizing physical, real world things. I mean the Metaverse can't be built if we don't know how to mirror or create or identify the objects that we want to interact with in our everyday lives. And where computer vision comes to play in, especially what we've seen at Roboflow is, you know, a little over a hundred thousand developers now have built with our tools. That's to the tune of a hundred million labeled open source images, over 10,000 pre-trained models. And they've kind of showcased to us all of the ways that computer vision is impacting and bringing the world to life. And these are things that, you know, even before large language models and generative AI, you had pretty impressive capabilities, and when you add the two together, it actually unlocks these kind of new capabilities. So for example, you know, one of our users actually powers the broadcast feeds at Wimbledon. So here we're talking about video, we're streaming, we're doing things live, we've got folks that are cropping and making sure we look good, and audio/visual all plugged in correctly. When you broadcast Wimbledon, you'll notice that the camera controllers need to do things like track the ball, which is moving at extremely high speeds and zoom crop, pan tilt, as well as determine if the ball bounced in or out. The very controversial but critical key to a lot of tennis matches. And a lot of that has been historically done with the trained, but fallible human eye and computer vision is, you know, well suited for this task to say, how do we track, pan, tilt, zoom, and see, track the tennis ball in real time, run at 30 plus frames per second, and do it all on the edge. And those are capabilities that, you know, were kind of like science fiction, maybe even a decade ago, and certainly five years ago. Now the interesting thing, is that with the advent of of generative AI, you can start to do things like create your own training data sets, or kind of create logic around once you have this visual input. And teams at Tesla have actually been speaking about, of course the autopilot team's focused on doing vision tasks, but they've combined large language models to add reasoning and logic. So given that you see, let's say the tennis ball, what do you want to do? And being able to combine the capabilities of what LLM's represent, which is really a lot of basically, core human reasoning and logic, with computer vision for the inputs of what's possible, creates these new capabilities, let alone multimodality, which I'm sure we'll talk more about. >> Yeah, and it's really, I mean it's almost intoxicating. It's amazing that this is so capable because the cloud scales here, you got the edge developing, you can decouple compute power, and let Moore's law and all the new silicone and the processors and the GPUs do their thing, and you got open source booming. You're kind of getting at this next segment I wanted to get into, which is the, how people should be thinking about these advances of the computer vision. So this is now a next wave, it's here. I mean I'd love to have that for baseball because I'm always like, "Oh, it should have been a strike." I'm sure that's going to be coming soon, but what is the computer vision capable of doing today? I guess that's my first question. You hit some of it, unpack that a little bit. What does general AI mean in computer vision? What's the new thing? Because there are old technology's been around, proprietary, bolted onto hardware, but hardware advances at a different pace, but now you got new capabilities, generative AI for vision, what does that mean? >> Yeah, so computer vision, you know, at its core is basically enabling machines, computers, to understand, process, and act on visual data as effective or more effective than people can. Traditionally this has been, you know, task types like classification, which you know, identifying if a given image belongs in a certain category of goods on maybe a retail site, is the shoes or is it clothing? Or object detection, which is, you know, creating bounding boxes, which allows you to do things like count how many things are present, or maybe measure the speed of something, or trigger an alert when something becomes visible in frame that wasn't previously visible in frame, or instant segmentation where you're creating pixel wise segmentations for both instance and semantic segmentation, where you often see these kind of beautiful visuals of the polygon surrounding objects that you see. Then you have key point detection, which is where you see, you know, athletes, and each of their joints are kind of outlined is another more traditional type problem in signal processing and computer vision. With generative AI, you kind of get a whole new class of problem types that are opened up. So in a lot of ways I think about generative AI in computer vision as some of the, you know, problems that you aimed to tackle, might still be better suited for one of the previous task types we were discussing. Some of those problem types may be better suited for using a generative technique, and some are problem types that just previously wouldn't have been possible absent generative AI. And so if you make that kind of Venn diagram in your head, you can think about, okay, you know, visual question answering is a task type where if I give you an image and I say, you know, "How many people are in this image?" We could either build an object detection model that might count all those people, or maybe a visual question answering system would sufficiently answer this type of problem. Let alone generative AI being able to create new training data for old systems. And that's something that we've seen be an increasingly prominent use case for our users, as much as things that we advise our customers and the community writ large to take advantage of. So ultimately those are kind of the traditional task types. I can give you some insight, maybe, into how I think about what's possible today, or five years or ten years as you sort go back. >> Yes, definitely. Let's get into that vision. >> So I kind of think about the types of use cases in terms of what's possible. If you just imagine a very simple bell curve, your normal distribution, for the longest time, the types of things that are in the center of that bell curve are identifying objects that are very common or common objects in context. Microsoft published the COCO Dataset in 2014 of common objects and contexts, of hundreds of thousands of images of chairs, forks, food, person, these sorts of things. And you know, the challenge of the day had always been, how do you identify just those 80 objects? So if we think about the bell curve, that'd be maybe the like dead center of the curve, where there's a lot of those objects present, and it's a very common thing that needs to be identified. But it's a very, very, very small sliver of the distribution. Now if you go out to the way long tail, let's go like deep into the tail of this imagined visual normal distribution, you're going to have a problem like one of our customers, Rivian, in tandem with AWS, is tackling, to do visual quality assurance and manufacturing in production processes. Now only Rivian knows what a Rivian is supposed to look like. Only they know the imagery of what their goods that are going to be produced are. And then between those long tails of proprietary data of highly specific things that need to be understood, in the center of the curve, you have a whole kind of messy middle, type of problems I like to say. The way I think about computer vision advancing, is it's basically you have larger and larger and more capable models that eat from the center out, right? So if you have a model that, you know, understands the 80 classes in COCO, well, pretty soon you have advances like Clip, which was trained on 400 million image text pairs, and has a greater understanding of a wider array of objects than just 80 classes in context. And over time you'll get more and more of these larger models that kind of eat outwards from that center of the distribution. And so the question becomes for companies, when can you rely on maybe a model that just already exists? How do you use your data to get what may be capable off the shelf, so to speak, into something that is usable for you? Or, if you're in those long tails and you have proprietary data, how do you take advantage of the greatest asset you have, which is observed visual information that you want to put to work for your customers, and you're kind of living in the long tails, and you need to adapt state of the art for your capabilities. So my mental model for like how computer vision advances is you have that bell curve, and you have increasingly powerful models that eat outward. And multimodality has a role to play in that, larger models have a role to play in that, more compute, more data generally has a role to play in that. But it will be a messy and I think long condition. >> Well, the thing I want to get, first of all, it's great, great mental model, I appreciate that, 'cause I think that makes a lot of sense. The question is, it seems now more than ever, with the scale and compute that's available, that not only can you eat out to the middle in your example, but there's other models you can integrate with. In the past there was siloed, static, almost bespoke. Now you're looking at larger models eating into the bell curve, as you said, but also integrating in with other stuff. So this seems to be part of that interaction. How does, first of all, is that really happening? Is that true? And then two, what does that mean for companies who want to take advantage of this? Because the old model was operational, you know? I have my cameras, they're watching stuff, whatever, and like now you're in this more of a, distributed computing, computer science mindset, not, you know, put the camera on the wall kind of- I'm oversimplifying, but you know what I'm saying. What's your take on that? >> Well, to the first point of, how are these advances happening? What I was kind of describing was, you know, almost uni-dimensional in that you have like, you're only thinking about vision, but the rise of generative techniques and multi-modality, like Clip is a multi-modal model, it has 400 million image text pairs. That will advance the generalizability at a faster rate than just treating everything as only vision. And that's kind of where LLMs and vision will intersect in a really nice and powerful way. Now in terms of like companies, how should they be thinking about taking advantage of these trends? The biggest thing that, and I think it's different, obviously, on the size of business, if you're an enterprise versus a startup. The biggest thing that I think if you're an enterprise, and you have an established scaled business model that is working for your customers, the question becomes, how do you take advantage of that established data moat, potentially, resource moats, and certainly, of course, establish a way of providing value to an end user. So for example, one of our customers, Walmart, has the advantage of one of the largest inventory and stock of any company in the world. And they also of course have substantial visual data, both from like their online catalogs, or understanding what's in stock or out of stock, or understanding, you know, the quality of things that they're going from the start of their supply chain to making it inside stores, for delivery of fulfillments. All these are are visual challenges. Now they already have a substantial trove of useful imagery to understand and teach and train large models to understand each of the individual SKUs and products that are in their stores. And so if I'm a Walmart, what I'm thinking is, how do I make sure that my petabytes of visual information is utilized in a way where I capture the proprietary benefit of the models that I can train to do tasks like, what item was this? Or maybe I'm going to create AmazonGo-like technology, or maybe I'm going to build like delivery robots, or I want to automatically know what's in and out of stock from visual input fees that I have across my in-store traffic. And that becomes the question and flavor of the day for enterprises. I've got this large amount of data, I've got an established way that I can provide more value to my own customers. How do I ensure I take advantage of the data advantage I'm already sitting on? If you're a startup, I think it's a pretty different question, and I'm happy to talk about. >> Yeah, what's startup angle on this? Because you know, they're going to want to take advantage. It's like cloud startups, cloud native startups, they were born in the cloud, they never had an IT department. So if you're a startup, is there a similar role here? And if I'm a computer vision startup, what's that mean? So can you share your your take on that, because there'll be a lot of people starting up from this. >> So the startup on the opposite advantage and disadvantage, right? Like a startup doesn't have an proven way of delivering repeatable value in the same way that a scaled enterprise does. But it does have the nimbleness to identify and take advantage of techniques that you can start from a blank slate. And I think the thing that startups need to be wary of in the generative AI enlarged language model, in multimodal world, is building what I like to call, kind of like sandcastles. A sandcastle is maybe a business model or a capability that's built on top of an assumption that is going to be pretty quickly wiped away by improving underlying model technology. So almost like if you imagine like the ocean, the waves are coming in, and they're going to wipe away your progress. You don't want to be in the position of building sandcastle business where, you don't want to bet on the fact that models aren't going to get good enough to solve the task type that you might be solving. In other words, don't take a screenshot of what's capable today. Assume that what's capable today is only going to continue to become possible. And so for a startup, what you can do, that like enterprises are quite comparatively less good at, is embedding these capabilities deeply within your products and delivering maybe a vertical based experience, where AI kind of exists in the background. >> Yeah. >> And we might not think of companies as, you know, even AI companies, it's just so embedded in the experience they provide, but that's like the vertical application example of taking AI and making it be immediately usable. Or, of course there's tons of picks and shovels businesses to be built like Roboflow, where you're enabling these enterprises to take advantage of something that they have, whether that's their data sets, their computes, or their intellect. >> Okay, so if I hear that right, by the way, I love, that's horizontally scalable, that's the large language models, go up and build them the apps, hence your developer focus. I'm sure that's probably the reason that the tsunami of developer's action. So you're saying picks and shovels tools, don't try to replicate the platform of what could be the platform. Oh, go to a VC, I'm going to build a platform. No, no, no, no, those are going to get wiped away by the large language models. Is there one large language model that will rule the world, or do you see many coming? >> Yeah, so to be clear, I think there will be useful platforms. I just think a lot of people think that they're building, let's say, you know, if we put this in the cloud context, you're building a specific type of EC2 instance. Well, it turns out that Amazon can offer that type of EC2 instance, and immediately distribute it to all of their customers. So you don't want to be in the position of just providing something that actually ends up looking like a feature, which in the context of AI, might be like a small incremental improvement on the model. If that's all you're doing, you're a sandcastle business. Now there's a lot of platform businesses that need to be built that enable businesses to get to value and do things like, how do I monitor my models? How do I create better models with my given data sets? How do I ensure that my models are doing what I want them to do? How do I find the right models to use? There's all these sorts of platform wide problems that certainly exist for businesses. I just think a lot of startups that I'm seeing right now are making the mistake of assuming the advances we're seeing are not going to accelerate or even get better. >> So if I'm a customer, if I'm a company, say I'm a startup or an enterprise, either one, same question. And I want to stand up, and I have developers working on stuff, I want to start standing up an environment to start doing stuff. Is that a service provider? Is that a managed service? Is that you guys? So how do you guys fit into your customers leaning in? Is it just for developers? Are you targeting with a specific like managed service? What's the product consumption? How do you talk to customers when they come to you? >> The thing that we do is enable, we give developers superpowers to build automated inventory tracking, self-checkout systems, identify if this image is malignant cancer or benign cancer, ensure that these products that I've produced are correct. Make sure that that the defect that might exist on this electric vehicle makes its way back for review. All these sorts of problems are immediately able to be solved and tackled. In terms of the managed services element, we have solutions as integrators that will often build on top of our tools, or we'll have companies that look to us for guidance, but ultimately the company is in control of developing and building and creating these capabilities in house. I really think the distinction is maybe less around managed service and tool, and more around ownership in the era of AI. So for example, if I'm using a managed service, in that managed service, part of their benefit is that they are learning across their customer sets, then it's a very different relationship than using a managed service where I'm developing some amount of proprietary advantages for my data sets. And I think that's a really important thing that companies are becoming attuned to, just the value of the data that they have. And so that's what we do. We tell companies that you have this proprietary, immense treasure trove of data, use that to your advantage, and think about us more like a set of tools that enable you to get value from that capability. You know, the HashiCorp's and GitLab's of the world have proven like what these businesses look like at scale. >> And you're targeting developers. When you go into a company, do you target developers with freemium, is there a paid service? Talk about the business model real quick. >> Sure, yeah. The tools are free to use and get started. When someone signs up for Roboflow, they may elect to make their work open source, in which case we're able to provide even more generous usage limits to basically move the computer vision community forward. If you elect to make your data private, you can use our hosted data set managing, data set training, model deployment, annotation tooling up to some limits. And then usually when someone validates that what they're doing gets them value, they purchase a subscription license to be able to scale up those capabilities. So like most developer centric products, it's free to get started, free to prove, free to poke around, develop what you think is possible. And then once you're getting to value, then we're able to capture the commercial upside in the value that's being provided. >> Love the business model. It's right in line with where the market is. There's kind of no standards bodies these days. The developers are the ones who are deciding kind of what the standards are by their adoption. I think making that easy for developers to get value as the model open sources continuing to grow, you can see more of that. Great perspective Joseph, thanks for sharing that. Put a plug in for the company. What are you guys doing right now? Where are you in your growth? What are you looking for? How should people engage? Give the quick commercial for the company. >> So as I mentioned, Roboflow is I think one of the largest, if not the largest collections of computer vision models and data sets that are open source, available on the web today, and have a private set of tools that over half the Fortune 100 now rely on those tools. So we're at the stage now where we know people want what we're working on, and we're continuing to drive that type of adoption. So companies that are looking to make better models, improve their data sets, train and deploy, often will get a lot of value from our tools, and certainly reach out to talk. I'm sure there's a lot of talented engineers that are tuning in too, we're aggressively hiring. So if you are interested in being a part of making the world programmable, and being at the ground floor of the company that's creating these capabilities to be writ large, we'd love to hear from you. >> Amazing, Joseph, thanks so much for coming on and being part of the AWS Startup Showcase. Man, if I was in my twenties, I'd be knocking on your door, because it's the hottest trend right now, it's super exciting. Generative AI is just the beginning of massive sea change. Congratulations on all your success, and we'll be following you guys. Thanks for spending the time, really appreciate it. >> Thanks for having me. >> Okay, this is season three, episode one of the ongoing series covering the exciting startups from the AWS ecosystem, talking about the hottest things in tech. I'm John Furrier, your host. Thanks for watching. (chill electronic music)

(upbeat music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase AI Machine Learning Top Startups Building Generative AI on AWS. This is season 3, episode 1 of the ongoing series covering the exciting startup from the AWS ecosystem to talk about AI and machine learning. I'm your host, John Furrier. I'm joined by two great guests here, Adam Wenchel, who's the CEO of Arthur, and Chief Scientist of Arthur, John Dickerson. Talk about how they help people build better LLM AI systems to get them into the market faster. Gentlemen, thank you for coming on. >> Yeah, thanks for having us, John. >> Well, I got to say I got to temper my enthusiasm because the last few months explosion of interest in LLMs with ChatGPT, has opened the eyes to everybody around the reality of that this is going next gen, this is it, this is the moment, this is the the point we're going to look back and say, this is the time where AI really hit the scene for real applications. So, a lot of Large Language Models, also known as LLMs, foundational models, and generative AI is all booming. This is where all the alpha developers are going. This is where everyone's focusing their business model transformations on. This is where developers are seeing action. So it's all happening, the wave is here. So I got to ask you guys, what are you guys seeing right now? You're in the middle of it, it's hitting you guys right on. You're in the front end of this massive wave. >> Yeah, John, I don't think you have to temper your enthusiasm at all. I mean, what we're seeing every single day is, everything from existing enterprise customers coming in with new ways that they're rethinking, like business things that they've been doing for many years that they can now do an entirely different way, as well as all manner of new companies popping up, applying LLMs to everything from generating code and SQL statements to generating health transcripts and just legal briefs. Everything you can imagine. And when you actually sit down and look at these systems and the demos we get of them, the hype is definitely justified. It's pretty amazing what they're going to do. And even just internally, we built, about a month ago in January, we built an Arthur chatbot so customers could ask questions, technical questions from our, rather than read our product documentation, they could just ask this LLM a particular question and get an answer. And at the time it was like state of the art, but then just last week we decided to rebuild it because the tooling has changed so much that we, last week, we've completely rebuilt it. It's now way better, built on an entirely different stack. And the tooling has undergone a full generation worth of change in six weeks, which is crazy. So it just tells you how much energy is going into this and how fast it's evolving right now. >> John, weigh in as a chief scientist. I mean, you must be blown away. Talk about kid in the candy store. I mean, you must be looking like this saying, I mean, she must be super busy to begin with, but the change, the acceleration, can you scope the kind of change you're seeing and be specific around the areas you're seeing movement and highly accelerated change? >> Yeah, definitely. And it is very, very exciting actually, thinking back to when ChatGPT was announced, that was a night our company was throwing an event at NeurIPS, which is maybe the biggest machine learning conference out there. And the hype when that happened was palatable and it was just shocking to see how well that performed. And then obviously over the last few months since then, as LLMs have continued to enter the market, we've seen use cases for them, like Adam mentioned all over the place. And so, some things I'm excited about in this space are the use of LLMs and more generally, foundation models to redesign traditional operations, research style problems, logistics problems, like auctions, decisioning problems. So moving beyond the already amazing news cases, like creating marketing content into more core integration and a lot of the bread and butter companies and tasks that drive the American ecosystem. And I think we're just starting to see some of that. And in the next 12 months, I think we're going to see a lot more. If I had to make other predictions, I think we're going to continue seeing a lot of work being done on managing like inference time costs via shrinking models or distillation. And I don't know how to make this prediction, but at some point we're going to be seeing lots of these very large scale models operating on the edge as well. So the time scales are extremely compressed, like Adam mentioned, 12 months from now, hard to say. >> We were talking on theCUBE prior to this session here. We had theCUBE conversation here and then the Wall Street Journal just picked up on the same theme, which is the printing press moment created the enlightenment stage of the history. Here we're in the whole nother automating intellect efficiency, doing heavy lifting, the creative class coming back, a whole nother level of reality around the corner that's being hyped up. The question is, is this justified? Is there really a breakthrough here or is this just another result of continued progress with AI? Can you guys weigh in, because there's two schools of thought. There's the, "Oh my God, we're entering a new enlightenment tech phase, of the equivalent of the printing press in all areas. Then there's, Ah, it's just AI (indistinct) inch by inch. What's your guys' opinion? >> Yeah, I think on the one hand when you're down in the weeds of building AI systems all day, every day, like we are, it's easy to look at this as an incremental progress. Like we have customers who've been building on foundation models since we started the company four years ago, particular in computer vision for classification tasks, starting with pre-trained models, things like that. So that part of it doesn't feel real new, but what does feel new is just when you apply these things to language with all the breakthroughs and computational efficiency, algorithmic improvements, things like that, when you actually sit down and interact with ChatGPT or one of the other systems that's out there that's building on top of LLMs, it really is breathtaking, like, the level of understanding that they have and how quickly you can accelerate your development efforts and get an actual working system in place that solves a really important real world problem and makes people way faster, way more efficient. So I do think there's definitely something there. It's more than just incremental improvement. This feels like a real trajectory inflection point for the adoption of AI. >> John, what's your take on this? As people come into the field, I'm seeing a lot of people move from, hey, I've been coding in Python, I've been doing some development, I've been a software engineer, I'm a computer science student. I'm coding in C++ old school, OG systems person. Where do they come in? Where's the focus, where's the action? Where are the breakthroughs? Where are people jumping in and rolling up their sleeves and getting dirty with this stuff? >> Yeah, all over the place. And it's funny you mentioned students in a different life. I wore a university professor hat and so I'm very, very familiar with the teaching aspects of this. And I will say toward Adam's point, this really is a leap forward in that techniques like in a co-pilot for example, everybody's using them right now and they really do accelerate the way that we develop. When I think about the areas where people are really, really focusing right now, tooling is certainly one of them. Like you and I were chatting about LangChain right before this interview started, two or three people can sit down and create an amazing set of pipes that connect different aspects of the LLM ecosystem. Two, I would say is in engineering. So like distributed training might be one, or just understanding better ways to even be able to train large models, understanding better ways to then distill them or run them. So like this heavy interaction now between engineering and what I might call traditional machine learning from 10 years ago where you had to know a lot of math, you had to know calculus very well, things like that. Now you also need to be, again, a very strong engineer, which is exciting. >> I interviewed Swami when he talked about the news. He's ahead of Amazon's machine learning and AI when they announced Hugging Face announcement. And I reminded him how Amazon was easy to get into if you were developing a startup back in 2007,8, and that the language models had that similar problem. It's step up a lot of content and a lot of expense to get provisioned up, now it's easy. So this is the next wave of innovation. So how do you guys see that from where we are right now? Are we at that point where it's that moment where it's that cloud-like experience for LLMs and large language models? >> Yeah, go ahead John. >> I think the answer is yes. We see a number of large companies that are training these and serving these, some of which are being co-interviewed in this episode. I think we're at that. Like, you can hit one of these with a simple, single line of Python, hitting an API, you can boot this up in seconds if you want. It's easy. >> Got it. >> So I (audio cuts out). >> Well let's take a step back and talk about the company. You guys being featured here on the Showcase. Arthur, what drove you to start the company? How'd this all come together? What's the origination story? Obviously you got a big customers, how'd get started? What are you guys doing? How do you make money? Give a quick overview. >> Yeah, I think John and I come at it from slightly different angles, but for myself, I have been a part of a number of technology companies. I joined Capital One, they acquired my last company and shortly after I joined, they asked me to start their AI team. And so even though I've been doing AI for a long time, I started my career back in DARPA. It was the first time I was really working at scale in AI at an organization where there were hundreds of millions of dollars in revenue at stake with the operation of these models and that they were impacting millions of people's financial livelihoods. And so it just got me hyper-focused on these issues around making sure that your AI worked well and it worked well for your company and it worked well for the people who were being affected by it. At the time when I was doing this 2016, 2017, 2018, there just wasn't any tooling out there to support this production management model monitoring life phase of the life cycle. And so we basically left to start the company that I wanted. And John has a his own story. I'll let let you share that one, John. >> Go ahead John, you're up. >> Yeah, so I'm coming at this from a different world. So I'm on leave now from a tenured role in academia where I was leading a large lab focusing on the intersection of machine learning and economics. And so questions like fairness or the response to the dynamism on the underlying environment have been around for quite a long time in that space. And so I've been thinking very deeply about some of those more like R and D style questions as well as having deployed some automation code across a couple of different industries, some in online advertising, some in the healthcare space and so on, where concerns of, again, fairness come to bear. And so Adam and I connected to understand the space of what that might look like in the 2018 20 19 realm from a quantitative and from a human-centered point of view. And so booted things up from there. >> Yeah, bring that applied engineering R and D into the Capital One, DNA that he had at scale. I could see that fit. I got to ask you now, next step, as you guys move out and think about LLMs and the recent AI news around the generative models and the foundational models like ChatGPT, how should we be looking at that news and everyone watching might be thinking the same thing. I know at the board level companies like, we should refactor our business, this is the future. It's that kind of moment, and the tech team's like, okay, boss, how do we do this again? Or are they prepared? How should we be thinking? How should people watching be thinking about LLMs? >> Yeah, I think they really are transformative. And so, I mean, we're seeing companies all over the place. Everything from large tech companies to a lot of our large enterprise customers are launching significant projects at core parts of their business. And so, yeah, I would be surprised, if you're serious about becoming an AI native company, which most leading companies are, then this is a trend that you need to be taking seriously. And we're seeing the adoption rate. It's funny, I would say the AI adoption in the broader business world really started, let's call it four or five years ago, and it was a relatively slow adoption rate, but I think all that kind of investment in and scaling the maturity curve has paid off because the rate at which people are adopting and deploying systems based on this is tremendous. I mean, this has all just happened in the few months and we're already seeing people get systems into production. So, now there's a lot of things you have to guarantee in order to put these in production in a way that basically is added into your business and doesn't cause more headaches than it solves. And so that's where we help customers is where how do you put these out there in a way that they're going to represent your company well, they're going to perform well, they're going to do their job and do it properly. >> So in the use case, as a customer, as I think about this, there's workflows. They might have had an ML AI ops team that's around IT. Their inference engines are out there. They probably don't have a visibility on say how much it costs, they're kicking the tires. When you look at the deployment, there's a cost piece, there's a workflow piece, there's fairness you mentioned John, what should be, I should be thinking about if I'm going to be deploying stuff into production, I got to think about those things. What's your opinion? >> Yeah, I'm happy to dive in on that one. So monitoring in general is extremely important once you have one of these LLMs in production, and there have been some changes versus traditional monitoring that we can dive deeper into that LLMs are really accelerated. But a lot of that bread and butter style of things you should be looking out for remain just as important as they are for what you might call traditional machine learning models. So the underlying environment of data streams, the way users interact with these models, these are all changing over time. And so any performance metrics that you care about, traditional ones like an accuracy, if you can define that for an LLM, ones around, for example, fairness or bias. If that is a concern for your particular use case and so on. Those need to be tracked. Now there are some interesting changes that LLMs are bringing along as well. So most ML models in production that we see are relatively static in the sense that they're not getting flipped in more than maybe once a day or once a week or they're just set once and then not changed ever again. With LLMs, there's this ongoing value alignment or collection of preferences from users that is often constantly updating the model. And so that opens up all sorts of vectors for, I won't say attack, but for problems to arise in production. Like users might learn to use your system in a different way and thus change the way those preferences are getting collected and thus change your system in ways that you never intended. So maybe that went through governance already internally at the company and now it's totally, totally changed and it's through no fault of your own, but you need to be watching over that for sure. >> Talk about the reinforced learnings from human feedback. How's that factoring in to the LLMs? Is that part of it? Should people be thinking about that? Is that a component that's important? >> It certainly is, yeah. So this is one of the big tweaks that happened with InstructGPT, which is the basis model behind ChatGPT and has since gone on to be used all over the place. So value alignment I think is through RLHF like you mentioned is a very interesting space to get into and it's one that you need to watch over. Like, you're asking humans for feedback over outputs from a model and then you're updating the model with respect to that human feedback. And now you've thrown humans into the loop here in a way that is just going to complicate things. And it certainly helps in many ways. You can ask humans to, let's say that you're deploying an internal chat bot at an enterprise, you could ask humans to align that LLM behind the chatbot to, say company values. And so you're listening feedback about these company values and that's going to scoot that chatbot that you're running internally more toward the kind of language that you'd like to use internally on like a Slack channel or something like that. Watching over that model I think in that specific case, that's a compliance and HR issue as well. So while it is part of the greater LLM stack, you can also view that as an independent bit to watch over. >> Got it, and these are important factors. When people see the Bing news, they freak out how it's doing great. Then it goes off the rails, it goes big, fails big. (laughing) So these models people see that, is that human interaction or is that feedback, is that not accepting it or how do people understand how to take that input in and how to build the right apps around LLMs? This is a tough question. >> Yeah, for sure. So some of the examples that you'll see online where these chatbots go off the rails are obviously humans trying to break the system, but some of them clearly aren't. And that's because these are large statistical models and we don't know what's going to pop out of them all the time. And even if you're doing as much in-house testing at the big companies like the Go-HERE's and the OpenAI's of the world, to try to prevent things like toxicity or racism or other sorts of bad content that might lead to bad pr, you're never going to catch all of these possible holes in the model itself. And so, again, it's very, very important to keep watching over that while it's in production. >> On the business model side, how are you guys doing? What's the approach? How do you guys engage with customers? Take a minute to explain the customer engagement. What do they need? What do you need? How's that work? >> Yeah, I can talk a little bit about that. So it's really easy to get started. It's literally a matter of like just handing out an API key and people can get started. And so we also offer alternative, we also offer versions that can be installed on-prem for models that, we find a lot of our customers have models that deal with very sensitive data. So you can run it in your cloud account or use our cloud version. And so yeah, it's pretty easy to get started with this stuff. We find people start using it a lot of times during the validation phase 'cause that way they can start baselining performance models, they can do champion challenger, they can really kind of baseline the performance of, maybe they're considering different foundation models. And so it's a really helpful tool for understanding differences in the way these models perform. And then from there they can just flow that into their production inferencing, so that as these systems are out there, you have really kind of real time monitoring for anomalies and for all sorts of weird behaviors as well as that continuous feedback loop that helps you make make your product get better and observability and you can run all sorts of aggregated reports to really understand what's going on with these models when they're out there deciding. I should also add that we just today have another way to adopt Arthur and that is we are in the AWS marketplace, and so we are available there just to make it that much easier to use your cloud credits, skip the procurement process, and get up and running really quickly. >> And that's great 'cause Amazon's got SageMaker, which handles a lot of privacy stuff, all kinds of cool things, or you can get down and dirty. So I got to ask on the next one, production is a big deal, getting stuff into production. What have you guys learned that you could share to folks watching? Is there a cost issue? I got to monitor, obviously you brought that up, we talked about the even reinforcement issues, all these things are happening. What is the big learnings that you could share for people that are going to put these into production to watch out for, to plan for, or be prepared for, hope for the best plan for the worst? What's your advice? >> I can give a couple opinions there and I'm sure Adam has. Well, yeah, the big one from my side is, again, I had mentioned this earlier, it's just the input data streams because humans are also exploring how they can use these systems to begin with. It's really, really hard to predict the type of inputs you're going to be seeing in production. Especially, we always talk about chatbots, but then any generative text tasks like this, let's say you're taking in news articles and summarizing them or something like that, it's very hard to get a good sampling even of the set of news articles in such a way that you can really predict what's going to pop out of that model. So to me, it's, adversarial maybe isn't the word that I would use, but it's an unnatural shifting input distribution of like prompts that you might see for these models. That's certainly one. And then the second one that I would talk about is, it can be hard to understand the costs, the inference time costs behind these LLMs. So the pricing on these is always changing as the models change size, it might go up, it might go down based on model size, based on energy cost and so on, but your pricing per token or per a thousand tokens and that I think can be difficult for some clients to wrap their head around. Again, you don't know how these systems are going to be used after all so it can be tough. And so again that's another metric that really should be tracked. >> Yeah, and there's a lot of trade off choices in there with like, how many tokens do you want at each step and in the sequence and based on, you have (indistinct) and you reject these tokens and so based on how your system's operating, that can make the cost highly variable. And that's if you're using like an API version that you're paying per token. A lot of people also choose to run these internally and as John mentioned, the inference time on these is significantly higher than a traditional classifi, even NLP classification model or tabular data model, like orders of magnitude higher. And so you really need to understand how that, as you're constantly iterating on these models and putting out new versions and new features in these models, how that's affecting the overall scale of that inference cost because you can use a lot of computing power very quickly with these profits. >> Yeah, scale, performance, price all come together. I got to ask while we're here on the secret sauce of the company, if you had to describe to people out there watching, what's the secret sauce of the company? What's the key to your success? >> Yeah, so John leads our research team and they've had a number of really cool, I think AI as much as it's been hyped for a while, it's still commercial AI at least is really in its infancy. And so the way we're able to pioneer new ways to think about performance for computer vision NLP LLMs is probably the thing that I'm proudest about. John and his team publish papers all the time at Navs and other places. But I think it's really being able to define what performance means for basically any kind of model type and give people really powerful tools to understand that on an ongoing basis. >> John, secret sauce, how would you describe it? You got all the action happening all around you. >> Yeah, well I going to appreciate Adam talking me up like that. No, I. (all laughing) >> Furrier: Robs to you. >> I would also say a couple of other things here. So we have a very strong engineering team and so I think some early hires there really set the standard at a very high bar that we've maintained as we've grown. And I think that's really paid dividends as scalabilities become even more of a challenge in these spaces, right? And so that's not just scalability when it comes to LLMs, that's scalability when it comes to millions of inferences per day, that kind of thing as well in traditional ML models. And I think that's compared to potential competitors, that's really... Well, it's made us able to just operate more efficiently and pass that along to the client. >> Yeah, and I think the infancy comment is really important because it's the beginning. You really is a long journey ahead. A lot of change coming, like I said, it's a huge wave. So I'm sure you guys got a lot of plannings at the foundation even for your own company, so I appreciate the candid response there. Final question for you guys is, what should the top things be for a company in 2023? If I'm going to set the agenda and I'm a customer moving forward, putting the pedal to the metal, so to speak, what are the top things I should be prioritizing or I need to do to be successful with AI in 2023? >> Yeah, I think, so number one, as we talked about, we've been talking about this entire episode, the things are changing so quickly and the opportunities for business transformation and really disrupting different applications, different use cases, is almost, I don't think we've even fully comprehended how big it is. And so really digging in to your business and understanding where I can apply these new sets of foundation models is, that's a top priority. The interesting thing is I think there's another force at play, which is the macroeconomic conditions and a lot of places are, they're having to work harder to justify budgets. So in the past, couple years ago maybe, they had a blank check to spend on AI and AI development at a lot of large enterprises that was limited primarily by the amount of talent they could scoop up. Nowadays these expenditures are getting scrutinized more. And so one of the things that we really help our customers with is like really calculating the ROI on these things. And so if you have models out there performing and you have a new version that you can put out that lifts the performance by 3%, how many tens of millions of dollars does that mean in business benefit? Or if I want to go to get approval from the CFO to spend a few million dollars on this new project, how can I bake in from the beginning the tools to really show the ROI along the way? Because I think in these systems when done well for a software project, the ROI can be like pretty spectacular. Like we see over a hundred percent ROI in the first year on some of these projects. And so, I think in 2023, you just need to be able to show what you're getting for that spend. >> It's a needle moving moment. You see it all the time with some of these aha moments or like, whoa, blown away. John, I want to get your thoughts on this because one of the things that comes up a lot for companies that I talked to, that are on my second wave, I would say coming in, maybe not, maybe the front wave of adopters is talent and team building. You mentioned some of the hires you got were game changing for you guys and set the bar high. As you move the needle, new developers going to need to come in. What's your advice given that you've been a professor, you've seen students, I know a lot of computer science people want to shift, they might not be yet skilled in AI, but they're proficient in programming, is that's going to be another opportunity with open source when things are happening. How do you talk to that next level of talent that wants to come in to this market to supplement teams and be on teams, lead teams? Any advice you have for people who want to build their teams and people who are out there and want to be a coder in AI? >> Yeah, I've advice, and this actually works for what it would take to be a successful AI company in 2023 as well, which is, just don't be afraid to iterate really quickly with these tools. The space is still being explored on what they can be used for. A lot of the tasks that they're used for now right? like creating marketing content using a machine learning is not a new thing to do. It just works really well now. And so I'm excited to see what the next year brings in terms of folks from outside of core computer science who are, other engineers or physicists or chemists or whatever who are learning how to use these increasingly easy to use tools to leverage LLMs for tasks that I think none of us have really thought about before. So that's really, really exciting. And so toward that I would say iterate quickly. Build things on your own, build demos, show them the friends, host them online and you'll learn along the way and you'll have somebody to show for it. And also you'll help us explore that space. >> Guys, congratulations with Arthur. Great company, great picks and shovels opportunities out there for everybody. Iterate fast, get in quickly and don't be afraid to iterate. Great advice and thank you for coming on and being part of the AWS showcase, thanks. >> Yeah, thanks for having us on John. Always a pleasure. >> Yeah, great stuff. Adam Wenchel, John Dickerson with Arthur. Thanks for coming on theCUBE. I'm John Furrier, your host. Generative AI and AWS. Keep it right there for more action with theCUBE. Thanks for watching. (upbeat music)

(upbeat music) >> Hello, everyone, and welcome to theCUBE's presentation of the "AWS Startup Showcase." This is season three, episode one. The focus of this episode is AI/ML: Top Startups Building Foundational Models, Infrastructure, and AI. It's great topics, super-relevant, and it's part of our ongoing coverage of startups in the AWS ecosystem. I'm your host, John Furrier, with theCUBE. Today, we're excited to be joined by Jay Marshall, VP of Business Development at Neural Magic. Jay, thanks for coming on theCUBE. >> Hey, John, thanks so much. Thanks for having us. >> We had a great CUBE conversation with you guys. This is very much about the company focuses. It's a feature presentation for the "Startup Showcase," and the machine learning at scale is the topic, but in general, it's more, (laughs) and we should call it "Machine Learning and AI: How to Get Started," because everybody is retooling their business. Companies that aren't retooling their business right now with AI first will be out of business, in my opinion. You're seeing massive shift. This is really truly the beginning of the next-gen machine learning AI trend. It's really seeing ChatGPT. Everyone sees that. That went mainstream. But this is just the beginning. This is scratching the surface of this next-generation AI with machine learning powering it, and with all the goodness of cloud, cloud scale, and how horizontally scalable it is. The resources are there. You got the Edge. Everything's perfect for AI 'cause data infrastructure's exploding in value. AI is just the applications. This is a super topic, so what do you guys see in this general area of opportunities right now in the headlines? And I'm sure you guys' phone must be ringing off the hook, metaphorically speaking, or emails and meetings and Zooms. What's going on over there at Neural Magic? >> No, absolutely, and you pretty much nailed most of it. I think that, you know, my background, we've seen for the last 20-plus years. Even just getting enterprise applications kind of built and delivered at scale, obviously, amazing things with AWS and the cloud to help accelerate that. And we just kind of figured out in the last five or so years how to do that productively and efficiently, kind of from an operations perspective. Got development and operations teams. We even came up with DevOps, right? But now, we kind of have this new kind of persona and new workload that developers have to talk to, and then it has to be deployed on those ITOps solutions. And so you pretty much nailed it. Folks are saying, "Well, how do I do this?" These big, generational models or foundational models, as we're calling them, they're great, but enterprises want to do that with their data, on their infrastructure, at scale, at the edge. So for us, yeah, we're helping enterprises accelerate that through optimizing models and then delivering them at scale in a more cost-effective fashion. >> Yeah, and I think one of the things, the benefits of OpenAI we saw, was not only is it open source, then you got also other models that are more proprietary, is that it shows the world that this is really happening, right? It's a whole nother level, and there's also new landscape kind of maps coming out. You got the generative AI, and you got the foundational models, large LLMs. Where do you guys fit into the landscape? Because you guys are in the middle of this. How do you talk to customers when they say, "I'm going down this road. I need help. I'm going to stand this up." This new AI infrastructure and applications, where do you guys fit in the landscape? >> Right, and really, the answer is both. I think today, when it comes to a lot of what for some folks would still be considered kind of cutting edge around computer vision and natural language processing, a lot of our optimization tools and our runtime are based around most of the common computer vision and natural language processing models. So your YOLOs, your BERTs, you know, your DistilBERTs and what have you, so we work to help optimize those, again, who've gotten great performance and great value for customers trying to get those into production. But when you get into the LLMs, and you mentioned some of the open source components there, our research teams have kind of been right in the trenches with those. So kind of the GPT open source equivalent being OPT, being able to actually take, you know, a multi-$100 billion parameter model and sparsify that or optimize that down, shaving away a ton of parameters, and being able to run it on smaller infrastructure. So I think the evolution here, you know, all this stuff came out in the last six months in terms of being turned loose into the wild, but we're staying in the trenches with folks so that we can help optimize those as well and not require, again, the heavy compute, the heavy cost, the heavy power consumption as those models evolve as well. So we're staying right in with everybody while they're being built, but trying to get folks into production today with things that help with business value today. >> Jay, I really appreciate you coming on theCUBE, and before we came on camera, you said you just were on a customer call. I know you got a lot of activity. What specific things are you helping enterprises solve? What kind of problems? Take us through the spectrum from the beginning, people jumping in the deep end of the pool, some people kind of coming in, starting out slow. What are the scale? Can you scope the kind of use cases and problems that are emerging that people are calling you for? >> Absolutely, so I think if I break it down to kind of, like, your startup, or I maybe call 'em AI native to kind of steal from cloud native years ago, that group, it's pretty much, you know, part and parcel for how that group already runs. So if you have a data science team and an ML engineering team, you're building models, you're training models, you're deploying models. You're seeing firsthand the expense of starting to try to do that at scale. So it's really just a pure operational efficiency play. They kind of speak natively to our tools, which we're doing in the open source. So it's really helping, again, with the optimization of the models they've built, and then, again, giving them an alternative to expensive proprietary hardware accelerators to have to run them. Now, on the enterprise side, it varies, right? You have some kind of AI native folks there that already have these teams, but you also have kind of, like, AI curious, right? Like, they want to do it, but they don't really know where to start, and so for there, we actually have an open source toolkit that can help you get into this optimization, and then again, that runtime, that inferencing runtime, purpose-built for CPUs. It allows you to not have to worry, again, about do I have a hardware accelerator available? How do I integrate that into my application stack? If I don't already know how to build this into my infrastructure, does my ITOps teams, do they know how to do this, and what does that runway look like? How do I cost for this? How do I plan for this? When it's just x86 compute, we've been doing that for a while, right? So it obviously still requires more, but at least it's a little bit more predictable. >> It's funny you mentioned AI native. You know, born in the cloud was a phrase that was out there. Now, you have startups that are born in AI companies. So I think you have this kind of cloud kind of vibe going on. You have lift and shift was a big discussion. Then you had cloud native, kind of in the cloud, kind of making it all work. Is there a existing set of things? People will throw on this hat, and then what's the difference between AI native and kind of providing it to existing stuff? 'Cause we're a lot of people take some of these tools and apply it to either existing stuff almost, and it's not really a lift and shift, but it's kind of like bolting on AI to something else, and then starting with AI first or native AI. >> Absolutely. It's a- >> How would you- >> It's a great question. I think that probably, where I'd probably pull back to kind of allow kind of retail-type scenarios where, you know, for five, seven, nine years or more even, a lot of these folks already have data science teams, you know? I mean, they've been doing this for quite some time. The difference is the introduction of these neural networks and deep learning, right? Those kinds of models are just a little bit of a paradigm shift. So, you know, I obviously was trying to be fun with the term AI native, but I think it's more folks that kind of came up in that neural network world, so it's a little bit more second nature, whereas I think for maybe some traditional data scientists starting to get into neural networks, you have the complexity there and the training overhead, and a lot of the aspects of getting a model finely tuned and hyperparameterization and all of these aspects of it. It just adds a layer of complexity that they're just not as used to dealing with. And so our goal is to help make that easy, and then of course, make it easier to run anywhere that you have just kind of standard infrastructure. >> Well, the other point I'd bring out, and I'd love to get your reaction to, is not only is that a neural network team, people who have been focused on that, but also, if you look at some of the DataOps lately, AIOps markets, a lot of data engineering, a lot of scale, folks who have been kind of, like, in that data tsunami cloud world are seeing, they kind of been in this, right? They're, like, been experiencing that. >> No doubt. I think it's funny the data lake concept, right? And you got data oceans now. Like, the metaphors just keep growing on us, but where it is valuable in terms of trying to shift the mindset, I've always kind of been a fan of some of the naming shift. I know with AWS, they always talk about purpose-built databases. And I always liked that because, you know, you don't have one database that can do everything. Even ones that say they can, like, you still have to do implementation detail differences. So sitting back and saying, "What is my use case, and then which database will I use it for?" I think it's kind of similar here. And when you're building those data teams, if you don't have folks that are doing data engineering, kind of that data harvesting, free processing, you got to do all that before a model's even going to care about it. So yeah, it's definitely a central piece of this as well, and again, whether or not you're going to be AI negative as you're making your way to kind of, you know, on that journey, you know, data's definitely a huge component of it. >> Yeah, you would have loved our Supercloud event we had. Talk about naming and, you know, around data meshes was talked about a lot. You're starting to see the control plane layers of data. I think that was the beginning of what I saw as that data infrastructure shift, to be horizontally scalable. So I have to ask you, with Neural Magic, when your customers and the people that are prospects for you guys, they're probably asking a lot of questions because I think the general thing that we see is, "How do I get started? Which GPU do I use?" I mean, there's a lot of things that are kind of, I won't say technical or targeted towards people who are living in that world, but, like, as the mainstream enterprises come in, they're going to need a playbook. What do you guys see, what do you guys offer your clients when they come in, and what do you recommend? >> Absolutely, and I think where we hook in specifically tends to be on the training side. So again, I've built a model. Now, I want to really optimize that model. And then on the runtime side when you want to deploy it, you know, we run that optimized model. And so that's where we're able to provide. We even have a labs offering in terms of being able to pair up our engineering teams with a customer's engineering teams, and we can actually help with most of that pipeline. So even if it is something where you have a dataset and you want some help in picking a model, you want some help training it, you want some help deploying that, we can actually help there as well. You know, there's also a great partner ecosystem out there, like a lot of folks even in the "Startup Showcase" here, that extend beyond into kind of your earlier comment around data engineering or downstream ITOps or the all-up MLOps umbrella. So we can absolutely engage with our labs, and then, of course, you know, again, partners, which are always kind of key to this. So you are spot on. I think what's happened with the kind of this, they talk about a hockey stick. This is almost like a flat wall now with the rate of innovation right now in this space. And so we do have a lot of folks wanting to go straight from curious to native. And so that's definitely where the partner ecosystem comes in so hard 'cause there just isn't anybody or any teams out there that, I literally do from, "Here's my blank database, and I want an API that does all the stuff," right? Like, that's a big chunk, but we can definitely help with the model to delivery piece. >> Well, you guys are obviously a featured company in this space. Talk about the expertise. A lot of companies are like, I won't say faking it till they make it. You can't really fake security. You can't really fake AI, right? So there's going to be a learning curve. They'll be a few startups who'll come out of the gate early. You guys are one of 'em. Talk about what you guys have as expertise as a company, why you're successful, and what problems do you solve for customers? >> No, appreciate that. Yeah, we actually, we love to tell the story of our founder, Nir Shavit. So he's a 20-year professor at MIT. Actually, he was doing a lot of work on kind of multicore processing before there were even physical multicores, and actually even did a stint in computational neurobiology in the 2010s, and the impetus for this whole technology, has a great talk on YouTube about it, where he talks about the fact that his work there, he kind of realized that the way neural networks encode and how they're executed by kind of ramming data layer by layer through these kind of HPC-style platforms, actually was not analogous to how the human brain actually works. So we're on one side, we're building neural networks, and we're trying to emulate neurons. We're not really executing them that way. So our team, which one of the co-founders, also an ex-MIT, that was kind of the birth of why can't we leverage this super-performance CPU platform, which has those really fat, fast caches attached to each core, and actually start to find a way to break that model down in a way that I can execute things in parallel, not having to do them sequentially? So it is a lot of amazing, like, talks and stuff that show kind of the magic, if you will, a part of the pun of Neural Magic, but that's kind of the foundational layer of all the engineering that we do here. And in terms of how we're able to bring it to reality for customers, I'll give one customer quote where it's a large retailer, and it's a people-counting application. So a very common application. And that customer's actually been able to show literally double the amount of cameras being run with the same amount of compute. So for a one-to-one perspective, two-to-one, business leaders usually like that math, right? So we're able to show pure cost savings, but even performance-wise, you know, we have some of the common models like your ResNets and your YOLOs, where we can actually even perform better than hardware-accelerated solutions. So we're trying to do, I need to just dumb it down to better, faster, cheaper, but from a commodity perspective, that's where we're accelerating. >> That's not a bad business model. Make things easier to use, faster, and reduce the steps it takes to do stuff. So, you know, that's always going to be a good market. Now, you guys have DeepSparse, which we've talked about on our CUBE conversation prior to this interview, delivers ML models through the software so the hardware allows for a decoupling, right? >> Yep. >> Which is going to drive probably a cost advantage. Also, it's also probably from a deployment standpoint it must be easier. Can you share the benefits? Is it a cost side? Is it more of a deployment? What are the benefits of the DeepSparse when you guys decouple the software from the hardware on the ML models? >> No you actually, you hit 'em both 'cause that really is primarily the value. Because ultimately, again, we're so early. And I came from this world in a prior life where I'm doing Java development, WebSphere, WebLogic, Tomcat open source, right? When we were trying to do innovation, we had innovation buckets, 'cause everybody wanted to be on the web and have their app and a browser, right? We got all the money we needed to build something and show, hey, look at the thing on the web, right? But when you had to get in production, that was the challenge. So to what you're speaking to here, in this situation, we're able to show we're just a Python package. So whether you just install it on the operating system itself, or we also have a containerized version you can drop on any container orchestration platform, so ECS or EKS on AWS. And so you get all the auto-scaling features. So when you think about that kind of a world where you have everything from real-time inferencing to kind of after hours batch processing inferencing, the fact that you can auto scale that hardware up and down and it's CPU based, so you're paying by the minute instead of maybe paying by the hour at a lower cost shelf, it does everything from pure cost to, again, I can have my standard IT team say, "Hey, here's the Kubernetes in the container," and it just runs on the infrastructure we're already managing. So yeah, operational, cost and again, and many times even performance. (audio warbles) CPUs if I want to. >> Yeah, so that's easier on the deployment too. And you don't have this kind of, you know, blank check kind of situation where you don't know what's on the backend on the cost side. >> Exactly. >> And you control the actual hardware and you can manage that supply chain. >> And keep in mind, exactly. Because the other thing that sometimes gets lost in the conversation, depending on where a customer is, some of these workloads, like, you know, you and I remember a world where even like the roundtrip to the cloud and back was a problem for folks, right? We're used to extremely low latency. And some of these workloads absolutely also adhere to that. But there's some workloads where the latency isn't as important. And we actually even provide the tuning. Now, if we're giving you five milliseconds of latency and you don't need that, you can tune that back. So less CPU, lower cost. Now, throughput and other things come into play. But that's the kind of configurability and flexibility we give for operations. >> All right, so why should I call you if I'm a customer or prospect Neural Magic, what problem do I have or when do I know I need you guys? When do I call you in and what does my environment look like? When do I know? What are some of the signals that would tell me that I need Neural Magic? >> No, absolutely. So I think in general, any neural network, you know, the process I mentioned before called sparcification, it's, you know, an optimization process that we specialize in. Any neural network, you know, can be sparcified. So I think if it's a deep-learning neural network type model. If you're trying to get AI into production, you have cost concerns even performance-wise. I certainly hate to be too generic and say, "Hey, we'll talk to everybody." But really in this world right now, if it's a neural network, it's something where you're trying to get into production, you know, we are definitely offering, you know, kind of an at-scale performant deployable solution for deep learning models. >> So neural network you would define as what? Just devices that are connected that need to know about each other? What's the state-of-the-art current definition of neural network for customers that may think they have a neural network or might not know they have a neural network architecture? What is that definition for neural network? >> That's a great question. So basically, machine learning models that fall under this kind of category, you hear about transformers a lot, or I mentioned about YOLO, the YOLO family of computer vision models, or natural language processing models like BERT. If you have a data science team or even developers, some even regular, I used to call myself a nine to five developer 'cause I worked in the enterprise, right? So like, hey, we found a new open source framework, you know, I used to use Spring back in the day and I had to go figure it out. There's developers that are pulling these models down and they're figuring out how to get 'em into production, okay? So I think all of those kinds of situations, you know, if it's a machine learning model of the deep learning variety that's, you know, really specifically where we shine. >> Okay, so let me pretend I'm a customer for a minute. I have all these videos, like all these transcripts, I have all these people that we've interviewed, CUBE alumnis, and I say to my team, "Let's AI-ify, sparcify theCUBE." >> Yep. >> What do I do? I mean, do I just like, my developers got to get involved and they're going to be like, "Well, how do I upload it to the cloud? Do I use a GPU?" So there's a thought process. And I think a lot of companies are going through that example of let's get on this AI, how can it help our business? >> Absolutely. >> What does that progression look like? Take me through that example. I mean, I made up theCUBE example up, but we do have a lot of data. We have large data models and we have people and connect to the internet and so we kind of seem like there's a neural network. I think every company might have a neural network in place. >> Well, and I was going to say, I think in general, you all probably do represent even the standard enterprise more than most. 'Cause even the enterprise is going to have a ton of video content, a ton of text content. So I think it's a great example. So I think that that kind of sea or I'll even go ahead and use that term data lake again, of data that you have, you're probably going to want to be setting up kind of machine learning pipelines that are going to be doing all of the pre-processing from kind of the raw data to kind of prepare it into the format that say a YOLO would actually use or let's say BERT for natural language processing. So you have all these transcripts, right? So we would do a pre-processing path where we would create that into the file format that BERT, the machine learning model would know how to train off of. So that's kind of all the pre-processing steps. And then for training itself, we actually enable what's called sparse transfer learning. So that's transfer learning is a very popular method of doing training with existing models. So we would be able to retrain that BERT model with your transcript data that we have now done the pre-processing with to get it into the proper format. And now we have a BERT natural language processing model that's been trained on your data. And now we can deploy that onto DeepSparse runtime so that now you can ask that model whatever questions, or I should say pass, you're not going to ask it those kinds of questions ChatGPT, although we can do that too. But you're going to pass text through the BERT model and it's going to give you answers back. It could be things like sentiment analysis or text classification. You just call the model, and now when you pass text through it, you get the answers better, faster or cheaper. I'll use that reference again. >> Okay, we can create a CUBE bot to give us questions on the fly from the the AI bot, you know, from our previous guests. >> Well, and I will tell you using that as an example. So I had mentioned OPT before, kind of the open source version of ChatGPT. So, you know, typically that requires multiple GPUs to run. So our research team, I may have mentioned earlier, we've been able to sparcify that over 50% already and run it on only a single GPU. And so in that situation, you could train OPT with that corpus of data and do exactly what you say. Actually we could use Alexa, we could use Alexa to actually respond back with voice. How about that? We'll do an API call and we'll actually have an interactive Alexa-enabled bot. >> Okay, we're going to be a customer, let's put it on the list. But this is a great example of what you guys call software delivered AI, a topic we chatted about on theCUBE conversation. This really means this is a developer opportunity. This really is the convergence of the data growth, the restructuring, how data is going to be horizontally scalable, meets developers. So this is an AI developer model going on right now, which is kind of unique. >> It is, John, I will tell you what's interesting. And again, folks don't always think of it this way, you know, the AI magical goodness is now getting pushed in the middle where the developers and IT are operating. And so it again, that paradigm, although for some folks seem obvious, again, if you've been around for 20 years, that whole all that plumbing is a thing, right? And so what we basically help with is when you deploy the DeepSparse runtime, we have a very rich API footprint. And so the developers can call the API, ITOps can run it, or to your point, it's developer friendly enough that you could actually deploy our off-the-shelf models. We have something called the SparseZoo where we actually publish pre-optimized or pre-sparcified models. And so developers could literally grab those right off the shelf with the training they've already had and just put 'em right into their applications and deploy them as containers. So yeah, we enable that for sure as well. >> It's interesting, DevOps was infrastructure as code and we had a last season, a series on data as code, which we kind of coined. This is data as code. This is a whole nother level of opportunity where developers just want to have programmable data and apps with AI. This is a whole new- >> Absolutely. >> Well, absolutely great, great stuff. Our news team at SiliconANGLE and theCUBE said you guys had a little bit of a launch announcement you wanted to make here on the "AWS Startup Showcase." So Jay, you have something that you want to launch here? >> Yes, and thank you John for teeing me up. So I'm going to try to put this in like, you know, the vein of like an AWS, like main stage keynote launch, okay? So we're going to try this out. So, you know, a lot of our product has obviously been built on top of x86. I've been sharing that the past 15 minutes or so. And with that, you know, we're seeing a lot of acceleration for folks wanting to run on commodity infrastructure. But we've had customers and prospects and partners tell us that, you know, ARM and all of its kind of variance are very compelling, both cost performance-wise and also obviously with Edge. And wanted to know if there was anything we could do from a runtime perspective with ARM. And so we got the work and, you know, it's a hard problem to solve 'cause the instructions set for ARM is very different than the instruction set for x86, and our deep tensor column technology has to be able to work with that lower level instruction spec. But working really hard, the engineering team's been at it and we are happy to announce here at the "AWS Startup Showcase," that DeepSparse inference now has, or inference runtime now has support for AWS Graviton instances. So it's no longer just x86, it is also ARM and that obviously also opens up the door to Edge and further out the stack so that optimize once run anywhere, we're not going to open up. So it is an early access. So if you go to neuralmagic.com/graviton, you can sign up for early access, but we're excited to now get into the ARM side of the fence as well on top of Graviton. >> That's awesome. Our news team is going to jump on that news. We'll get it right up. We get a little scoop here on the "Startup Showcase." Jay Marshall, great job. That really highlights the flexibility that you guys have when you decouple the software from the hardware. And again, we're seeing open source driving a lot more in AI ops now with with machine learning and AI. So to me, that makes a lot of sense. And congratulations on that announcement. Final minute or so we have left, give a summary of what you guys are all about. Put a plug in for the company, what you guys are looking to do. I'm sure you're probably hiring like crazy. Take the last few minutes to give a plug for the company and give a summary. >> No, I appreciate that so much. So yeah, joining us out neuralmagic.com, you know, part of what we didn't spend a lot of time here, our optimization tools, we are doing all of that in the open source. It's called SparseML and I mentioned SparseZoo briefly. So we really want the data scientists community and ML engineering community to join us out there. And again, the DeepSparse runtime, it's actually free to use for trial purposes and for personal use. So you can actually run all this on your own laptop or on an AWS instance of your choice. We are now live in the AWS marketplace. So push button, deploy, come try us out and reach out to us on neuralmagic.com. And again, sign up for the Graviton early access. >> All right, Jay Marshall, Vice President of Business Development Neural Magic here, talking about performant, cost effective machine learning at scale. This is season three, episode one, focusing on foundational models as far as building data infrastructure and AI, AI native. I'm John Furrier with theCUBE. Thanks for watching. (bright upbeat music)

(soft music) >> Hello, everyone. Welcome to theCUBE's presentation of the AWS Startup Showcase. AI and Machine Learning: Top Startups Building Foundational Model Infrastructure. This is season 3, episode 1 of the ongoing series covering the exciting stuff from the AWS ecosystem, talking about machine learning and AI. I'm your host, John Furrier and today we are excited to be joined by Luis Ceze who's the CEO of OctoML and Anna Connolly, VP of customer success and experience OctoML. Great to have you on again, Luis. Anna, thanks for coming on. Appreciate it. >> Thank you, John. It's great to be here. >> Thanks for having us. >> I love the company. We had a CUBE conversation about this. You guys are really addressing how to run foundational models faster for less. And this is like the key theme. But before we get into it, this is a hot trend, but let's explain what you guys do. Can you set the narrative of what the company's about, why it was founded, what's your North Star and your mission? >> Yeah, so John, our mission is to make AI sustainable and accessible for everyone. And what we offer customers is, you know, a way of taking their models into production in the most efficient way possible by automating the process of getting a model and optimizing it for a variety of hardware and making cost-effective. So better, faster, cheaper model deployment. >> You know, the big trend here is AI. Everyone's seeing the ChatGPT, kind of the shot heard around the world. The BingAI and this fiasco and the ongoing experimentation. People are into it, and I think the business impact is clear. I haven't seen this in all of my career in the technology industry of this kind of inflection point. And every senior leader I talk to is rethinking about how to rebuild their business with AI because now the large language models have come in, these foundational models are here, they can see value in their data. This is a 10 year journey in the big data world. Now it's impacting that, and everyone's rebuilding their company around this idea of being AI first 'cause they see ways to eliminate things and make things more efficient. And so now they telling 'em to go do it. And they're like, what do we do? So what do you guys think? Can you explain what is this wave of AI and why is it happening, why now, and what should people pay attention to? What does it mean to them? >> Yeah, I mean, it's pretty clear by now that AI can do amazing things that captures people's imaginations. And also now can show things that are really impactful in businesses, right? So what people have the opportunity to do today is to either train their own model that adds value to their business or find open models out there that can do very valuable things to them. So the next step really is how do you take that model and put it into production in a cost-effective way so that the business can actually get value out of it, right? >> Anna, what's your take? Because customers are there, you're there to make 'em successful, you got the new secret weapon for their business. >> Yeah, I think we just see a lot of companies struggle to get from a trained model into a model that is deployed in a cost-effective way that actually makes sense for the application they're building. I think that's a huge challenge we see today, kind of across the board across all of our customers. >> Well, I see this, everyone asking the same question. I have data, I want to get value out of it. I got to get these big models, I got to train it. What's it going to cost? So I think there's a reality of, okay, I got to do it. Then no one has any visibility on what it costs. When they get into it, this is going to break the bank. So I have to ask you guys, the cost of training these models is on everyone's mind. OctoML, your company's focus on the cost side of it as well as the efficiency side of running these models in production. Why are the production costs such a concern and where specifically are people looking at it and why did it get here? >> Yeah, so training costs get a lot of attention because normally a large number, but we shouldn't forget that it's a large, typically one time upfront cost that customers pay. But, you know, when the model is put into production, the cost grows directly with model usage and you actually want your model to be used because it's adding value, right? So, you know, the question that a customer faces is, you know, they have a model, they have a trained model and now what? So how much would it cost to run in production, right? And now without the big wave in generative AI, which rightfully is getting a lot of attention because of the amazing things that it can do. It's important for us to keep in mind that generative AI models like ChatGPT are huge, expensive energy hogs. They cost a lot to run, right? And given that model usage growth directly, model cost grows directly with usage, what you want to do is make sure that once you put a model into production, you have the best cost structure possible so that you're not surprised when it's gets popular, right? So let me give you an example. So if you have a model that costs, say 1 to $2 million to train, but then it costs about one to two cents per session to use it, right? So if you have a million active users, even if they use just once a day, it's 10 to $20,000 a day to operate that model in production. And that very, very quickly, you know, get beyond what you paid to train it. >> Anna, these aren't small numbers, and it's cost to train and cost to operate, it kind of reminds me of when the cloud came around and the data center versus cloud options. Like, wait a minute, one, it costs a ton of cash to deploy, and then running it. This is kind of a similar dynamic. What are you seeing? >> Yeah, absolutely. I think we are going to see increasingly the cost and production outpacing the costs and training by a lot. I mean, people talk about training costs now because that's what they're confronting now because people are so focused on getting models performant enough to even use in an application. And now that we have them and they're that capable, we're really going to start to see production costs go up a lot. >> Yeah, Luis, if you don't mind, I know this might be a little bit of a tangent, but, you know, training's super important. I get that. That's what people are doing now, but then there's the deployment side of production. Where do people get caught up and miss the boat or misconfigure? What's the gotcha? Where's the trip wire or so to speak? Where do people mess up on the cost side? What do they do? Is it they don't think about it, they tie it to proprietary hardware? What's the issue? >> Yeah, several things, right? So without getting really technical, which, you know, I might get into, you know, you have to understand relationship between performance, you know, both in terms of latency and throughput and cost, right? So reducing latency is important because you improve responsiveness of the model. But it's really important to keep in mind that it often leads diminishing returns. Below a certain latency, making it faster won't make a measurable difference in experience, but it's going to cost a lot more. So understanding that is important. Now, if you care more about throughputs, which is the time it takes for you to, you know, units per period of time, you care about time to solution, we should think about this throughput per dollar. And understand what you want is the highest throughput per dollar, which may come at the cost of higher latency, which you're not going to care about, right? So, and the reality here, John, is that, you know, humans and especially folks in this space want to have the latest and greatest hardware. And often they commit a lot of money to get access to them and have to commit upfront before they understand the needs that their models have, right? So common mistake here, one is not spending time to understand what you really need, and then two, over-committing and using more hardware than you actually need. And not giving yourself enough freedom to get your workload to move around to the more cost-effective choice, right? So this is just a metaphoric choice. And then another thing that's important here too is making a model run faster on the hardware directly translates to lower cost, right? So, but it takes a lot of engineers, you need to think of ways of producing very efficient versions of your model for the target hardware that you're going to use. >> Anna, what's the customer angle here? Because price performance has been around for a long time, people get that, but now latency and throughput, that's key because we're starting to see this in apps. I mean, there's an end user piece. I even seeing it on the infrastructure side where they're taking a heavy lifting away from operational costs. So you got, you know, application specific to the user and/or top of the stack, and then you got actually being used in operations where they want both. >> Yeah, absolutely. Maybe I can illustrate this with a quick story with the customer that we had recently been working with. So this customer is planning to run kind of a transformer based model for tech generation at super high scale on Nvidia T4 GPU, so kind of a commodity GPU. And the scale was so high that they would've been paying hundreds of thousands of dollars in cloud costs per year just to serve this model alone. You know, one of many models in their application stack. So we worked with this team to optimize our model and then benchmark across several possible targets. So that matching the hardware that Luis was just talking about, including the newer kind of Nvidia A10 GPUs. And what they found during this process was pretty interesting. First, the team was able to shave a quarter of their spend just by using better optimization techniques on the T4, the older hardware. But actually moving to a newer GPU would allow them to serve this model in a sub two milliseconds latency, so super fast, which was able to unlock an entirely new kind of user experience. So they were able to kind of change the value they're delivering in their application just because they were able to move to this new hardware easily. So they ultimately decided to plan their deployment on the more expensive A10 because of this, but because of the hardware specific optimizations that we helped them with, they managed to even, you know, bring costs down from what they had originally planned. And so if you extend this kind of example to everything that's happening with generative AI, I think the story we just talked about was super relevant, but the scale can be even higher, you know, it can be tenfold that. We were recently conducting kind of this internal study using GPT-J as a proxy to illustrate the experience of just a company trying to use one of these large language models with an example scenario of creating a chatbot to help job seekers prepare for interviews. So if you imagine kind of a conservative usage scenario where the model generates just 3000 words per user per day, which is, you know, pretty conservative for how people are interacting with these models. It costs 5 cents a session and if you're a company and your app goes viral, so from, you know, beginning of the year there's nobody, at the end of the year there's a million daily active active users in that year alone, going from zero to a million. You'll be spending about $6 million a year, which is pretty unmanageable. That's crazy, right? >> Yeah. >> For a company or a product that's just launching. So I think, you know, for us we see the real way to make these kind of advancements accessible and sustainable, as we said is to bring down cost to serve using these techniques. >> That's a great story and I think that illustrates this idea that deployment cost can vary from situation to situation, from model to model and that the efficiency is so strong with this new wave, it eliminates heavy lifting, creates more efficiency, automates intellect. I mean, this is the trend, this is radical, this is going to increase. So the cost could go from nominal to millions, literally, potentially. So, this is what customers are doing. Yeah, that's a great story. What makes sense on a financial, is there a cost of ownership? Is there a pattern for best practice for training? What do you guys advise cuz this is a lot of time and money involved in all potential, you know, good scenarios of upside. But you can get over your skis as they say, and be successful and be out of business if you don't manage it. I mean, that's what people are talking about, right? >> Yeah, absolutely. I think, you know, we see kind of three main vectors to reduce cost. I think one is make your deployment process easier overall, so that your engineering effort to even get your app running goes down. Two, would be get more from the compute you're already paying for, you're already paying, you know, for your instances in the cloud, but can you do more with that? And then three would be shop around for lower cost hardware to match your use case. So on the first one, I think making the deployment easier overall, there's a lot of manual work that goes into benchmarking, optimizing and packaging models for deployment. And because the performance of machine learning models can be really hardware dependent, you have to go through this process for each target you want to consider running your model on. And this is hard, you know, we see that every day. But for teams who want to incorporate some of these large language models into their applications, it might be desirable because licensing a model from a large vendor like OpenAI can leave you, you know, over provision, kind of paying for capabilities you don't need in your application or can lock you into them and you lose flexibility. So we have a customer whose team actually prepares models for deployment in a SaaS application that many of us use every day. And they told us recently that without kind of an automated benchmarking and experimentation platform, they were spending several days each to benchmark a single model on a single hardware type. So this is really, you know, manually intensive and then getting more from the compute you're already paying for. We do see customers who leave money on the table by running models that haven't been optimized specifically for the hardware target they're using, like Luis was mentioning. And for some teams they just don't have the time to go through an optimization process and for others they might lack kind of specialized expertise and this is something we can bring. And then on shopping around for different hardware types, we really see a huge variation in model performance across hardware, not just CPU vs. GPU, which is, you know, what people normally think of. But across CPU vendors themselves, high memory instances and across cloud providers even. So the best strategy here is for teams to really be able to, we say, look before you leap by running real world benchmarking and not just simulations or predictions to find the best software, hardware combination for their workload. >> Yeah. You guys sound like you have a very impressive customer base deploying large language models. Where would you categorize your current customer base? And as you look out, as you guys are growing, you have new customers coming in, take me through the progression. Take me through the profile of some of your customers you have now, size, are they hyperscalers, are they big app folks, are they kicking the tires? And then as people are out there scratching heads, I got to get in this game, what's their psychology like? Are they coming in with specific problems or do they have specific orientation point of view about what they want to do? Can you share some data around what you're seeing? >> Yeah, I think, you know, we have customers that kind of range across the spectrum of sophistication from teams that basically don't have MLOps expertise in their company at all. And so they're really looking for us to kind of give a full service, how should I do everything from, you know, optimization, find the hardware, prepare for deployment. And then we have teams that, you know, maybe already have their serving and hosting infrastructure up and ready and they already have models in production and they're really just looking to, you know, take the extra juice out of the hardware and just do really specific on that optimization piece. I think one place where we're doing a lot more work now is kind of in the developer tooling, you know, model selection space. And that's kind of an area that we're creating more tools for, particularly within the PyTorch ecosystem to bring kind of this power earlier in the development cycle so that as people are grabbing a model off the shelf, they can, you know, see how it might perform and use that to inform their development process. >> Luis, what's the big, I like this idea of picking the models because isn't that like going to the market and picking the best model for your data? It's like, you know, it's like, isn't there a certain approaches? What's your view on this? 'Cause this is where everyone, I think it's going to be a land rush for this and I want to get your thoughts. >> For sure, yeah. So, you know, I guess I'll start with saying the one main takeaway that we got from the GPT-J study is that, you know, having a different understanding of what your model's compute and memory requirements are, very quickly, early on helps with the much smarter AI model deployments, right? So, and in fact, you know, Anna just touched on this, but I want to, you know, make sure that it's clear that OctoML is putting that power into user's hands right now. So in partnership with AWS, we are launching this new PyTorch native profiler that allows you with a single, you know, one line, you know, code decorator allows you to see how your code runs on a variety of different hardware after accelerations. So it gives you very clear, you know, data on how you should think about your model deployments. And this ties back to choices of models. So like, if you have a set of choices that are equally good of models in terms of functionality and you want to understand after acceleration how are you going to deploy, how much they're going to cost or what are the options using a automated process of making a decision is really, really useful. And in fact, so I think these events can get early access to this by signing up for the Octopods, you know, this is exclusive group for insiders here, so you can go to OctoML.ai/pods to sign up. >> So that Octopod, is that a program? What is that, is that access to code? Is that a beta, what is that? Explain, take a minute and explain Octopod. >> I think the Octopod would be a group of people who is interested in experiencing this functionality. So it is the friends and users of OctoML that would be the Octopod. And then yes, after you sign up, we would provide you essentially the tool in code form for you to try out in your own. I mean, part of the benefit of this is that it happens in your own local environment and you're in control of everything kind of within the workflow that developers are already using to create and begin putting these models into their applications. So it would all be within your control. >> Got it. I think the big question I have for you is when do you, when does that one of your customers know they need to call you? What's their environment look like? What are they struggling with? What are the conversations they might be having on their side of the fence? If anyone's watching this, they're like, "Hey, you know what, I've got my team, we have a lot of data. Do we have our own language model or do I use someone else's?" There's a lot of this, I will say discovery going on around what to do, what path to take, what does that customer look like, if someone's listening, when do they know to call you guys, OctoML? >> Well, I mean the most obvious one is that you have a significant spend on AI/ML, come and talk to us, you know, putting AIML into production. So that's the clear one. In fact, just this morning I was talking to someone who is in life sciences space and is having, you know, 15 to $20 million a year cloud related to AI/ML deployment is a clear, it's a pretty clear match right there, right? So that's on the cost side. But I also want to emphasize something that Anna said earlier that, you know, the hardware and software complexity involved in putting model into production is really high. So we've been able to abstract that away, offering a clean automation flow enables one, to experiment early on, you know, how models would run and get them to production. And then two, once they are into production, gives you an automated flow to continuously updating your model and taking advantage of all this acceleration and ability to run the model on the right hardware. So anyways, let's say one then is cost, you know, you have significant cost and then two, you have an automation needs. And Anna please compliment that. >> Yeah, Anna you can please- >> Yeah, I think that's exactly right. Maybe the other time is when you are expecting a big scale up in serving your application, right? You're launching a new feature, you expect to get a lot of usage or, and you want to kind of anticipate maybe your CTO, your CIO, whoever pays your cloud bills is going to come after you, right? And so they want to know, you know, what's the return on putting this model essentially into my application stack? Am I going to, is the usage going to match what I'm paying for it? And then you can understand that. >> So you guys have a lot of the early adopters, they got big data teams, they're pushed in the production, they want to get a little QA, test the waters, understand, use your technology to figure it out. Is there any cases where people have gone into production, they have to pull it out? It's like the old lemon laws with your car, you buy a car and oh my god, it's not the way I wanted it. I mean, I can imagine the early people through the wall, so to speak, in the wave here are going to be bloody in the sense that they've gone in and tried stuff and get stuck with huge bills. Are you seeing that? Are people pulling stuff out of production and redeploying? Or I can imagine that if I had a bad deployment, I'd want to refactor that or actually replatform that. Do you see that too? >> Definitely after a sticker shock, yes, your customers will come and make sure that, you know, the sticker shock won't happen again. >> Yeah. >> But then there's another more thorough aspect here that I think we likely touched on, be worth elaborating a bit more is just how are you going to scale in a way that's feasible depending on the allocation that you get, right? So as we mentioned several times here, you know, model deployment is so hardware dependent and so complex that you tend to get a model for a hardware choice and then you want to scale that specific type of instance. But what if, when you want to scale because suddenly luckily got popular and, you know, you want to scale it up and then you don't have that instance anymore. So how do you live with whatever you have at that moment is something that we see customers needing as well. You know, so in fact, ideally what we want is customers to not think about what kind of specific instances they want. What they want is to know what their models need. Say, they know the SLA and then find a set of hybrid targets and instances that hit the SLA whenever they're also scaling, they're going to scale with more freedom, right? Instead of having to wait for AWS to give them more specific allocation for a specific instance. What if you could live with other types of hardware and scale up in a more free way, right? So that's another thing that we see customers, you know, like they need more freedom to be able to scale with whatever is available. >> Anna, you touched on this with the business model impact to that 6 million cost, if that goes out of control, there's a business model aspect and there's a technical operation aspect to the cost side too. You want to be mindful of riding the wave in a good way, but not getting over your skis. So that brings up the point around, you know, confidence, right? And teamwork. Because if you're in production, there's probably a team behind it. Talk about the team aspect of your customers. I mean, they're dedicated, they go put stuff into production, they're developers, there're data. What's in it for them? Are they getting better, are they in the beach, you know, reading the book. Are they, you know, are there easy street for them? What's the customer benefit to the teams? >> Yeah, absolutely. With just a few clicks of a button, you're in production, right? That's the dream. So yeah, I mean I think that, you know, we illustrated it before a little bit. I think the automated kind of benchmarking and optimization process, like when you think about the effort it takes to get that data by hand, which is what people are doing today, they just don't do it. So they're making decisions without the best information because it's, you know, there just isn't the bandwidth to get the information that they need to make the best decision and then know exactly how to deploy it. So I think it's actually bringing kind of a new insight and capability to these teams that they didn't have before. And then maybe another aspect on the team side is that it's making the hand-off of the models from the data science teams to the model deployment teams more seamless. So we have, you know, we have seen in the past that this kind of transition point is the place where there are a lot of hiccups, right? The data science team will give a model to the production team and it'll be too slow for the application or it'll be too expensive to run and it has to go back and be changed and kind of this loop. And so, you know, with the PyTorch profiler that Luis was talking about, and then also, you know, the other ways we do optimization that kind of prevents that hand-off problem from happening. >> Luis and Anna, you guys have a great company. Final couple minutes left. Talk about the company, the people there, what's the culture like, you know, if Intel has Moore's law, which is, you know, doubling the performance in few years, what's the culture like there? Is it, you know, more throughput, better pricing? Explain what's going on with the company and put a plug in. Luis, we'll start with you. >> Yeah, absolutely. I'm extremely proud of the team that we built here. You know, we have a people first culture, you know, very, very collaborative and folks, we all have a shared mission here of making AI more accessible and sustainable. We have a very diverse team in terms of backgrounds and life stories, you know, to do what we do here, we need a team that has expertise in software engineering, in machine learning, in computer architecture. Even though we don't build chips, we need to understand how they work, right? So, and then, you know, the fact that we have this, this very really, really varied set of backgrounds makes the environment, you know, it's say very exciting to learn more about, you know, assistance end-to-end. But also makes it for a very interesting, you know, work environment, right? So people have different backgrounds, different stories. Some of them went to grad school, others, you know, were in intelligence agencies and now are working here, you know. So we have a really interesting set of people and, you know, life is too short not to work with interesting humans. You know, that's something that I like to think about, you know. >> I'm sure your off-site meetings are a lot of fun, people talking about computer architectures, silicon advances, the next GPU, the big data models coming in. Anna, what's your take? What's the culture like? What's the company vibe and what are you guys looking to do? What's the customer success pattern? What's up? >> Yeah, absolutely. I mean, I, you know, second all of the great things that Luis just said about the team. I think one that I, an additional one that I'd really like to underscore is kind of this customer obsession, to use a term you all know well. And focus on the end users and really making the experiences that we're bringing to our user who are developers really, you know, useful and valuable for them. And so I think, you know, all of these tools that we're trying to put in the hands of users, the industry and the market is changing so rapidly that our products across the board, you know, all of the companies that, you know, are part of the showcase today, we're all evolving them so quickly and we can only do that kind of really hand in glove with our users. So that would be another thing I'd emphasize. >> I think the change dynamic, the power dynamics of this industry is just the beginning. I'm very bullish that this is going to be probably one of the biggest inflection points in history of the computer industry because of all the dynamics of the confluence of all the forces, which you mentioned some of them, I mean PC, you know, interoperability within internetworking and you got, you know, the web and then mobile. Now we have this, I mean, I wouldn't even put social media even in the close to this. Like, this is like, changes user experience, changes infrastructure. There's going to be massive accelerations in performance on the hardware side from AWS's of the world and cloud and you got the edge and more data. This is really what big data was going to look like. This is the beginning. Final question, what do you guys see going forward in the future? >> Well, it's undeniable that machine learning and AI models are becoming an integral part of an interesting application today, right? So, and the clear trends here are, you know, more and more competitional needs for these models because they're only getting more and more powerful. And then two, you know, seeing the complexity of the infrastructure where they run, you know, just considering the cloud, there's like a wide variety of choices there, right? So being able to live with that and making the most out of it in a way that does not require, you know, an impossible to find team is something that's pretty clear. So the need for automation, abstracting with the complexity is definitely here. And we are seeing this, you know, trends are that you also see models starting to move to the edge as well. So it's clear that we're seeing, we are going to live in a world where there's no large models living in the cloud. And then, you know, edge models that talk to these models in the cloud to form, you know, an end-to-end truly intelligent application. >> Anna? >> Yeah, I think, you know, our, Luis said it at the beginning. Our vision is to make AI sustainable and accessible. And I think as this technology just expands in every company and every team, that's going to happen kind of on its own. And we're here to help support that. And I think you can't do that without tools like those like OctoML. >> I think it's going to be an error of massive invention, creativity, a lot of the format heavy lifting is going to allow the talented people to automate their intellect. I mean, this is really kind of what we see going on. And Luis, thank you so much. Anna, thanks for coming on this segment. Thanks for coming on theCUBE and being part of the AWS Startup Showcase. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(upbeat music) >> Welcome everyone to theCUBE's presentation of the AWS Startup Showcase AI/ML Top Startups Building Foundation Model Infrastructure. This is season three, episode one of our ongoing series covering exciting startups from the AWS ecosystem to talk about data and analytics. I'm your host, Lisa Martin and today we're excited to be joined by two guests from Astronomer. Steven Hillion joins us, it's Chief Data Officer and Jeff Fletcher, it's director of ML. They're here to talk about machine learning and data orchestration. Guys, thank you so much for joining us today. >> Thank you. >> It's great to be here. >> Before we get into machine learning let's give the audience an overview of Astronomer. Talk about what that is, Steven. Talk about what you mean by data orchestration. >> Yeah, let's start with Astronomer. We're the Airflow company basically. The commercial developer behind the open-source project, Apache Airflow. I don't know if you've heard of Airflow. It's sort of de-facto standard these days for orchestrating data pipelines, data engineering pipelines, and as we'll talk about later, machine learning pipelines. It's really is the de-facto standard. I think we're up to about 12 million downloads a month. That's actually as a open-source project. I think at this point it's more popular by some measures than Slack. Airflow was created by Airbnb some years ago to manage all of their data pipelines and manage all of their workflows and now it powers the data ecosystem for organizations as diverse as Electronic Arts, Conde Nast is one of our big customers, a big user of Airflow. And also not to mention the biggest banks on Wall Street use Airflow and Astronomer to power the flow of data throughout their organizations. >> Talk about that a little bit more, Steven, in terms of the business impact. You mentioned some great customer names there. What is the business impact or outcomes that a data orchestration strategy enables businesses to achieve? >> Yeah, I mean, at the heart of it is quite simply, scheduling and managing data pipelines. And so if you have some enormous retailer who's managing the flow of information throughout their organization they may literally have thousands or even tens of thousands of data pipelines that need to execute every day to do things as simple as delivering metrics for the executives to consume at the end of the day, to producing on a weekly basis new machine learning models that can be used to drive product recommendations. One of our customers, for example, is a British food delivery service. And you get those recommendations in your application that says, "Well, maybe you want to have samosas with your curry." That sort of thing is powered by machine learning models that they train on a regular basis to reflect changing conditions in the market. And those are produced through Airflow and through the Astronomer platform, which is essentially a managed platform for running airflow. So at its simplest it really is just scheduling and managing those workflows. But that's easier said than done of course. I mean if you have 10 thousands of those things then you need to make sure that they all run that they all have sufficient compute resources. If things fail, how do you track those down across those 10,000 workflows? How easy is it for an average data scientist or data engineer to contribute their code, their Python notebooks or their SQL code into a production environment? And then you've got reproducibility, governance, auditing, like managing data flows across an organization which we think of as orchestrating them is much more than just scheduling. It becomes really complicated pretty quickly. >> I imagine there's a fair amount of complexity there. Jeff, let's bring you into the conversation. Talk a little bit about Astronomer through your lens, data orchestration and how it applies to MLOps. >> So I come from a machine learning background and for me the interesting part is that machine learning requires the expansion into orchestration. A lot of the same things that you're using to go and develop and build pipelines in a standard data orchestration space applies equally well in a machine learning orchestration space. What you're doing is you're moving data between different locations, between different tools, and then tasking different types of tools to act on that data. So extending it made logical sense from a implementation perspective. And a lot of my focus at Astronomer is really to explain how Airflow can be used well in a machine learning context. It is being used well, it is being used a lot by the customers that we have and also by users of the open source version. But it's really being able to explain to people why it's a natural extension for it and how well it fits into that. And a lot of it is also extending some of the infrastructure capabilities that Astronomer provides to those customers for them to be able to run some of the more platform specific requirements that come with doing machine learning pipelines. >> Let's get into some of the things that make Astronomer unique. Jeff, sticking with you, when you're in customer conversations, what are some of the key differentiators that you articulate to customers? >> So a lot of it is that we are not specific to one cloud provider. So we have the ability to operate across all of the big cloud providers. I know, I'm certain we have the best developers that understand how best practices implementations for data orchestration works. So we spend a lot of time talking to not just the business outcomes and the business users of the product, but also also for the technical people, how to help them better implement things that they may have come across on a Stack Overflow article or not necessarily just grown with how the product has migrated. So it's the ability to run it wherever you need to run it and also our ability to help you, the customer, better implement and understand those workflows that I think are two of the primary differentiators that we have. >> Lisa: Got it. >> I'll add another one if you don't mind. >> You can go ahead, Steven. >> Is lineage and dependencies between workflows. One thing we've done is to augment core Airflow with Lineage services. So using the Open Lineage framework, another open source framework for tracking datasets as they move from one workflow to another one, team to another, one data source to another is a really key component of what we do and we bundle that within the service so that as a developer or as a production engineer, you really don't have to worry about lineage, it just happens. Jeff, may show us some of this later that you can actually see as data flows from source through to a data warehouse out through a Python notebook to produce a predictive model or a dashboard. Can you see how those data products relate to each other? And when something goes wrong, figure out what upstream maybe caused the problem, or if you're about to change something, figure out what the impact is going to be on the rest of the organization. So Lineage is a big deal for us. >> Got it. >> And just to add on to that, the other thing to think about is that traditional Airflow is actually a complicated implementation. It required quite a lot of time spent understanding or was almost a bespoke language that you needed to be able to develop in two write these DAGs, which is like fundamental pipelines. So part of what we are focusing on is tooling that makes it more accessible to say a data analyst or a data scientist who doesn't have or really needs to gain the necessary background in how the semantics of Airflow DAGs works to still be able to get the benefit of what Airflow can do. So there is new features and capabilities built into the astronomer cloud platform that effectively obfuscates and removes the need to understand some of the deep work that goes on. But you can still do it, you still have that capability, but we are expanding it to be able to have orchestrated and repeatable processes accessible to more teams within the business. >> In terms of accessibility to more teams in the business. You talked about data scientists, data analysts, developers. Steven, I want to talk to you, as the chief data officer, are you having more and more conversations with that role and how is it emerging and evolving within your customer base? >> Hmm. That's a good question, and it is evolving because I think if you look historically at the way that Airflow has been used it's often from the ground up. You have individual data engineers or maybe single data engineering teams who adopt Airflow 'cause it's very popular. Lots of people know how to use it and they bring it into an organization and say, "Hey, let's use this to run our data pipelines." But then increasingly as you turn from pure workflow management and job scheduling to the larger topic of orchestration you realize it gets pretty complicated, you want to have coordination across teams, and you want to have standardization for the way that you manage your data pipelines. And so having a managed service for Airflow that exists in the cloud is easy to spin up as you expand usage across the organization. And thinking long term about that in the context of orchestration that's where I think the chief data officer or the head of analytics tends to get involved because they really want to think of this as a strategic investment that they're making. Not just per team individual Airflow deployments, but a network of data orchestrators. >> That network is key. Every company these days has to be a data company. We talk about companies being data driven. It's a common word, but it's true. It's whether it is a grocer or a bank or a hospital, they've got to be data companies. So talk to me a little bit about Astronomer's business model. How is this available? How do customers get their hands on it? >> Jeff, go ahead. >> Yeah, yeah. So we have a managed cloud service and we have two modes of operation. One, you can bring your own cloud infrastructure. So you can say here is an account in say, AWS or Azure and we can go and deploy the necessary infrastructure into that, or alternatively we can host everything for you. So it becomes a full SaaS offering. But we then provide a platform that connects at the backend to your internal IDP process. So however you are authenticating users to make sure that the correct people are accessing the services that they need with role-based access control. From there we are deploying through Kubernetes, the different services and capabilities into either your cloud account or into an account that we host. And from there Airflow does what Airflow does, which is its ability to then reach to different data systems and data platforms and to then run the orchestration. We make sure we do it securely, we have all the necessary compliance certifications required for GDPR in Europe and HIPAA based out of the US, and a whole bunch host of others. So it is a secure platform that can run in a place that you need it to run, but it is a managed Airflow that includes a lot of the extra capabilities like the cloud developer environment and the open lineage services to enhance the overall airflow experience. >> Enhance the overall experience. So Steven, going back to you, if I'm a Conde Nast or another organization, what are some of the key business outcomes that I can expect? As one of the things I think we've learned during the pandemic is access to realtime data is no longer a nice to have for organizations. It's really an imperative. It's that demanding consumer that wants to have that personalized, customized, instant access to a product or a service. So if I'm a Conde Nast or I'm one of your customers, what can I expect my business to be able to achieve as a result of data orchestration? >> Yeah, I think in a nutshell it's about providing a reliable, scalable, and easy to use service for developing and running data workflows. And talking of demanding customers, I mean, I'm actually a customer myself, as you mentioned, I'm the head of data for Astronomer. You won't be surprised to hear that we actually use Astronomer and Airflow to run all of our data pipelines. And so I can actually talk about my experience. When I started I was of course familiar with Airflow, but it always seemed a little bit unapproachable to me if I was introducing that to a new team of data scientists. They don't necessarily want to have to think about learning something new. But I think because of the layers that Astronomer has provided with our Astro service around Airflow it was pretty easy for me to get up and running. Of course I've got an incentive for doing that. I work for the Airflow company, but we went from about, at the beginning of last year, about 500 data tasks that we were running on a daily basis to about 15,000 every day. We run something like a million data operations every month within my team. And so as one outcome, just the ability to spin up new production workflows essentially in a single day you go from an idea in the morning to a new dashboard or a new model in the afternoon, that's really the business outcome is just removing that friction to operationalizing your machine learning and data workflows. >> And I imagine too, oh, go ahead, Jeff. >> Yeah, I think to add to that, one of the things that becomes part of the business cycle is a repeatable capabilities for things like reporting, for things like new machine learning models. And the impediment that has existed is that it's difficult to take that from a team that's an analyst team who then provide that or a data science team that then provide that to the data engineering team who have to work the workflow all the way through. What we're trying to unlock is the ability for those teams to directly get access to scheduling and orchestrating capabilities so that a business analyst can have a new report for C-suite execs that needs to be done once a week, but the time to repeatability for that report is much shorter. So it is then immediately in the hands of the person that needs to see it. It doesn't have to go into a long list of to-dos for a data engineering team that's already overworked that they eventually get it to it in a month's time. So that is also a part of it is that the realizing, orchestration I think is fairly well and a lot of people get the benefit of being able to orchestrate things within a business, but it's having more people be able to do it and shorten the time that that repeatability is there is one of the main benefits from good managed orchestration. >> So a lot of workforce productivity improvements in what you're doing to simplify things, giving more people access to data to be able to make those faster decisions, which ultimately helps the end user on the other end to get that product or the service that they're expecting like that. Jeff, I understand you have a demo that you can share so we can kind of dig into this. >> Yeah, let me take you through a quick look of how the whole thing works. So our starting point is our cloud infrastructure. This is the login. You go to the portal. You can see there's a a bunch of workspaces that are available. Workspaces are like individual places for people to operate in. I'm not going to delve into all the deep technical details here, but starting point for a lot of our data science customers is we have what we call our Cloud IDE, which is a web-based development environment for writing and building out DAGs without actually having to know how the underpinnings of Airflow work. This is an internal one, something that we use. You have a notebook-like interface that lets you write python code and SQL code and a bunch of specific bespoke type of blocks if you want. They all get pulled together and create a workflow. So this is a workflow, which gets compiled to something that looks like a complicated set of Python code, which is the DAG. I then have a CICD process pipeline where I commit this through to my GitHub repo. So this comes to a repo here, which is where these DAGs that I created in the previous step exist. I can then go and say, all right, I want to see how those particular DAGs have been running. We then get to the actual Airflow part. So this is the managed Airflow component. So we add the ability for teams to fairly easily bring up an Airflow instance and write code inside our notebook-like environment to get it into that instance. So you can see it's been running. That same process that we built here that graph ends up here inside this, but you don't need to know how the fundamentals of Airflow work in order to get this going. Then we can run one of these, it runs in the background and we can manage how it goes. And from there, every time this runs, it's emitting to a process underneath, which is the open lineage service, which is the lineage integration that allows me to come in here and have a look and see this was that actual, that same graph that we built, but now it's the historic version. So I know where things started, where things are going, and how it ran. And then I can also do a comparison. So if I want to see how this particular run worked compared to one historically, I can grab one from a previous date and it will show me the comparison between the two. So that combination of managed Airflow, getting Airflow up and running very quickly, but the Cloud IDE that lets you write code and know how to get something into a repeatable format get that into Airflow and have that attached to the lineage process adds what is a complete end-to-end orchestration process for any business looking to get the benefit from orchestration. >> Outstanding. Thank you so much Jeff for digging into that. So one of my last questions, Steven is for you. This is exciting. There's a lot that you guys are enabling organizations to achieve here to really become data-driven companies. So where can folks go to get their hands on this? >> Yeah, just go to astronomer.io and we have plenty of resources. If you're new to Airflow, you can read our documentation, our guides to getting started. We have a CLI that you can download that is really I think the easiest way to get started with Airflow. But you can actually sign up for a trial. You can sign up for a guided trial where our teams, we have a team of experts, really the world experts on getting Airflow up and running. And they'll take you through that trial and allow you to actually kick the tires and see how this works with your data. And I think you'll see pretty quickly that it's very easy to get started with Airflow, whether you're doing that from the command line or doing that in our cloud service. And all of that is available on our website >> astronomer.io. Jeff, last question for you. What are you excited about? There's so much going on here. What are some of the things, maybe you can give us a sneak peek coming down the road here that prospects and existing customers should be excited about? >> I think a lot of the development around the data awareness components, so one of the things that's traditionally been complicated with orchestration is you leave your data in the place that you're operating on and we're starting to have more data processing capability being built into Airflow. And from a Astronomer perspective, we are adding more capabilities around working with larger datasets, doing bigger data manipulation with inside the Airflow process itself. And that lends itself to better machine learning implementation. So as we start to grow and as we start to get better in the machine learning context, well, in the data awareness context, it unlocks a lot more capability to do and implement proper machine learning pipelines. >> Awesome guys. Exciting stuff. Thank you so much for talking to me about Astronomer, machine learning, data orchestration, and really the value in it for your customers. Steve and Jeff, we appreciate your time. >> Thank you. >> My pleasure, thanks. >> And we thank you for watching. This is season three, episode one of our ongoing series covering exciting startups from the AWS ecosystem. I'm your host, Lisa Martin. You're watching theCUBE, the leader in live tech coverage. (upbeat music)

(upbeat music) >> Hello everyone. Welcome to theCube's presentation of the "AWS Startup Showcase." The topic this episode is AI and machine learning, top startups building foundational model infrastructure. This is season three, episode one of the ongoing series covering exciting startups from the AWS ecosystem. And this time we're talking about AI and machine learning. I'm your host, John Furrier. I'm excited I'm joined today by Robert Nishihara, who's the co-founder and CEO of a hot startup called Anyscale. He's here to talk about Ray, the open source project, Anyscale's infrastructure for foundation as well. Robert, thank you for joining us today. >> Yeah, thanks so much as well. >> I've been following your company since the founding pre pandemic and you guys really had a great vision scaled up and in a perfect position for this big wave that we all see with ChatGPT and OpenAI that's gone mainstream. Finally, AI has broken out through the ropes and now gone mainstream, so I think you guys are really well positioned. I'm looking forward to to talking with you today. But before we get into it, introduce the core mission for Anyscale. Why do you guys exist? What is the North Star for Anyscale? >> Yeah, like you mentioned, there's a tremendous amount of excitement about AI right now. You know, I think a lot of us believe that AI can transform just every different industry. So one of the things that was clear to us when we started this company was that the amount of compute needed to do AI was just exploding. Like to actually succeed with AI, companies like OpenAI or Google or you know, these companies getting a lot of value from AI, were not just running these machine learning models on their laptops or on a single machine. They were scaling these applications across hundreds or thousands or more machines and GPUs and other resources in the Cloud. And so to actually succeed with AI, and this has been one of the biggest trends in computing, maybe the biggest trend in computing in, you know, in recent history, the amount of compute has been exploding. And so to actually succeed with that AI, to actually build these scalable applications and scale the AI applications, there's a tremendous software engineering lift to build the infrastructure to actually run these scalable applications. And that's very hard to do. So one of the reasons many AI projects and initiatives fail is that, or don't make it to production, is the need for this scale, the infrastructure lift, to actually make it happen. So our goal here with Anyscale and Ray, is to make that easy, is to make scalable computing easy. So that as a developer or as a business, if you want to do AI, if you want to get value out of AI, all you need to know is how to program on your laptop. Like, all you need to know is how to program in Python. And if you can do that, then you're good to go. Then you can do what companies like OpenAI or Google do and get value out of machine learning. >> That programming example of how easy it is with Python reminds me of the early days of Cloud, when infrastructure as code was talked about was, it was just code the infrastructure programmable. That's super important. That's what AI people wanted, first program AI. That's the new trend. And I want to understand, if you don't mind explaining, the relationship that Anyscale has to these foundational models and particular the large language models, also called LLMs, was seen with like OpenAI and ChatGPT. Before you get into the relationship that you have with them, can you explain why the hype around foundational models? Why are people going crazy over foundational models? What is it and why is it so important? >> Yeah, so foundational models and foundation models are incredibly important because they enable businesses and developers to get value out of machine learning, to use machine learning off the shelf with these large models that have been trained on tons of data and that are useful out of the box. And then, of course, you know, as a business or as a developer, you can take those foundational models and repurpose them or fine tune them or adapt them to your specific use case and what you want to achieve. But it's much easier to do that than to train them from scratch. And I think there are three, for people to actually use foundation models, there are three main types of workloads or problems that need to be solved. One is training these foundation models in the first place, like actually creating them. The second is fine tuning them and adapting them to your use case. And the third is serving them and actually deploying them. Okay, so Ray and Anyscale are used for all of these three different workloads. Companies like OpenAI or Cohere that train large language models. Or open source versions like GPTJ are done on top of Ray. There are many startups and other businesses that fine tune, that, you know, don't want to train the large underlying foundation models, but that do want to fine tune them, do want to adapt them to their purposes, and build products around them and serve them, those are also using Ray and Anyscale for that fine tuning and that serving. And so the reason that Ray and Anyscale are important here is that, you know, building and using foundation models requires a huge scale. It requires a lot of data. It requires a lot of compute, GPUs, TPUs, other resources. And to actually take advantage of that and actually build these scalable applications, there's a lot of infrastructure that needs to happen under the hood. And so you can either use Ray and Anyscale to take care of that and manage the infrastructure and solve those infrastructure problems. Or you can build the infrastructure and manage the infrastructure yourself, which you can do, but it's going to slow your team down. It's going to, you know, many of the businesses we work with simply don't want to be in the business of managing infrastructure and building infrastructure. They want to focus on product development and move faster. >> I know you got a keynote presentation we're going to go to in a second, but I think you hit on something I think is the real tipping point, doing it yourself, hard to do. These are things where opportunities are and the Cloud did that with data centers. Turned a data center and made it an API. The heavy lifting went away and went to the Cloud so people could be more creative and build their product. In this case, build their creativity. Is that kind of what's the big deal? Is that kind of a big deal happening that you guys are taking the learnings and making that available so people don't have to do that? >> That's exactly right. So today, if you want to succeed with AI, if you want to use AI in your business, infrastructure work is on the critical path for doing that. To do AI, you have to build infrastructure. You have to figure out how to scale your applications. That's going to change. We're going to get to the point, and you know, with Ray and Anyscale, we're going to remove the infrastructure from the critical path so that as a developer or as a business, all you need to focus on is your application logic, what you want the the program to do, what you want your application to do, how you want the AI to actually interface with the rest of your product. Now the way that will happen is that Ray and Anyscale will still, the infrastructure work will still happen. It'll just be under the hood and taken care of by Ray in Anyscale. And so I think something like this is really necessary for AI to reach its potential, for AI to have the impact and the reach that we think it will, you have to make it easier to do. >> And just for clarification to point out, if you don't mind explaining the relationship of Ray and Anyscale real quick just before we get into the presentation. >> So Ray is an open source project. We created it. We were at Berkeley doing machine learning. We started Ray so that, in order to provide an easy, a simple open source tool for building and running scalable applications. And Anyscale is the managed version of Ray, basically we will run Ray for you in the Cloud, provide a lot of tools around the developer experience and managing the infrastructure and providing more performance and superior infrastructure. >> Awesome. I know you got a presentation on Ray and Anyscale and you guys are positioning as the infrastructure for foundational models. So I'll let you take it away and then when you're done presenting, we'll come back, I'll probably grill you with a few questions and then we'll close it out so take it away. >> Robert: Sounds great. So I'll say a little bit about how companies are using Ray and Anyscale for foundation models. The first thing I want to mention is just why we're doing this in the first place. And the underlying observation, the underlying trend here, and this is a plot from OpenAI, is that the amount of compute needed to do machine learning has been exploding. It's been growing at something like 35 times every 18 months. This is absolutely enormous. And other people have written papers measuring this trend and you get different numbers. But the point is, no matter how you slice and dice it, it' a astronomical rate. Now if you compare that to something we're all familiar with, like Moore's Law, which says that, you know, the processor performance doubles every roughly 18 months, you can see that there's just a tremendous gap between the needs, the compute needs of machine learning applications, and what you can do with a single chip, right. So even if Moore's Law were continuing strong and you know, doing what it used to be doing, even if that were the case, there would still be a tremendous gap between what you can do with the chip and what you need in order to do machine learning. And so given this graph, what we've seen, and what has been clear to us since we started this company, is that doing AI requires scaling. There's no way around it. It's not a nice to have, it's really a requirement. And so that led us to start Ray, which is the open source project that we started to make it easy to build these scalable Python applications and scalable machine learning applications. And since we started the project, it's been adopted by a tremendous number of companies. Companies like OpenAI, which use Ray to train their large models like ChatGPT, companies like Uber, which run all of their deep learning and classical machine learning on top of Ray, companies like Shopify or Spotify or Instacart or Lyft or Netflix, ByteDance, which use Ray for their machine learning infrastructure. Companies like Ant Group, which makes Alipay, you know, they use Ray across the board for fraud detection, for online learning, for detecting money laundering, you know, for graph processing, stream processing. Companies like Amazon, you know, run Ray at a tremendous scale and just petabytes of data every single day. And so the project has seen just enormous adoption since, over the past few years. And one of the most exciting use cases is really providing the infrastructure for building training, fine tuning, and serving foundation models. So I'll say a little bit about, you know, here are some examples of companies using Ray for foundation models. Cohere trains large language models. OpenAI also trains large language models. You can think about the workloads required there are things like supervised pre-training, also reinforcement learning from human feedback. So this is not only the regular supervised learning, but actually more complex reinforcement learning workloads that take human input about what response to a particular question, you know is better than a certain other response. And incorporating that into the learning. There's open source versions as well, like GPTJ also built on top of Ray as well as projects like Alpa coming out of UC Berkeley. So these are some of the examples of exciting projects in organizations, training and creating these large language models and serving them using Ray. Okay, so what actually is Ray? Well, there are two layers to Ray. At the lowest level, there's the core Ray system. This is essentially low level primitives for building scalable Python applications. Things like taking a Python function or a Python class and executing them in the cluster setting. So Ray core is extremely flexible and you can build arbitrary scalable applications on top of Ray. So on top of Ray, on top of the core system, what really gives Ray a lot of its power is this ecosystem of scalable libraries. So on top of the core system you have libraries, scalable libraries for ingesting and pre-processing data, for training your models, for fine tuning those models, for hyper parameter tuning, for doing batch processing and batch inference, for doing model serving and deployment, right. And a lot of the Ray users, the reason they like Ray is that they want to run multiple workloads. They want to train and serve their models, right. They want to load their data and feed that into training. And Ray provides common infrastructure for all of these different workloads. So this is a little overview of what Ray, the different components of Ray. So why do people choose to go with Ray? I think there are three main reasons. The first is the unified nature. The fact that it is common infrastructure for scaling arbitrary workloads, from data ingest to pre-processing to training to inference and serving, right. This also includes the fact that it's future proof. AI is incredibly fast moving. And so many people, many companies that have built their own machine learning infrastructure and standardized on particular workflows for doing machine learning have found that their workflows are too rigid to enable new capabilities. If they want to do reinforcement learning, if they want to use graph neural networks, they don't have a way of doing that with their standard tooling. And so Ray, being future proof and being flexible and general gives them that ability. Another reason people choose Ray in Anyscale is the scalability. This is really our bread and butter. This is the reason, the whole point of Ray, you know, making it easy to go from your laptop to running on thousands of GPUs, making it easy to scale your development workloads and run them in production, making it easy to scale, you know, training to scale data ingest, pre-processing and so on. So scalability and performance, you know, are critical for doing machine learning and that is something that Ray provides out of the box. And lastly, Ray is an open ecosystem. You can run it anywhere. You can run it on any Cloud provider. Google, you know, Google Cloud, AWS, Asure. You can run it on your Kubernetes cluster. You can run it on your laptop. It's extremely portable. And not only that, it's framework agnostic. You can use Ray to scale arbitrary Python workloads. You can use it to scale and it integrates with libraries like TensorFlow or PyTorch or JAX or XG Boost or Hugging Face or PyTorch Lightning, right, or Scikit-learn or just your own arbitrary Python code. It's open source. And in addition to integrating with the rest of the machine learning ecosystem and these machine learning frameworks, you can use Ray along with all of the other tooling in the machine learning ecosystem. That's things like weights and biases or ML flow, right. Or you know, different data platforms like Databricks, you know, Delta Lake or Snowflake or tools for model monitoring for feature stores, all of these integrate with Ray. And that's, you know, Ray provides that kind of flexibility so that you can integrate it into the rest of your workflow. And then Anyscale is the scalable compute platform that's built on top, you know, that provides Ray. So Anyscale is a managed Ray service that runs in the Cloud. And what Anyscale does is it offers the best way to run Ray. And if you think about what you get with Anyscale, there are fundamentally two things. One is about moving faster, accelerating the time to market. And you get that by having the managed service so that as a developer you don't have to worry about managing infrastructure, you don't have to worry about configuring infrastructure. You also, it provides, you know, optimized developer workflows. Things like easily moving from development to production, things like having the observability tooling, the debug ability to actually easily diagnose what's going wrong in a distributed application. So things like the dashboards and the other other kinds of tooling for collaboration, for monitoring and so on. And then on top of that, so that's the first bucket, developer productivity, moving faster, faster experimentation and iteration. The second reason that people choose Anyscale is superior infrastructure. So this is things like, you know, cost deficiency, being able to easily take advantage of spot instances, being able to get higher GPU utilization, things like faster cluster startup times and auto scaling. Things like just overall better performance and faster scheduling. And so these are the kinds of things that Anyscale provides on top of Ray. It's the managed infrastructure. It's fast, it's like the developer productivity and velocity as well as performance. So this is what I wanted to share about Ray in Anyscale. >> John: Awesome. >> Provide that context. But John, I'm curious what you think. >> I love it. I love the, so first of all, it's a platform because that's the platform architecture right there. So just to clarify, this is an Anyscale platform, not- >> That's right. >> Tools. So you got tools in the platform. Okay, that's key. Love that managed service. Just curious, you mentioned Python multiple times, is that because of PyTorch and TensorFlow or Python's the most friendly with machine learning or it's because it's very common amongst all developers? >> That's a great question. Python is the language that people are using to do machine learning. So it's the natural starting point. Now, of course, Ray is actually designed in a language agnostic way and there are companies out there that use Ray to build scalable Java applications. But for the most part right now we're focused on Python and being the best way to build these scalable Python and machine learning applications. But, of course, down the road there always is that potential. >> So if you're slinging Python code out there and you're watching that, you're watching this video, get on Anyscale bus quickly. Also, I just, while you were giving the presentation, I couldn't help, since you mentioned OpenAI, which by the way, congratulations 'cause they've had great scale, I've noticed in their rapid growth 'cause they were the fastest company to the number of users than anyone in the history of the computer industry, so major successor, OpenAI and ChatGPT, huge fan. I'm not a skeptic at all. I think it's just the beginning, so congratulations. But I actually typed into ChatGPT, what are the top three benefits of Anyscale and came up with scalability, flexibility, and ease of use. Obviously, scalability is what you guys are called. >> That's pretty good. >> So that's what they came up with. So they nailed it. Did you have an inside prompt training, buy it there? Only kidding. (Robert laughs) >> Yeah, we hard coded that one. >> But that's the kind of thing that came up really, really quickly if I asked it to write a sales document, it probably will, but this is the future interface. This is why people are getting excited about the foundational models and the large language models because it's allowing the interface with the user, the consumer, to be more human, more natural. And this is clearly will be in every application in the future. >> Absolutely. This is how people are going to interface with software, how they're going to interface with products in the future. It's not just something, you know, not just a chat bot that you talk to. This is going to be how you get things done, right. How you use your web browser or how you use, you know, how you use Photoshop or how you use other products. Like you're not going to spend hours learning all the APIs and how to use them. You're going to talk to it and tell it what you want it to do. And of course, you know, if it doesn't understand it, it's going to ask clarifying questions. You're going to have a conversation and then it'll figure it out. >> This is going to be one of those things, we're going to look back at this time Robert and saying, "Yeah, from that company, that was the beginning of that wave." And just like AWS and Cloud Computing, the folks who got in early really were in position when say the pandemic came. So getting in early is a good thing and that's what everyone's talking about is getting in early and playing around, maybe replatforming or even picking one or few apps to refactor with some staff and managed services. So people are definitely jumping in. So I have to ask you the ROI cost question. You mentioned some of those, Moore's Law versus what's going on in the industry. When you look at that kind of scale, the first thing that jumps out at people is, "Okay, I love it. Let's go play around." But what's it going to cost me? Am I going to be tied to certain GPUs? What's the landscape look like from an operational standpoint, from the customer? Are they locked in and the benefit was flexibility, are you flexible to handle any Cloud? What is the customers, what are they looking at? Basically, that's my question. What's the customer looking at? >> Cost is super important here and many of the companies, I mean, companies are spending a huge amount on their Cloud computing, on AWS, and on doing AI, right. And I think a lot of the advantage of Anyscale, what we can provide here is not only better performance, but cost efficiency. Because if we can run something faster and more efficiently, it can also use less resources and you can lower your Cloud spending, right. We've seen companies go from, you know, 20% GPU utilization with their current setup and the current tools they're using to running on Anyscale and getting more like 95, you know, 100% GPU utilization. That's something like a five x improvement right there. So depending on the kind of application you're running, you know, it's a significant cost savings. We've seen companies that have, you know, processing petabytes of data every single day with Ray going from, you know, getting order of magnitude cost savings by switching from what they were previously doing to running their application on Ray. And when you have applications that are spending, you know, potentially $100 million a year and getting a 10 X cost savings is just absolutely enormous. So these are some of the kinds of- >> Data infrastructure is super important. Again, if the customer, if you're a prospect to this and thinking about going in here, just like the Cloud, you got infrastructure, you got the platform, you got SaaS, same kind of thing's going to go on in AI. So I want to get into that, you know, ROI discussion and some of the impact with your customers that are leveraging the platform. But first I hear you got a demo. >> Robert: Yeah, so let me show you, let me give you a quick run through here. So what I have open here is the Anyscale UI. I've started a little Anyscale Workspace. So Workspaces are the Anyscale concept for interactive developments, right. So here, imagine I'm just, you want to have a familiar experience like you're developing on your laptop. And here I have a terminal. It's not on my laptop. It's actually in the cloud running on Anyscale. And I'm just going to kick this off. This is going to train a large language model, so OPT. And it's doing this on 32 GPUs. We've got a cluster here with a bunch of CPU cores, bunch of memory. And as that's running, and by the way, if I wanted to run this on instead of 32 GPUs, 64, 128, this is just a one line change when I launch the Workspace. And what I can do is I can pull up VS code, right. Remember this is the interactive development experience. I can look at the actual code. Here it's using Ray train to train the torch model. We've got the training loop and we're saying that each worker gets access to one GPU and four CPU cores. And, of course, as I make the model larger, this is using deep speed, as I make the model larger, I could increase the number of GPUs that each worker gets access to, right. And how that is distributed across the cluster. And if I wanted to run on CPUs instead of GPUs or a different, you know, accelerator type, again, this is just a one line change. And here we're using Ray train to train the models, just taking my vanilla PyTorch model using Hugging Face and then scaling that across a bunch of GPUs. And, of course, if I want to look at the dashboard, I can go to the Ray dashboard. There are a bunch of different visualizations I can look at. I can look at the GPU utilization. I can look at, you know, the CPU utilization here where I think we're currently loading the model and running that actual application to start the training. And some of the things that are really convenient here about Anyscale, both I can get that interactive development experience with VS code. You know, I can look at the dashboards. I can monitor what's going on. It feels, I have a terminal, it feels like my laptop, but it's actually running on a large cluster. And I can, with however many GPUs or other resources that I want. And so it's really trying to combine the best of having the familiar experience of programming on your laptop, but with the benefits, you know, being able to take advantage of all the resources in the Cloud to scale. And it's like when, you know, you're talking about cost efficiency. One of the biggest reasons that people waste money, one of the silly reasons for wasting money is just forgetting to turn off your GPUs. And what you can do here is, of course, things will auto terminate if they're idle. But imagine you go to sleep, I have this big cluster. You can turn it off, shut off the cluster, come back tomorrow, restart the Workspace, and you know, your big cluster is back up and all of your code changes are still there. All of your local file edits. It's like you just closed your laptop and came back and opened it up again. And so this is the kind of experience we want to provide for our users. So that's what I wanted to share with you. >> Well, I think that whole, couple of things, lines of code change, single line of code change, that's game changing. And then the cost thing, I mean human error is a big deal. People pass out at their computer. They've been coding all night or they just forget about it. I mean, and then it's just like leaving the lights on or your water running in your house. It's just, at the scale that it is, the numbers will add up. That's a huge deal. So I think, you know, compute back in the old days, there's no compute. Okay, it's just compute sitting there idle. But you know, data cranking the models is doing, that's a big point. >> Another thing I want to add there about cost efficiency is that we make it really easy to use, if you're running on Anyscale, to use spot instances and these preemptable instances that can just be significantly cheaper than the on-demand instances. And so when we see our customers go from what they're doing before to using Anyscale and they go from not using these spot instances 'cause they don't have the infrastructure around it, the fault tolerance to handle the preemption and things like that, to being able to just check a box and use spot instances and save a bunch of money. >> You know, this was my whole, my feature article at Reinvent last year when I met with Adam Selipsky, this next gen Cloud is here. I mean, it's not auto scale, it's infrastructure scale. It's agility. It's flexibility. I think this is where the world needs to go. Almost what DevOps did for Cloud and what you were showing me that demo had this whole SRE vibe. And remember Google had site reliability engines to manage all those servers. This is kind of like an SRE vibe for data at scale. I mean, a similar kind of order of magnitude. I mean, I might be a little bit off base there, but how would you explain it? >> It's a nice analogy. I mean, what we are trying to do here is get to the point where developers don't think about infrastructure. Where developers only think about their application logic. And where businesses can do AI, can succeed with AI, and build these scalable applications, but they don't have to build, you know, an infrastructure team. They don't have to develop that expertise. They don't have to invest years in building their internal machine learning infrastructure. They can just focus on the Python code, on their application logic, and run the stuff out of the box. >> Awesome. Well, I appreciate the time. Before we wrap up here, give a plug for the company. I know you got a couple websites. Again, go, Ray's got its own website. You got Anyscale. You got an event coming up. Give a plug for the company looking to hire. Put a plug in for the company. >> Yeah, absolutely. Thank you. So first of all, you know, we think AI is really going to transform every industry and the opportunity is there, right. We can be the infrastructure that enables all of that to happen, that makes it easy for companies to succeed with AI, and get value out of AI. Now we have, if you're interested in learning more about Ray, Ray has been emerging as the standard way to build scalable applications. Our adoption has been exploding. I mentioned companies like OpenAI using Ray to train their models. But really across the board companies like Netflix and Cruise and Instacart and Lyft and Uber, you know, just among tech companies. It's across every industry. You know, gaming companies, agriculture, you know, farming, robotics, drug discovery, you know, FinTech, we see it across the board. And all of these companies can get value out of AI, can really use AI to improve their businesses. So if you're interested in learning more about Ray and Anyscale, we have our Ray Summit coming up in September. This is going to highlight a lot of the most impressive use cases and stories across the industry. And if your business, if you want to use LLMs, you want to train these LLMs, these large language models, you want to fine tune them with your data, you want to deploy them, serve them, and build applications and products around them, give us a call, talk to us. You know, we can really take the infrastructure piece, you know, off the critical path and make that easy for you. So that's what I would say. And, you know, like you mentioned, we're hiring across the board, you know, engineering, product, go-to-market, and it's an exciting time. >> Robert Nishihara, co-founder and CEO of Anyscale, congratulations on a great company you've built and continuing to iterate on and you got growth ahead of you, you got a tailwind. I mean, the AI wave is here. I think OpenAI and ChatGPT, a customer of yours, have really opened up the mainstream visibility into this new generation of applications, user interface, roll of data, large scale, how to make that programmable so we're going to need that infrastructure. So thanks for coming on this season three, episode one of the ongoing series of the hot startups. In this case, this episode is the top startups building foundational model infrastructure for AI and ML. I'm John Furrier, your host. Thanks for watching. (upbeat music)