(upbeat music) >> Hello and welcome to theCube's coverage of International Women's Day. I'm John Furrier, your host of "theCUBE." We've got great two remote guests coming into our Palo Alto Studios, some tech athletes, as we say, people that've been in the trenches, years of experience, Lena Smart, CISO at MongoDB, Cube alumni, and Tara Hernandez, VP of Developer Productivity at MongoDB as well. Thanks for coming in to this program and supporting our efforts today. Thanks so much. >> Thanks for having us. >> Yeah, everyone talk about the journey in tech, where it all started. Before we get there, talk about what you guys are doing at MongoDB specifically. MongoDB is kind of gone the next level as a platform. You have your own ecosystem, lot of developers, very technical crowd, but it's changing the business transformation. What do you guys do at Mongo? We'll start with you, Lena. >> So I'm the CISO, so all security goes through me. I like to say, well, I don't like to say, I'm described as the ones throat to choke. So anything to do with security basically starts and ends with me. We do have a fantastic Cloud engineering security team and a product security team, and they don't report directly to me, but obviously we have very close relationships. I like to keep that kind of church and state separate and I know I've spoken about that before. And we just recently set up a physical security team with an amazing gentleman who left the FBI and he came to join us after 26 years for the agency. So, really starting to look at the physical aspects of what we offer as well. >> I interviewed a CISO the other day and she said, "Every day is day zero for me." Kind of goofing on the Amazon Day one thing, but Tara, go ahead. Tara, go ahead. What's your role there, developer productivity? What are you focusing on? >> Sure. Developer productivity is kind of the latest description for things that we've described over the years as, you know, DevOps oriented engineering or platform engineering or build and release engineering development infrastructure. It's all part and parcel, which is how do we actually get our code from developer to customer, you know, and all the mechanics that go into that. It's been something I discovered from my first job way back in the early '90s at Borland. And the art has just evolved enormously ever since, so. >> Yeah, this is a very great conversation both of you guys, right in the middle of all the action and data infrastructures changing, exploding, and involving big time AI and data tsunami and security never stops. Well, let's get into, we'll talk about that later, but let's get into what motivated you guys to pursue a career in tech and what were some of the challenges that you faced along the way? >> I'll go first. The fact of the matter was I intended to be a double major in history and literature when I went off to university, but I was informed that I had to do a math or a science degree or else the university would not be paid for. At the time, UC Santa Cruz had a policy that called Open Access Computing. This is, you know, the late '80s, early '90s. And anybody at the university could get an email account and that was unusual at the time if you were, those of us who remember, you used to have to pay for that CompuServe or AOL or, there's another one, I forget what it was called, but if a student at Santa Cruz could have an email account. And because of that email account, I met people who were computer science majors and I'm like, "Okay, I'll try that." That seems good. And it was a little bit of a struggle for me, a lot I won't lie, but I can't complain with how it ended up. And certainly once I found my niche, which was development infrastructure, I found my true love and I've been doing it for almost 30 years now. >> Awesome. Great story. Can't wait to ask a few questions on that. We'll go back to that late '80s, early '90s. Lena, your journey, how you got into it. >> So slightly different start. I did not go to university. I had to leave school when I was 16, got a job, had to help support my family. Worked a bunch of various jobs till I was about 21 and then computers became more, I think, I wouldn't say they were ubiquitous, but they were certainly out there. And I'd also been saving up every penny I could earn to buy my own computer and bought an Amstrad 1640, 20 meg hard drive. It rocked. And kind of took that apart, put it back together again, and thought that could be money in this. And so basically just teaching myself about computers any job that I got. 'Cause most of my jobs were like clerical work and secretary at that point. But any job that had a computer in front of that, I would make it my business to go find the guy who did computing 'cause it was always a guy. And I would say, you know, I want to learn how these work. Let, you know, show me. And, you know, I would take my lunch hour and after work and anytime I could with these people and they were very kind with their time and I just kept learning, so yep. >> Yeah, those early days remind me of the inflection point we're going through now. This major C change coming. Back then, if you had a computer, you had to kind of be your own internal engineer to fix things. Remember back on the systems revolution, late '80s, Tara, when, you know, your career started, those were major inflection points. Now we're seeing a similar wave right now, security, infrastructure. It feels like it's going to a whole nother level. At Mongo, you guys certainly see this as well, with this AI surge coming in. A lot more action is coming in. And so there's a lot of parallels between these inflection points. How do you guys see this next wave of change? Obviously, the AI stuff's blowing everyone away. Oh, new user interface. It's been called the browser moment, the mobile iPhone moment, kind of for this generation. There's a lot of people out there who are watching that are young in their careers, what's your take on this? How would you talk to those folks around how important this wave is? >> It, you know, it's funny, I've been having this conversation quite a bit recently in part because, you know, to me AI in a lot of ways is very similar to, you know, back in the '90s when we were talking about bringing in the worldwide web to the forefront of the world, right. And we tended to think in terms of all the optimistic benefits that would come of it. You know, free passing of information, availability to anyone, anywhere. You just needed an internet connection, which back then of course meant a modem. >> John: Not everyone had though. >> Exactly. But what we found in the subsequent years is that human beings are what they are and we bring ourselves to whatever platforms that are there, right. And so, you know, as much as it was amazing to have this freely available HTML based internet experience, it also meant that the negatives came to the forefront quite quickly. And there were ramifications of that. And so to me, when I look at AI, we're already seeing the ramifications to that. Yes, are there these amazing, optimistic, wonderful things that can be done? Yes. >> Yeah. >> But we're also human and the bad stuff's going to come out too. And how do we- >> Yeah. >> How do we as an industry, as a community, you know, understand and mitigate those ramifications so that we can benefit more from the positive than the negative. So it is interesting that it comes kind of full circle in really interesting ways. >> Yeah. The underbelly takes place first, gets it in the early adopter mode. Normally industries with, you know, money involved arbitrage, no standards. But we've seen this movie before. Is there hope, Lena, that we can have a more secure environment? >> I would hope so. (Lena laughs) Although depressingly, we've been in this well for 30 years now and we're, at the end of the day, still telling people not to click links on emails. So yeah, that kind of still keeps me awake at night a wee bit. The whole thing about AI, I mean, it's, obviously I am not an expert by any stretch of the imagination in AI. I did read (indistinct) book recently about AI and that was kind of interesting. And I'm just trying to teach myself as much as I can about it to the extent of even buying the "Dummies Guide to AI." Just because, it's actually not a dummies guide. It's actually fairly interesting, but I'm always thinking about it from a security standpoint. So it's kind of my worst nightmare and the best thing that could ever happen in the same dream. You know, you've got this technology where I can ask it a question and you know, it spits out generally a reasonable answer. And my team are working on with Mark Porter our CTO and his team on almost like an incubation of AI link. What would it look like from MongoDB? What's the legal ramifications? 'Cause there will be legal ramifications even though it's the wild, wild west just now, I think. Regulation's going to catch up to us pretty quickly, I would think. >> John: Yeah, yeah. >> And so I think, you know, as long as companies have a seat at the table and governments perhaps don't become too dictatorial over this, then hopefully we'll be in a good place. But we'll see. I think it's a really interest, there's that curse, we're living in interesting times. I think that's where we are. >> It's interesting just to stay on this tech trend for a minute. The standards bodies are different now. Back in the old days there were, you know, IEEE standards, ITF standards. >> Tara: TPC. >> The developers are the new standard. I mean, now you're seeing open source completely different where it was in the '90s to here beginning, that was gen one, some say gen two, but I say gen one, now we're exploding with open source. You have kind of developers setting the standards. If developers like it in droves, it becomes defacto, which then kind of rolls into implementation. >> Yeah, I mean I think if you don't have developer input, and this is why I love working with Tara and her team so much is 'cause they get it. If we don't have input from developers, it's not going to get used. There's going to be ways of of working around it, especially when it comes to security. If they don't, you know, if you're a developer and you're sat at your screen and you don't want to do that particular thing, you're going to find a way around it. You're a smart person. >> Yeah. >> So. >> Developers on the front lines now versus, even back in the '90s, they're like, "Okay, consider the dev's, got a QA team." Everything was Waterfall, now it's Cloud, and developers are on the front lines of everything. Tara, I mean, this is where the standards are being met. What's your reaction to that? >> Well, I think it's outstanding. I mean, you know, like I was at Netscape and part of the crowd that released the browser as open source and we founded mozilla.org, right. And that was, you know, in many ways kind of the birth of the modern open source movement beyond what we used to have, what was basically free software foundation was sort of the only game in town. And I think it is so incredibly valuable. I want to emphasize, you know, and pile onto what Lena was saying, it's not just that the developers are having input on a sort of company by company basis. Open source to me is like a checks and balance, where it allows us as a broader community to be able to agree on and enforce certain standards in order to try and keep the technology platforms as accessible as possible. I think Kubernetes is a great example of that, right. If we didn't have Kubernetes, that would've really changed the nature of how we think about container orchestration. But even before that, Linux, right. Linux allowed us as an industry to end the Unix Wars and as someone who was on the front lines of that as well and having to support 42 different operating systems with our product, you know, that was a huge win. And it allowed us to stop arguing about operating systems and start arguing about software or not arguing, but developing it in positive ways. So with, you know, with Kubernetes, with container orchestration, we all agree, okay, that's just how we're going to orchestrate. Now we can build up this huge ecosystem, everybody gets taken along, right. And now it changes the game for what we're defining as business differentials, right. And so when we talk about crypto, that's a little bit harder, but certainly with AI, right, you know, what are the checks and balances that as an industry and as the developers around this, that we can in, you know, enforce to make sure that no one company or no one body is able to overly control how these things are managed, how it's defined. And I think that is only for the benefit in the industry as a whole, particularly when we think about the only other option is it gets regulated in ways that do not involve the people who actually know the details of what they're talking about. >> Regulated and or thrown away or bankrupt or- >> Driven underground. >> Yeah. >> Which would be even worse actually. >> Yeah, that's a really interesting, the checks and balances. I love that call out. And I was just talking with another interview part of the series around women being represented in the 51% ratio. Software is for everybody. So that we believe that open source movement around the collective intelligence of the participants in the industry and independent of gender, this is going to be the next wave. You're starting to see these videos really have impact because there are a lot more leaders now at the table in companies developing software systems and with AI, the aperture increases for applications. And this is the new dynamic. What's your guys view on this dynamic? How does this go forward in a positive way? Is there a certain trajectory you see? For women in the industry? >> I mean, I think some of the states are trying to, again, from the government angle, some of the states are trying to force women into the boardroom, for example, California, which can be no bad thing, but I don't know, sometimes I feel a bit iffy about all this kind of forced- >> John: Yeah. >> You know, making, I don't even know how to say it properly so you can cut this part of the interview. (John laughs) >> Tara: Well, and I think that they're >> I'll say it's not organic. >> No, and I think they're already pulling it out, right. It's already been challenged so they're in the process- >> Well, this is the open source angle, Tara, you are getting at it. The change agent is open, right? So to me, the history of the proven model is openness drives transparency drives progress. >> No, it's- >> If you believe that to be true, this could have another impact. >> Yeah, it's so interesting, right. Because if you look at McKinsey Consulting or Boston Consulting or some of the other, I'm blocking on all of the names. There has been a decade or more of research that shows that a non homogeneous employee base, be it gender or ethnicity or whatever, generates more revenue, right? There's dollar signs that can be attached to this, but it's not enough for all companies to want to invest in that way. And it's not enough for all, you know, venture firms or investment firms to grant that seed money or do those seed rounds. I think it's getting better very slowly, but socialization is a much harder thing to overcome over time. Particularly, when you're not just talking about one country like the United States in our case, but around the world. You know, tech centers now exist all over the world, including places that even 10 years ago we might not have expected like Nairobi, right. Which I think is amazing, but you have to factor in the cultural implications of that as well, right. So yes, the openness is important and we have, it's important that we have those voices, but I don't think it's a panacea solution, right. It's just one more piece. I think honestly that one of the most important opportunities has been with Cloud computing and Cloud's been around for a while. So why would I say that? It's because if you think about like everybody holds up the Steve Jobs, Steve Wozniak, back in the '70s, or Sergey and Larry for Google, you know, you had to have access to enough credit card limit to go to Fry's and buy your servers and then access to somebody like Susan Wojcicki to borrow the garage or whatever. But there was still a certain amount of upfrontness that you had to be able to commit to, whereas now, and we've, I think, seen a really good evidence of this being able to lease server resources by the second and have development platforms that you can do on your phone. I mean, for a while I think Africa, that the majority of development happened on mobile devices because there wasn't a sufficient supply chain of laptops yet. And that's no longer true now as far as I know. But like the power that that enables for people who would otherwise be underrepresented in our industry instantly opens it up, right? And so to me that's I think probably the biggest opportunity that we've seen from an industry on how to make more availability in underrepresented representation for entrepreneurship. >> Yeah. >> Something like AI, I think that's actually going to take us backwards if we're not careful. >> Yeah. >> Because of we're reinforcing that socialization. >> Well, also the bias. A lot of people commenting on the biases of the large language inherently built in are also problem. Lena, I want you to weigh on this too, because I think the skills question comes up here and I've been advocating that you don't need the pedigree, college pedigree, to get into a certain jobs, you mentioned Cloud computing. I mean, it's been around for you think a long time, but not really, really think about it. The ability to level up, okay, if you're going to join something new and half the jobs in cybersecurity are created in the past year, right? So, you have this what used to be a barrier, your degree, your pedigree, your certification would take years, would be a blocker. Now that's gone. >> Lena: Yeah, it's the opposite. >> That's, in fact, psychology. >> I think so, but the people who I, by and large, who I interview for jobs, they have, I think security people and also I work with our compliance folks and I can't forget them, but let's talk about security just now. I've always found a particular kind of mindset with security folks. We're very curious, not very good at following rules a lot of the time, and we'd love to teach others. I mean, that's one of the big things stem from the start of my career. People were always interested in teaching and I was interested in learning. So it was perfect. And I think also having, you know, strong women leaders at MongoDB allows other underrepresented groups to actually apply to the company 'cause they see that we're kind of talking the talk. And that's been important. I think it's really important. You know, you've got Tara and I on here today. There's obviously other senior women at MongoDB that you can talk to as well. There's a bunch of us. There's not a whole ton of us, but there's a bunch of us. And it's good. It's definitely growing. I've been there for four years now and I've seen a growth in women in senior leadership positions. And I think having that kind of track record of getting really good quality underrepresented candidates to not just interview, but come and join us, it's seen. And it's seen in the industry and people take notice and they're like, "Oh, okay, well if that person's working, you know, if Tara Hernandez is working there, I'm going to apply for that." And that in itself I think can really, you know, reap the rewards. But it's getting started. It's like how do you get your first strong female into that position or your first strong underrepresented person into that position? It's hard. I get it. If it was easy, we would've sold already. >> It's like anything. I want to see people like me, my friends in there. Am I going to be alone? Am I going to be of a group? It's a group psychology. Why wouldn't? So getting it out there is key. Is there skills that you think that people should pay attention to? One's come up as curiosity, learning. What are some of the best practices for folks trying to get into the tech field or that's in the tech field and advancing through? What advice are you guys- >> I mean, yeah, definitely, what I say to my team is within my budget, we try and give every at least one training course a year. And there's so much free stuff out there as well. But, you know, keep learning. And even if it's not right in your wheelhouse, don't pick about it. Don't, you know, take a look at what else could be out there that could interest you and then go for it. You know, what does it take you few minutes each night to read a book on something that might change your entire career? You know, be enthusiastic about the opportunities out there. And there's so many opportunities in security. Just so many. >> Tara, what's your advice for folks out there? Tons of stuff to taste, taste test, try things. >> Absolutely. I mean, I always say, you know, my primary qualifications for people, I'm looking for them to be smart and motivated, right. Because the industry changes so quickly. What we're doing now versus what we did even last year versus five years ago, you know, is completely different though themes are certainly the same. You know, we still have to code and we still have to compile that code or package the code and ship the code so, you know, how well can we adapt to these new things instead of creating floppy disks, which was my first job. Five and a quarters, even. The big ones. >> That's old school, OG. There it is. Well done. >> And now it's, you know, containers, you know, (indistinct) image containers. And so, you know, I've gotten a lot of really great success hiring boot campers, you know, career transitioners. Because they bring a lot experience in addition to the technical skills. I think the most important thing is to experiment and figuring out what do you like, because, you know, maybe you are really into security or maybe you're really into like deep level coding and you want to go back, you know, try to go to school to get a degree where you would actually want that level of learning. Or maybe you're a front end engineer, you want to be full stacked. Like there's so many different things, data science, right. Maybe you want to go learn R right. You know, I think it's like figure out what you like because once you find that, that in turn is going to energize you 'cause you're going to feel motivated. I think the worst thing you could do is try to force yourself to learn something that you really could not care less about. That's just the worst. You're going in handicapped. >> Yeah and there's choices now versus when we were breaking into the business. It was like, okay, you software engineer. They call it software engineering, that's all it was. You were that or you were in sales. Like, you know, some sort of systems engineer or sales and now it's,- >> I had never heard of my job when I was in school, right. I didn't even know it was a possibility. But there's so many different types of technical roles, you know, absolutely. >> It's so exciting. I wish I was young again. >> One of the- >> Me too. (Lena laughs) >> I don't. I like the age I am. So one of the things that I did to kind of harness that curiosity is we've set up a security champions programs. About 120, I guess, volunteers globally. And these are people from all different backgrounds and all genders, diversity groups, underrepresented groups, we feel are now represented within this champions program. And people basically give up about an hour or two of their time each week, with their supervisors permission, and we basically teach them different things about security. And we've now had seven full-time people move from different areas within MongoDB into my team as a result of that program. So, you know, monetarily and time, yeah, saved us both. But also we're showing people that there is a path, you know, if you start off in Tara's team, for example, doing X, you join the champions program, you're like, "You know, I'd really like to get into red teaming. That would be so cool." If it fits, then we make that happen. And that has been really important for me, especially to give, you know, the women in the underrepresented groups within MongoDB just that window into something they might never have seen otherwise. >> That's a great common fit is fit matters. Also that getting access to what you fit is also access to either mentoring or sponsorship or some sort of, at least some navigation. Like what's out there and not being afraid to like, you know, just ask. >> Yeah, we just actually kicked off our big mentor program last week, so I'm the executive sponsor of that. I know Tara is part of it, which is fantastic. >> We'll put a plug in for it. Go ahead. >> Yeah, no, it's amazing. There's, gosh, I don't even know the numbers anymore, but there's a lot of people involved in this and so much so that we've had to set up mentoring groups rather than one-on-one. And I think it was 45% of the mentors are actually male, which is quite incredible for a program called Mentor Her. And then what we want to do in the future is actually create a program called Mentor Them so that it's not, you know, not just on the female and so that we can live other groups represented and, you know, kind of break down those groups a wee bit more and have some more granularity in the offering. >> Tara, talk about mentoring and sponsorship. Open source has been there for a long time. People help each other. It's community-oriented. What's your view of how to work with mentors and sponsors if someone's moving through ranks? >> You know, one of the things that was really interesting, unfortunately, in some of the earliest open source communities is there was a lot of pervasive misogyny to be perfectly honest. >> Yeah. >> And one of the important adaptations that we made as an open source community was the idea, an introduction of code of conducts. And so when I'm talking to women who are thinking about expanding their skills, I encourage them to join open source communities to have opportunity, even if they're not getting paid for it, you know, to develop their skills to work with people to get those code reviews, right. I'm like, "Whatever you join, make sure they have a code of conduct and a good leadership team. It's very important." And there are plenty, right. And then that idea has come into, you know, conferences now. So now conferences have codes of contact, if there are any good, and maybe not all of them, but most of them, right. And the ideas of expanding that idea of intentional healthy culture. >> John: Yeah. >> As a business goal and business differentiator. I mean, I won't lie, when I was recruited to come to MongoDB, the culture that I was able to discern through talking to people, in addition to seeing that there was actually women in senior leadership roles like Lena, like Kayla Nelson, that was a huge win. And so it just builds on momentum. And so now, you know, those of us who are in that are now representing. And so that kind of reinforces, but it's all ties together, right. As the open source world goes, particularly for a company like MongoDB, which has an open source product, you know, and our community builds. You know, it's a good thing to be mindful of for us, how we interact with the community and you know, because that could also become an opportunity for recruiting. >> John: Yeah. >> Right. So we, in addition to people who might become advocates on Mongo's behalf in their own company as a solution for themselves, so. >> You guys had great successful company and great leadership there. I mean, I can't tell you how many times someone's told me "MongoDB doesn't scale. It's going to be dead next year." I mean, I was going back 10 years. It's like, just keeps getting better and better. You guys do a great job. So it's so fun to see the success of developers. Really appreciate you guys coming on the program. Final question, what are you guys excited about to end the segment? We'll give you guys the last word. Lena will start with you and Tara, you can wrap us up. What are you excited about? >> I'm excited to see what this year brings. I think with ChatGPT and its copycats, I think it'll be a very interesting year when it comes to AI and always in the lookout for the authentic deep fakes that we see coming out. So just trying to make people aware that this is a real thing. It's not just pretend. And then of course, our old friend ransomware, let's see where that's going to go. >> John: Yeah. >> And let's see where we get to and just genuine hygiene and housekeeping when it comes to security. >> Excellent. Tara. >> Ah, well for us, you know, we're always constantly trying to up our game from a security perspective in the software development life cycle. But also, you know, what can we do? You know, one interesting application of AI that maybe Google doesn't like to talk about is it is really cool as an addendum to search and you know, how we might incorporate that as far as our learning environment and developer productivity, and how can we enable our developers to be more efficient, productive in their day-to-day work. So, I don't know, there's all kinds of opportunities that we're looking at for how we might improve that process here at MongoDB and then maybe be able to share it with the world. One of the things I love about working at MongoDB is we get to use our own products, right. And so being able to have this interesting document database in order to put information and then maybe apply some sort of AI to get it out again, is something that we may well be looking at, if not this year, then certainly in the coming year. >> Awesome. Lena Smart, the chief information security officer. Tara Hernandez, vice president developer of productivity from MongoDB. Thank you so much for sharing here on International Women's Day. We're going to do this quarterly every year. We're going to do it and then we're going to do quarterly updates. Thank you so much for being part of this program. >> Thank you. >> Thanks for having us. >> Okay, this is theCube's coverage of International Women's Day. I'm John Furrier, your host. Thanks for watching. (upbeat music)

(inspirational music) (music ends) >> Okay, so now that we've heard Scott talk about smart data fabrics, it's time to see this in action. Right now we're joined by Jess Jowdy, who's the manager of Healthcare Field Engineering at InterSystems. She's going to give a demo of how smart data fabrics actually work, and she's going to show how embedding a wide range of analytics capabilities, including data exploration business intelligence, natural language processing and machine learning directly within the fabric makes it faster and easier for organizations to gain new insights and power intelligence predictive and prescriptive services and applications. Now, according to InterSystems, smart data fabrics are applicable across many industries from financial services to supply chain to healthcare and more. Jess today is going to be speaking through the lens of a healthcare focused demo. Don't worry, Joe Lichtenberg will get into some of the other use cases that you're probably interested in hearing about. That will be in our third segment, but for now let's turn it over to Jess. Jess, good to see you. >> Hi, yeah, thank you so much for having me. And so for this demo, we're really going to be bucketing these features of a smart data fabric into four different segments. We're going to be dealing with connections, collections, refinements, and analysis. And so we'll see that throughout the demo as we go. So without further ado, let's just go ahead and jump into this demo, and you'll see my screen pop up here. I actually like to start at the end of the demo. So I like to begin by illustrating what an end user's going to see, and don't mind the screen 'cause I gave you a little sneak peek of what's about to happen. But essentially what I'm going to be doing is using Postman to simulate a call from an external application. So we talked about being in the healthcare industry. This could be, for instance, a mobile application that a patient is using to view an aggregated summary of information across that patient's continuity of care or some other kind of application. So we might be pulling information in this case from an electronic medical record. We might be grabbing clinical history from that. We might be grabbing clinical notes from a medical transcription software, or adverse reaction warnings from a clinical risk grouping application, and so much more. So I'm really going to be simulating a patient logging in on their phone and retrieving this information through this Postman call. So what I'm going to do is I'm just going to hit send, I've already preloaded everything here, and I'm going to be looking for information where the last name of this patient is Simmons, and their medical record number or their patient identifier in the system is 32345. And so as you can see, I have this single JSON payload that showed up here of, just, relevant clinical information for my patient whose last name is Simmons, all within a single response. So fantastic, right? Typically though, when we see responses that look like this there is an assumption that this service is interacting with a single backend system, and that single backend system is in charge of packaging that information up and returning it back to this caller. But in a smart data fabric architecture, we're able to expand the scope to handle information across different, in this case, clinical applications. So how did this actually happen? Let's peel back another layer and really take a look at what happened in the background. What you're looking at here is our mission control center for our smart data fabric. On the left we have our APIs that allow users to interact with particular services. On the right we have our connections to our different data silos. And in the middle here, we have our data fabric coordinator which is going to be in charge of this refinement and analysis, those key pieces of our smart data fabric. So let's look back and think about the example we just showed. I received an inbound request for information for a patient whose last name is Simmons. My end user is requesting to connect to that service, and that's happening here at my patient data retrieval API location. Users can define any number of different services and APIs depending on their use cases. And to that end, we do also support full life cycle API management within this platform. When you're dealing with APIs, I always like to make a little shout out on this, that you really want to make sure you have enough, like a granular enough security model to handle and limit which APIs and which services a consumer can interact with. In this IRIS platform, which we're talking about today we have a very granular role-based security model that allows you to handle that, but it's really important in a smart data fabric to consider who's accessing your data and in what context. >> Can I just interrupt you for a second, Jess? >> Yeah, please. >> So you were showing on the left hand side of the demo a couple of APIs. I presume that can be a very long list. I mean, what do you see as typical? >> I mean you could have hundreds of these APIs depending on what services an organization is serving up for their consumers. So yeah, we've seen hundreds of these services listed here. >> So my question is, obviously security is critical in the healthcare industry, and API securities are like, really hot topic these days. How do you deal with that? >> Yeah, and I think API security is interesting 'cause it can happen at so many layers. So, there's interactions with the API itself. So can I even see this API and leverage it? And then within an API call, you then have to deal with all right, which end points or what kind of interactions within that API am I allowed to do? What data am I getting back? And with healthcare data, the whole idea of consent to see certain pieces of data is critical. So, the way that we handle that is, like I said, same thing at different layers. There is access to a particular API, which can happen within the IRIS product, and also we see it happening with an API management layer, which has become a really hot topic with a lot of organizations. And then when it comes to data security, that really happens under the hood within your smart data fabric. So, that role-based access control becomes very important in assigning, you know, roles and permissions to certain pieces of information. Getting that granular becomes the cornerstone of the security. >> And that's been designed in, it's not a bolt on as they like to say. >> Absolutely. >> Okay, can we get into collect now? >> Of course, we're going to move on to the collection piece at this point in time, which involves pulling information from each of my different data silos to create an overall aggregated record. So commonly, each data source requires a different method for establishing connections and collecting this information. So for instance, interactions with an EMR may require leveraging a standard healthcare messaging format like Fire. Interactions with a homegrown enterprise data warehouse for instance, may use SQL. For a cloud-based solutions managed by a vendor, they may only allow you to use web service calls to pull data. So it's really important that your data fabric platform that you're using has the flexibility to connect to all of these different systems and applications. And I'm about to log out, so I'm going to (chuckles) keep my session going here. So therefore it's incredibly important that your data fabric has the flexibility to connect to all these different kinds of applications and data sources, and all these different kinds of formats and over all of these different kinds of protocols. So let's think back on our example here. I had four different applications that I was requesting information for to create that payload that we saw initially. Those are listed here under this operations section. So these are going out and connecting to downstream systems to pull information into my smart data fabric. What's great about the IRIS platform is, it has an embedded interoperability platform. So there's all of these native adapters that can support these common connections that we see for different kinds of applications. So using REST, or SOAP, or SQL, or FTP, regardless of that protocol, there's an adapter to help you work with that. And we also think of the types of formats that we typically see data coming in as in healthcare we have HL7, we have Fire, we have CCDs, across the industry, JSON is, you know, really hitting a market strong now, and XML payloads, flat files. We need to be able to handle all of these different kinds of formats over these different kinds of protocols. So to illustrate that, if I click through these when I select a particular connection on the right side panel, I'm going to see the different settings that are associated with that particular connection that allows me to collect information back into my smart data fabric. In this scenario, my connection to my chart script application in this example, communicates over a SOAP connection. When I'm grabbing information from my clinical risk grouping application I'm using a SQL based connection. When I'm connecting to my EMR, I'm leveraging a standard healthcare messaging format known as Fire, which is a REST based protocol. And then when I'm working with my health record management system, I'm leveraging a standard HTTP adapter. So you can see how we can be flexible when dealing with these different kinds of applications and systems. And then it becomes important to be able to validate that you've established those connections correctly, and be able to do it in a reliable and quick way. Because if you think about it, you could have hundreds of these different kinds of applications built out and you want to make sure that you're maintaining and understanding those connections. So I can actually go ahead and test one of these applications and put in, for instance my patient's last name and their MRN, and make sure that I'm actually getting data back from that system. So it's a nice little sanity check as we're building out that data fabric to ensure that we're able to establish these connections appropriately. So turnkey adapters are fantastic, as you can see we're leveraging them all here, but sometimes these connections are going to require going one step further and building something really specific for an application. So why don't we go one step further here and talk about doing something custom or doing something innovative. And so it's important for users to have the ability to develop and go beyond what's an out-of-the box or black box approach to be able to develop things that are specific to their data fabric, or specific to their particular connection. In this scenario, the IRIS data platform gives users access to the entire underlying code base. So you not only get an opportunity to view how we're establishing these connections or how we're building out these processes, but you have the opportunity to inject your own kind of processing, your own kinds of pipelines into this. So as an example, you can leverage any number of different programming languages right within this pipeline. And so I went ahead and I injected Python. So Python is a very up and coming language, right? We see more and more developers turning towards Python to do their development. So it's important that your data fabric supports those kinds of developers and users that have standardized on these kinds of programming languages. This particular script here, as you can see actually calls out to our turnkey adapters. So we see a combination of out-of-the-box code that is provided in this data fabric platform from IRIS, combined with organization specific or user specific customizations that are included in this Python method. So it's a nice little combination of how do we bring the developer experience in and mix it with out-of-the-box capabilities that we can provide in a smart data fabric. >> Wow. >> Yeah, I'll pause. (laughs) >> It's a lot here. You know, actually- >> I can pause. >> If I could, if we just want to sort of play that back. So we went to the connect and the collect phase. >> Yes, we're going into refine. So it's a good place to stop. >> So before we get there, so we heard a lot about fine grain security, which is crucial. We heard a lot about different data types, multiple formats. You've got, you know, the ability to bring in different dev tools. We heard about Fire, which of course big in healthcare. And that's the standard, and then SQL for traditional kind of structured data, and then web services like HTTP you mentioned. And so you have a rich collection of capabilities within this single platform. >> Absolutely. And I think that's really important when you're dealing with a smart data fabric because what you're effectively doing is you're consolidating all of your processing, all of your collection, into a single platform. So that platform needs to be able to handle any number of different kinds of scenarios and technical challenges. So you've got to pack that platform with as many of these features as you can to consolidate that processing. >> All right, so now we're going into refinement. >> We're going into refinement. Exciting. (chuckles) So how do we actually do refinement? Where does refinement happen? And how does this whole thing end up being performant? Well the key to all of that is this SDF coordinator, or stands for Smart Data Fabric coordinator. And what this particular process is doing is essentially orchestrating all of these calls to all of these different downstream systems. It's aggregating, it's collecting that information, it's aggregating it, and it's refining it into that single payload that we saw get returned to the user. So really this coordinator is the main event when it comes to our data fabric. And in the IRIS platform we actually allow users to build these coordinators using web-based tool sets to make it intuitive. So we can take a sneak peek at what that looks like. And as you can see, it follows a flow chart like structure. So there's a start, there is an end, and then there are these different arrows that point to different activities throughout the business process. And so there's all these different actions that are being taken within our coordinator. You can see an action for each of the calls to each of our different data sources to go retrieve information. And then we also have the sync call at the end that is in charge of essentially making sure that all of those responses come back before we package them together and send them out. So this becomes really crucial when we're creating that data fabric. And you know, this is a very simple data fabric example where we're just grabbing data and we're consolidating it together. But you can have really complex orchestrators and coordinators that do any number of different things. So for instance, I could inject SQL logic into this or SQL code, I can have conditional logic, I can do looping, I can do error trapping and handling. So we're talking about a whole number of different features that can be included in this coordinator. So like I said, we have a really very simple process here that's just calling out, grabbing all those different data elements from all those different data sources and consolidating it. We'll look back at this coordinator in a second when we introduce, or we make this data fabric a bit smarter, and we start introducing that analytics piece to it. So this is in charge of the refinement. And so at this point in time we've looked at connections, collections, and refinements. And just to summarize what we've seen 'cause I always like to go back and take a look at everything that we've seen. We have our initial API connection, we have our connections to our individual data sources and we have our coordinators there in the middle that are in charge of collecting the data and refining it into a single payload. As you can imagine, there's a lot going on behind the scenes of a smart data fabric, right? There's all these different processes that are interacting. So it's really important that your smart data fabric platform has really good traceability, really good logging, 'cause you need to be able to know, you know, if there was an issue, where did that issue happen in which connected process, and how did it affect the other processes that are related to it? In IRIS, we have this concept called a visual trace. And what our clients use this for is basically to be able to step through the entire history of a request from when it initially came into the smart data fabric, to when data was sent back out from that smart data fabric. So I didn't record the time, but I bet if you recorded the time it was this time that we sent that request in and you can see my patient's name and their medical record number here, and you can see that that instigated four different calls to four different systems, and they're represented by these arrows going out. So we sent something to chart script, to our health record management system, to our clinical risk grouping application, into my EMR through their Fire server. So every request, every outbound application gets a request and we pull back all of those individual pieces of information from all of those different systems, and we bundle them together. And from my Fire lovers, here's our Fire bundle that we got back from our Fire server. So this is a really good way of being able to validate that I am appropriately grabbing the data from all these different applications and then ultimately consolidating it into one payload. Now we change this into a JSON format before we deliver it, but this is those data elements brought together. And this screen would also be used for being able to see things like error trapping, or errors that were thrown, alerts, warnings, developers might put log statements in just to validate that certain pieces of code are executing. So this really becomes the one stop shop for understanding what's happening behind the scenes with your data fabric. >> Sure, who did what when where, what did the machine do what went wrong, and where did that go wrong? Right at your fingertips. >> Right. And I'm a visual person so a bunch of log files to me is not the most helpful. While being able to see this happened at this time in this location, gives me that understanding I need to actually troubleshoot a problem. >> This business orchestration piece, can you say a little bit more about that? How people are using it? What's the business impact of the business orchestration? >> The business orchestration, especially in the smart data fabric, is really that crucial part of being able to create a smart data fabric. So think of your business orchestrator as doing the heavy lifting of any kind of processing that involves data, right? It's bringing data in, it's analyzing that information it's transforming that data, in a format that your consumer's not going to understand. It's doing any additional injection of custom logic. So really your coordinator or that orchestrator that sits in the middle is the brains behind your smart data fabric. >> And this is available today? It all works? >> It's all available today. Yeah, it all works. And we have a number of clients that are using this technology to support these kinds of use cases. >> Awesome demo. Anything else you want to show us? >> Well, we can keep going. I have a lot to say, but really this is our data fabric. The core competency of IRIS is making it smart, right? So I won't spend too much time on this, but essentially if we go back to our coordinator here, we can see here's that original, that pipeline that we saw where we're pulling data from all these different systems and we're collecting it and we're sending it out. But then we see two more at the end here, which involves getting a readmission prediction and then returning a prediction. So we can not only deliver data back as part of a smart data fabric, but we can also deliver insights back to users and consumers based on data that we've aggregated as part of a smart data fabric. So in this scenario, we're actually taking all that data that we just looked at, and we're running it through a machine learning model that exists within the smart data fabric pipeline, and producing a readmission score to determine if this particular patient is at risk for readmission within the next 30 days. Which is a typical problem that we see in the healthcare space. So what's really exciting about what we're doing in the IRIS world, is we're bringing analytics close to the data with integrated ML. So in this scenario we're actually creating the model, training the model, and then executing the model directly within the IRIS platform. So there's no shuffling of data, there's no external connections to make this happen. And it doesn't really require having a PhD in data science to understand how to do that. It leverages all really basic SQL-like syntax to be able to construct and execute these predictions. So, it's going one step further than the traditional data fabric example to introduce this ability to define actionable insights to our users based on the data that we've brought together. >> Well that readmission probability is huge, right? Because it directly affects the cost for the provider and the patient, you know. So if you can anticipate the probability of readmission and either do things at that moment, or, you know, as an outpatient perhaps, to minimize the probability then that's huge. That drops right to the bottom line. >> Absolutely. And that really brings us from that data fabric to that smart data fabric at the end of the day, which is what makes this so exciting. >> Awesome demo. >> Thank you! >> Jess, are you cool if people want to get in touch with you? Can they do that? >> Oh yes, absolutely. So you can find me on LinkedIn, Jessica Jowdy, and we'd love to hear from you. I always love talking about this topic so we'd be happy to engage on that. >> Great stuff. Thank you Jessica, appreciate it. >> Thank you so much. >> Okay, don't go away because in the next segment, we're going to dig into the use cases where data fabric is driving business value. Stay right there. (inspirational music) (music fades)

(intro music) >> Now, as we all know, businesses are awash with data, from financial services to healthcare to supply chain and logistics and more. Our activities, and increasingly, actions from machines are generating new and more useful information in much larger volumes than we've ever seen. Now, meanwhile, our data-hungry society's expectations for experiences are increasingly elevated. Everybody wants to leverage and monetize all this new data coming from smart devices and innumerable sources around the globe. All this data, it surrounds us, but more often than not, it lives in silos, which makes it very difficult to consume, share, and make valuable. These factors, combined with new types of data and analytics, make things even more complicated. Data from ERP systems to images, to data generated from deep learning and machine learning platforms, this is the reality that organizations are facing today. And as such, effectively leveraging all of this data has become an enormous challenge. So, today, we're going to be discussing these modern data challenges and the emergence of so-called "Smart Data Fabrics" as a key solution to said challenges. To do so, we're joined by thought leaders from InterSystems. This is a really creative technology provider that's attacking some of the most challenging data obstacles. InterSystems tells us that they're dedicated to helping customers address their critical scalability, interoperability, and speed-to-value challenges. And in this first segment, we welcome Scott Gnau, he's the global Head of Data Platforms at InterSystems, to discuss the context behind these issues and how smart data fabrics provide a solution. Scott, welcome. Good to see you again. >> Thanks a lot. It's good to be here. >> Yeah. So, look, you and I go back, you know, several years and, you know, you've worked in Tech, you've worked in Data Management your whole career. You've seen many data management solutions, you know, from the early days. And then we went through the hoop, the Hadoop era together and you've come across a number of customer challenges that sort of change along the way. And they've evolved. So, what are some of the most pressing issues that you see today when you're talking to customers and, you know, put on your technical hat if you want to. >> (chuckles) Well, Dave, I think you described it well. It's a perfect storm out there. You know, combined with there's just data everywhere and it's coming up on devices, it's coming from new different kinds of paradigms of processing and people are trying to capture and harness the value from this data. At the same time, you talked about silos and I've talked about data silos through my entire career. And I think, I think the interesting thing about it is for so many years we've talked about, "We've got to reduce the silos and we've got to integrate the data, we've got to consolidate the data." And that was a really good paradigm for a long time. But frankly, the perfect storm that you described? The sources are just too varied. The required agility for a business unit to operate and manage their customers is creating an enormous presser and I think ultimately, silos aren't going away. So, there's a realization that, "Okay, we're going to have these silos, we want to manage them, but how do we really take advantage of data that may live across different parts of our business and in different organizations?" And then of course, the expectation of the consumer is at an all-time high, right? They expect that we're going to treat them and understand their needs or they're going to find some other provider. So, you know, pulling all of this together really means that, you know, our customers and businesses around the world are struggling to keep up and it's forcing a real, a new paradigm shift in underlying data management, right? We started, you know, many, many years ago with data marts and then data warehouses and then we graduated to data lakes, where we expanded beyond just traditional transactional data into all kinds of different data. And at each step along the way, we help businesses to thrive and survive and compete and win. But with the perfect storm that you've described, I think those technologies are now just a piece of the puzzle that is really required for success. And this is really what's leading to data fabrics and data meshes in the industry. >> So what are data fabrics? What problems do they solve? How do they work? Can you just- >> Yeah. So the idea behind it is, and this is not to the exclusion of other technologies that I described in data warehouses and data lakes and so on, but data fabrics kind of take the best of those worlds but add in the notion of being able to do data connectivity with provenance as a way to integrate data versus data consolidation. And when you think about it, you know, data has gravity, right? It's expensive to move data. It's expensive in terms of human cost to do ETL processes where you don't have known provenance of data. So, being able to play data where it lies and connect the information from disparate systems to learn new things about your business is really the ultimate goal. You think about in the world today, we hear about issues with the supply chain and supply and logistics is a big issue, right? Why is that an issue? Because all of these companies are data-driven. They've got lots of access to data. They have formalized and automated their processes, they've installed software, and all of that software is in different systems within different companies. But being able to connect that information together, without changing the underlying system, is an important way to learn and optimize for supply and logistics, as an example. And that's a key use case for data fabrics. Being able to connect, have provenance, not interfere with the operational system, but glean additional knowledge by combining multiple different operational systems' data together. >> And to your point, data is by its very nature, you know, distributed around the globe, it's on different clouds, it's in different systems. You mentioned "data mesh" before. How do data fabrics relate to this concept of data mesh? Are they competing? Are they complimentary? >> Ultimately, we think that they're complimentary. And we actually like to talk about smart data fabrics as a way to kind of combine the best of the two worlds. >> What is that? >> The biggest thing really is there's a lot around data fabric architecture that talks about centralized processing. And in data meshes, it's more about distributed processing. Ultimately, we think a smart data fabric will support both and have them be interchangeable and be able to be used where it makes the most sense. There are some things where it makes sense to process, you know, for a local business unit, or even on a device for real-time kinds of implementations. There are some other areas where centralized processing of multiple different data sources make sense. And what we're saying is, "Your technology and the architecture that you define behind that technology should allow for both where they make the most sense." >> What's the bottom line business benefit of implementing a data fabric? What can I expect if I go that route? >> I think there are a couple of things, right? Certainly, being able to interact with customers in real time and being able to manage through changes in the marketplace is certainly a key concept. Time-to-value is another key concept. You know, if you think about the supply and logistics discussion that I had before, right? No company is going to rewrite their ERP operational system. It's how they manage and run their business. But being able to glean additional insights from that data combined with data from a partner combined with data from a customer or combined with algorithmic data that, you know, you may create some sort of forecast and that you want to fit into. And being able to combine that together without interfering with the operational process and get those answers quickly is an important thing. So, seeing through the silos and being able to do the connectivity, being able to have interoperability, and then, combining that with flexibility on the analytics and flexibility on the algorithms you might want to run against that data. Because in today's world, of course, you know, certainly there's the notion of predictive modeling and relational theory, but also now adding in machine learning, deep learning algorithms, and have all of those things kind of be interchangeable is another important concept behind data fabric. So you're not relegated to one type of processing. You're saying, "It's data and I have multiple different processing engines and I may want to interchange them over time." >> So, I know, well actually, you know, when you said "real time", I infer from that, I don't have a zillion copies of the data and it's not in a bunch of silos. Is that a correct premise? >> You try to minimize your copies of the data? >> Yeah. Okay. >> There's certainly, there's a nirvana that says, "There's only ever one copy of data." That's probably impossible. But you certainly don't want to be forced into making multiple copies of data to support different processing engines unnecessarily. >> And so, you've recently made some enhancements to the data fabric capability that takes it, you know, ostensibly to the next level. Is that the smart piece? Is that machine intelligence? Can you describe what's in there? >> Well, you know, ultimately, the business benefit is be able to have a single source of the truth for a company. And so, what we're doing is combining multiple technologies in a single set of software that makes that software agile and supportable and not fragile for deployment of applications. At its core, what we're saying is, you know, we want to be able to consume any kind of data and I think your data fabric architecture is predicated on the fact that you're going to have relational data, you're going to have document data, you may have key-value store data, you may have images, you may have other things, and you want to be able to not be limited by the kind of data that you want to process. And so that certainly is what we build into our product set. And then, you want to be able to have any kind of algorithm, where appropriate, run against that data without having to do a bunch of massive ETL processes or make another copy of the data and move it somewhere else. And so, to that end, we have, taking our award-winning engine, which, you know, provides, you know, traditional analytic capabilities and relational capabilities, we've now integrated machine learning. So, you basically can bring machine learning algorithms to the data without having to move data to the machine learning algorithm. What does that mean? Well, number one, your application developer doesn't have to think differently to take advantage of the new algorithm. So that's a really good thing. The other thing that happens is if you, you're playing that algorithm where the data actually exists from your operational system, that means the round trip from running the model to inferring some decision you want to make to actually implementing that decision can happen instantaneously, as opposed to, you know, other kinds of architectures, where you may want to make a copy of the data and move it somewhere else. That takes time, latency. Now the data gets stale, your model may not be as efficient because you're running against stale data. We've now taken all of that off the table by being able to pull that processing inside the data fabric, inside of the single source of truth. >> And you got to manage all that complexity. So you got one system, so that makes it, you know, cost-effective, and you're bringing modern tooling to the platform. Is that right? >> That's correct. >> How can people learn more and maybe continue the conversation with you if they have other questions? (both chuckle) >> Call or write. >> Yeah. >> Yeah, I mean, certainly, check out our website. We've got a lot of information about the different kinds of solutions, the different industries, the different technologies. Reach out: scottg@intersystems.com. >> Excellent. Thank you, Scott. Really appreciate it and great to see you again. >> Good to see you. >> All right, keep it right there. We have a demo coming up next. You want to see smart data fabrics in action? Stay tuned. (ambient music)

(upbeat music) >> Today more than ever before, organizations are striving to gain a competitive advantage, deliver more value to customers, reduce risk, and respond more quickly to the needs of businesses. Now, to achieve these goals, organizations need easy access to a single view of accurate, consistent and very importantly, trusted data. If it's not trusted, nobody's going to use it and all in near real time. However, the growing volumes and complexities of data make this difficult to achieve in practice. Not to mention the organizational challenges that have evolved as data becomes increasingly important to winning in the marketplace. Specifically as data grows, so does the prevalence of data silos, making, integrating and leveraging data from internal and external sources a real challenge. Now, in this final segment, we'll hear from Joe Lichtenberg who's the global head of product and industry marketing, and he's going to discuss how smart data fabrics can be applied to different industries. And by way of these use cases, we'll probe Joe's vast knowledge base and ask him to highlight how InterSystems, which touts a next gen approach to Customer 360, how the company leverages a smart data fabric to provide organizations of varying sizes and sectors in financial services, supply chain, logistics and healthcare with a better, faster and easier way to deliver value to the business. Joe welcome, great to have you here. >> Thank you, it's great to be here. That was some intro. I could not have said it better myself, so thank you for that. >> Thank you. Well, we're happy to have you on this show now. I understand- >> It's great to be here. >> You you've made a career helping large businesses with technology solutions, small businesses, and then scale those solutions to meet whatever needs they had. And of course, you're a vocal advocate as is your company of data fabrics. We talked to Scott earlier about data fabrics, how it relates to data mesh big discussions in the industry. So tell us more about your perspective. >> Sure, so first I would say that I have been in this industry for a very long time so I've been like you, I'm sure, for decades working with customers and with technology, really to solve these same kinds of challenges. So for decades, companies have been working with lots and lots of data and trying to get business value to solve all sorts of different challenges. And I will tell you that I've seen many different approaches and different technologies over the years. So, early on, point to point connections with custom coding, and I've worked with integration platforms 20 years ago with the advent of web services and service-oriented architectures and exposing endpoints with wisdom and getting access to disparate data from across the organization. And more recently, obviously with data warehouses and data lakes and now moving workloads to the cloud with cloud-based data marts and data warehouses. Lots of approaches that I've seen over the years but yet still challenges remain in terms of getting access to a single trusted real-time view of data. And so, recently, we ran a survey of more than 500 different business users across different industries and 86% told us that they still lack confidence in using their data to make decisions. That's a huge number, right? And if you think about all of the work and all of the technology and approaches over the years, that is a surprising number and drilling into why that is, there were three main reasons. One is latency. So the amount of time that it takes to access the data and process the data and make it fit for purpose by the time the business has access to the data and the information that they need, the opportunity has passed. >> Elapsed time, not speed a light, right? But that too maybe. >> But it takes a long time if you think about these processes and you have to take the data and copy it and run ETL processes and prepare it. So that's one, one is just the amount of data that's disparate in data silos. So still struggling with data that is dispersed across different systems in different formats. And the third, is data democratization. So the business really wants to have access to the data so that they can drill into the data and ask ad hoc questions and the next question and drill into the information and see where it leads them rather than having sort of pre-structured data and pre-structured queries and having to go back to IT and put the request back on the queue again and waiting. >> So it takes too long, the data's too hard to get to 'cause it's in silos and the data lacks context because it's technical people that are serving up the data to the business people. >> Exactly. >> And there's a mismatch. >> Exactly right. So they call that data democratization or giving the business access to the data and the tools that they need to get the answers that they need in the moment. >> So the skeptic in me, 'cause you're right I have seen this story before and the problems seem like they keep coming up, year after year, decade after decade. But I'm an optimist and so. >> As am I. >> And so I sometimes say, okay, same wine new bottle, but it feels like it's different this time around with data fabrics. You guys talk about smart data fabrics from your perspective, what's different? >> Yeah, it's very exciting and it's a fundamentally different approach. So if you think about all of these prior approaches, and by the way, all of these prior approaches have added value, right? It's not like they were bad, but there's still limitations and the business still isn't getting access to all the data that they need in the moment, right? So data warehouses are terrific if you know the questions that you want answered and you take the data and you structure the data in advance. And so now you're serving the business with sort of pre-planned answers to pre-planned queries, right? The data fabric, what we call a smart data fabric is fundamentally different. It's a fundamentally different approach in that rather than sort of in batch mode, taking the data and making it fit for purpose with all the complexity and delays associated with it, with a data fabric where accessing the data on demand as it's needed, as it's requested, either by the business or by applications or by the data scientists directly from the source systems. >> So you're not copying it necessarily to that to make that you're not FTPing it, for instance. I've got it, you take it, you're basically using the same source. >> You're pulling the data on demand as it's being requested by the consumers. And then all of the data management processes that need to be applied for integration and transformation to get the data into a consistent format and business rules and analytic queries. And with Jess showed with machine learning, predictive prescriptive analytics all sorts of powerful capabilities are built into the fabric so that as you're pulling the data on demand, right, all of these processes are being applied and the net result is you're addressing these limitations around latency and silos that we've seen in the past. >> Okay, so you've talked about you have a lot of customers, InterSystems does in different industries supply chain, financial services, manufacturing. We heard from just healthcare. What are you seeing in terms of applications of smart data fabrics in the real world? >> Yeah, so we see it in every industry. So InterSystems, as you know, has been around now for 43 years, and we have tens of thousands of customers in every industry. And this architectural pattern now is providing value for really critical use cases in every industry. So I'm happy to talk to you about some that we're seeing. I could actually spend like three hours here and there but I'm very passionate about working with customers and there's all sorts of exciting. >> What are some of your favorites? >> So, obviously supply chain right now is going through a very challenging time. So the combination of what's happening with the pandemic and disruptions and now I understand eggs are difficult to come by I just heard on NPR. >> Yeah and it's in part a data problem and a big part of data problem, is that fair? >> Yeah and so, in supply chain, first there's supply chain visibility. So organizations want a real time or near real time expansive view of what's happening across the entire supply chain from a supply all the way through distribution, right? So that's only part of the issue but that's a huge sort of real-time data silos problem. So if you think about your extended supply chain, it's complicated enough with all the systems and silos inside your firewall, before all of your suppliers even just thinking about your tier one suppliers let alone tier two and tier three. And then building on top of real-time visibility is what the industry calls a control tower, what we call the ultimate control tower. And so it's built in analytics to be able to sense disruptions and exceptions as they occur and predict the likelihood of these disruptions occurring. And then having data driven and analytics driven guidance in terms of the best way to deal with these disruptions. So for example, an order is missing line items or a cargo ship is stuck off port somewhere. What do you do about it? Do you reroute a different cargo ship, right? Do you take an order that's en route to a different client and reroute that? What's the cost associated? What's the impact associated with it? So that's a huge issue right now around control towers for supply chain. So that's one. >> Can I ask you a question about that? Because you and I have both seen a lot but we've never seen, at least I haven't the economy completely shut down like it was in March of 2020, and now we're seeing this sort of slingshot effect almost like you're driving on the highway sometimes you don't know why, but all of a sudden you slow down and then you speed up, you think it's okay then you slow down again. Do you feel like you guys can help get a handle on that product because it goes on both sides. Sometimes you can't get the product, sometimes there's too much of a product as well and that's not good for business. >> Yeah, absolutely. You want to smooth out the peaks and valleys. >> Yeah. >> And that's a big business goal, business challenge for supply chain executives, right? So you want to make sure that you can respond to demand but you don't want to overstock because there's cost associated with that as well. So how do you optimize the supply chains and it's very much a data silo and a real time challenge. So it's a perfect fit for this new architectural pattern. >> All right, what else? >> So if we look at financial services, we have many, many customers in financial services and that's another industry where they have many different sources of data that all have information that organizations can use to really move the needle if they could just get to that single source of truth in real time. So we sort of bucket many different implementations and use cases that we do around what we call Business 360 and Customer 360. So Business 360, there's all sorts of ways to add business value in terms of having a real-time operational view across all of the different GOs and parts of the business, especially in these very large global financial services institutions like capital markets and investment firms and so forth. So around Business 360, having a realtime view of risk, operational performance regulatory compliance, things like that. Customer 360, there's a whole set of use cases around Customer 360 around hyper-personalization of customers and in realtime next best action looking to see how you can sell more increase share of wallet, cross-sell, upsell to customers. We also do a lot in terms of predicting customer churn. So if you have all the historical data and what's the likelihood of customers churning to be able to proactively intercede, right? It's much more cost effective to keep assets under management and keep clients rather than going and getting new clients to come to the firm. A very interesting use case from one of our customers in Latin America, so Banco do Brasil largest bank in all of Latin America and they have a very innovative CTO who's always looking for new ways to move the needle for the bank. And so one of their ideas and we're working with them to do this is how can they generate net new revenue streams by bringing in new business to the bank? And so they identified a large percentage of the population in Latin America that does no banking. So they have no banking history not only with Banco do Brasil, but with any bank. So there's a fair amount of risk associated with offering services to this segment of the population that's not associated with any banks or financial institutions. >> There is no historical data on them, there's no. >> So it's a data challenge. And so, they're bringing in data from a variety of different sources, social media, open source data that they find online and so forth. And with us running risk models to identify which are the citizens that there's acceptable risk to offer their services. >> It's going to be huge market of unbanked people in vision Latin America. >> Wow, that's interesting. >> Yeah, yeah, totally vision. >> And if you can lower the risk and you could tap that market and be first >> And they are, yeah. >> Yeah. >> So very exciting. Manufacturing, we know industry 4.0 which is about taking the OT data, so the data from the MES systems and the streaming data, real-time streaming data from the machine controllers and integrating it with the IT data, so your data warehouses and your ERP systems and so forth to have not only a real-time view of manufacturing from supply and source all the way through demand but also predictive maintenance and things like that. So that's very big right now in manufacturing. >> Kind of cool to hear these use cases beyond your healthcare, which is obviously, your wheelhouse, Scott defined this term of smart data fabrics, different than data fabrics, I guess. So when we think about these use cases what's the value add of so-called smart data fabrics? >> Yeah, it's a great question. So we did not define the term data fabric or enterprise data fabric. The analysts now are all over it. They're all saying it's the future of data management. It's a fundamentally different approach this architectural approach to be able to access the data on demand. The canonical definition of a data fabric is to access the data where it lies and apply a set of data management processes, but it does not include analytics, interestingly. And so we firmly believe that most of these use cases gain value from having analytics built directly into the fabric. So whether that's business rules or predictive analytics to predict the likelihood of a customer churn or a machine on the shop floor failing or prescriptive analytics. So if there's a problem in the supply chain, what's the guidance for the supply chain managers to take the best action, right? Prescriptive analytics based on data. So rather than taking the data and the data fabric and moving it to another environment to run those analytics where you have complexity and latency, having tall of those analytics capabilities built directly into the fabric, which is why we call it a smart data fabric, brings a lot of value to our customers. >> So simplifies the whole data lifecycle, data pipelining, the hyper-specialized roles that you have to have, you can really just focus on one platform, is that? >> Exactly, basically, yeah. And it's a simplicity of architecture and faster speed to production. So a big differentiator for our technology, for InterSystems, Iris, is most if not all of the capabilities that are needed are built into one engine, right? So you don't need to stitch together 10 or 15 or 20 different data management services for relational database in a non-relational database and a caching layer and a data warehouse and security and so forth. And so you can do that. There's many ways to build this data fabric architecture, right? InterSystems is not the only way. >> Right? >> But if you can speed and simplify the implementation of the fabric by having most of what you need in one engine, one product that gets you to where you need to go much, much faster. >> Joe, how can people learn more about smart data Fabric some of the use cases that you've presented here? >> Yeah, come to our website, intersystems.com. If you go to intersystems.com/smartdatafabric that'll take you there. >> I know that you have like probably dozens more examples but it would be cool- >> I do. >> If people reach out to you, how can they get in touch? >> Oh, I would love that. So feel free to reach out to me on LinkedIn. It's Joe Lichtenberg I think it's linkedin.com/joeLichtenberg and I'd love to connect. >> Awesome. Joe, thanks so much for your time. Really appreciate it. >> It was great to be here. Thank you, Dave. >> All right, I hope you've enjoyed our program today. You know, we heard Scott now he helped us understand this notion of data fabrics and smart data fabrics and how they can address the data challenges faced by the vast majority of organizations today. Jess Jody's demo was awesome. It was really a highlight of the program where she showed the smart data fabrics inaction and Joe Lichtenberg, we just heard from him dug in to some of the prominent use cases and proof points. We hope this content was educational and inspires you to action. Now, don't forget all these videos are available on Demand to watch, rewatch and share. Go to theCUBE.net, check out siliconangle.com for all the news and analysis and we'll summarize the highlights of this program and go to intersystems.com because there are a ton of resources there. In particular, there's a knowledge hub where you'll find some excellent educational content and online learning courses. There's a resource library with analyst reports, technical documentation videos, some great freebies. So check it out. This is Dave Vellante. On behalf of theCUBE and our supporter, InterSystems, thanks for watching and we'll see you next time. (upbeat music)

(upbeat music) >> Now, as we all know, businesses are awash with data, from financial services to healthcare to supply chain and logistics and more. Our activities, and increasingly, actions from machines are generating new and more useful information in much larger volumes than we've ever seen. Now, meanwhile, our data hungry society's expectations for experiences are increasingly elevated. Everybody wants to leverage and monetize all this new data coming from smart devices and innumerable sources around the globe. All this data, it surrounds us, but more often than not, it lives in silos, which makes it very difficult to consume, share, and make valuable. These factors combined with new types of data and analytics make things even more complicated. Data from ERP systems to images, to data generated from deep learning and machine learning platforms, this is the reality that organizations are facing today. And as such, effectively leveraging all of this data has become an enormous challenge. So today, we're going to be discussing these modern data challenges in the emergence of so-called smart data fabrics as a key solution to said challenges. To do so, we're joined by thought leaders from InterSystems. This is a really creative technology provider that's attacking some of the most challenging data obstacles. InterSystems tells us that they're dedicated to helping customers address their critical scalability, interoperability, and speed to value challenges. And in this first segment, we welcome Scott now. He's the global head of data platforms at InterSystems to discuss the context behind these issues and how smart data fabrics provide a solution. Scott, welcome, good to see you again. >> Thanks a lot. It's good to be here. >> Yeah, so look, you and I go back, you know, several years and you've worked in tech. You've worked in data management your whole career. You've seen many data management solutions, you know, from the early days. And then we went through the Hadoop era together. And you've come across a number of customer challenges that sort of changed along the way, and they've evolved. So what are some of the most pressing issues that you see today when you're talking to customers, and, you know, put on your technical hat if you want to? >> Well, Dave, I think you described it well. It's a perfect storm out there, you know, combined with, there's just data everywhere. And it's coming up on devices, it's coming from new different kinds of paradigms of processing and people are trying to capture and harness the value from this data. At the same time, you talked about silos, and I've talked about data silos through my entire career. And I think the interesting thing about it is for so many years we've talked about we've got to reduce the silos, and we've got to integrate the data, we've got to consolidate the data. And that was a really good paradigm for a long time. But frankly, the perfect storm that you described, the sources are just too varied. The required agility for a business unit to operate and manage their customers is creating an enormous pressure. And I think, ultimately, silos aren't going away. So there's a realization that, okay, we're going to have these silos, we want to manage them, but how do we really take advantage of data that may live across different parts of our business and in different organizations? And then, of course, the expectation of the consumer is at an all-time high, right? They expect that we're going to treat them and understand their needs, or they're going to find some other provider. So, you know, pulling all of this together really means that, you know, our customers and businesses around the world are struggling to keep up, and it's forcing a new paradigm shift in underlying data management, right? We started, you know, many, many years ago with data marts and then data warehouses, and then we graduated to data lakes where we expanded beyond just traditional transactional data into all kinds of different data. And at each step along the way, we help businesses to thrive and survive and compete and win. But with the perfect storm that you've described, I think those technologies are now just a piece of the puzzle that is really required for success. And this is really what's leading to data fabrics and data meshes in the industry. >> So what are data fabrics? What problems do they solve? How do they work? Can you just add- >> Yeah, so the idea behind it is, and this is not to the exclusion of other technologies that I described in data warehouses and data lakes and so on. But data fabrics kind of take the best of those worlds, but add in the notion of being able to do data connectivity with provenance as a way to integrate data versus data consolidation. And when you think about it, you know, data has gravity, right? It's expensive to move data. It's expensive in terms of human cost to do ETL processes where you don't have known provenance of data. So being able to play data where it lies and connect the information from disparate systems to learn new things about your business is really the ultimate goal. You think about in the world today, we hear about issues with the supply chain, and supply and logistics is a big issue, right? Why is that an issue? Because all of these companies are data driven. They've got lots of access to data. They have formalized and automated their processes. They've installed software. And all of that software is in different systems within different companies. But being able to connect that information together without changing the underlying system is an important way to learn and optimize for supply and logistics, as an example. And that's a key use case for data fabrics being able to connect, have provenance, not interfere with the operational system, but glean additional knowledge by combining multiple different operational systems' data together. >> And to your point, data is by its very nature, you're distributed around the globe, it's on different clouds, it's in different systems. You mentioned data mesh before. How do data fabrics relate to this concept of data mesh? Are they competing? Are they complimentary? >> Ultimately, we think that they're complimentary. And we actually like to talk about smart data fabrics as a way to kind of combine the best of the two worlds. >> What is that? I mean, the biggest thing really is there's a lot around data fabric architecture that talks about centralized processing. And in data meshes, it's more about distributed processing. Ultimately, we think a smart data fabric will support both and have them be interchangeable and be able to be used where it makes the most sense. There are some things where it makes sense to process, you know, for a local business unit, or even on a device for real time kinds of implementations. There are some other areas where centralized processing of multiple different data sources make sense. And what we're saying is your technology and the architecture that you define behind that technology should allow for both where they make the most sense. >> What's the bottom line business benefit of implementing a data fabric? What can I expect if I go that route? >> I think there are a couple of things, right? Certainly being able to interact with customers in real time and being able to manage through changes in the marketplace is certainly a key concept. Time to value is another key concept. You know, if you think about the supply and logistics discussion that I had before, right? No company is going to rewrite their ERP operational system. It's how they manage and run their business. But being able to glean additional insights from that data combined with data from a partner, combined with data from a customer, or combined with algorithmic data that, you know, you may create some sort of forecast and that you want to fit into. And being able to combine that together without interfering with the operational process and get those answers quickly is an important thing. So seeing through the silos and being able to do the connectivity being able to have interoperability, and then combining that with flexibility on the analytics and flexibility on the algorithms you might want to run against that data. Because in today's world, of course, certainly there's the notion of predictive modeling and relational theory, but also now adding in machine learning, deep learning algorithms, and have all of those things kind of be interchangeable is another important concept behind data fabrics. So you're not relegated to one type of processing. You're saying it's data, and I have multiple different processing engines and I may want to interchange them over time. >> So, I know, well actually, when you said real time, I infer from that I don't have a zillion copies of the data and it's not in a bunch of silos. Is that a correct premise? >> You try to minimize your copies of the data. There's a nirvana that says there's only ever one copy of data. That's probably impossible. But you certainly don't want to be forced into making multiple copies of data to support different processing engines unnecessarily. >> And so you've recently made some enhancements to the data fabric capability that takes it, you know, ostensibly to the next level. Is that the smart piece, is that machine intelligence? Can you describe what's in there? >> Well, you know, ultimately the business benefit is be able to have a single source of the truth for a company. And so what we're doing is combining multiple technologies in a single set of software that makes that software agile and supportable and not fragile for deployment of applications. At its core, what we're saying is, we want to be able to consume any kind of data, and I think your data fabric architecture is predicated on the fact that you're going to have relational data you're going to have document data, you may have key value store data, you may have images, you may have other things, and you want to be able to not be limited by the kind of data that you want to process. And so that certainly is what we build into our product set. And then you want to be able to have any kind of algorithm where appropriate run against that data without having to do a bunch of massive ETL processes or make another copy of the data and move it somewhere else. And so to that end, we have taken our award-winning engine, which, you know, provides traditional analytic capabilities and relational capabilities. We've now integrated machine learning. So you basically can bring machine learning algorithms to the data without having to move data to the machine learning algorithm. What does that mean? Well, number one, your application developer doesn't have to think differently to take advantage of the new algorithms. So that's a really good thing. The other thing that happens is if you're playing that algorithm where the data actually exists from your operational system, that means the roundtrip from running the model to inferring some decision you want to make to actually implementing that decision can happen instantaneously. As opposed to, you know, other kinds of architectures where you may want to make a copy of the data and move it somewhere else. That takes time, latency. Now the data gets stale. Your model may not be as efficient because you're running against stale data. We've now taken all of that off the table by being able to pull that processing inside the data fabric, inside of the single source of truth. >> And you got to manage all that complexity. So you got one system, so that makes it cost effective, and you're bringing modern tooling to the platform. Is that right? >> That's correct. How can people learn more and maybe continue the conversation with you if they have other questions? >> (Scott laughs) Call or write. Yeah, I mean, certainly check out our website. We've got a lot of information about the different kinds of solutions, the different industries, the different technologies. Reach out at scottg@intersystems.com. >> Excellent, thank you, Scott. Really appreciate it. And great to see you again. >> Good to see you. All right, keep it right there. We have a demo coming up next. If you want to see smart data fabrics in action, stay tuned. (upbeat music)

>> Okay, so now that we've heard Scott talk about smart data fabrics, it's time to see this in action. Right now we're joined by Jess Jowdy, who's the manager of Healthcare Field Engineering at InterSystems. She's going to give a demo of how smart data fabrics actually work, and she's going to show how embedding a wide range of analytics capabilities including data exploration, business intelligence natural language processing, and machine learning directly within the fabric, makes it faster and easier for organizations to gain new insights and power intelligence, predictive and prescriptive services and applications. Now, according to InterSystems, smart data fabrics are applicable across many industries from financial services to supply chain to healthcare and more. Jess today is going to be speaking through the lens of a healthcare focused demo. Don't worry, Joe Lichtenberg will get into some of the other use cases that you're probably interested in hearing about. That will be in our third segment, but for now let's turn it over to Jess. Jess, good to see you. >> Hi. Yeah, thank you so much for having me. And so for this demo we're really going to be bucketing these features of a smart data fabric into four different segments. We're going to be dealing with connections, collections, refinements and analysis. And so we'll see that throughout the demo as we go. So without further ado, let's just go ahead and jump into this demo and you'll see my screen pop up here. I actually like to start at the end of the demo. So I like to begin by illustrating what an end user's going to see and don't mind the screen 'cause I gave you a little sneak peek of what's about to happen. But essentially what I'm going to be doing is using Postman to simulate a call from an external application. So we talked about being in the healthcare industry. This could be for instance, a mobile application that a patient is using to view an aggregated summary of information across that patient's continuity of care or some other kind of application. So we might be pulling information in this case from an electronic medical record. We might be grabbing clinical history from that. We might be grabbing clinical notes from a medical transcription software or adverse reaction warnings from a clinical risk grouping application and so much more. So I'm really going to be assimilating a patient logging on in on their phone and retrieving this information through this Postman call. So what I'm going to do is I'm just going to hit send, I've already preloaded everything here and I'm going to be looking for information where the last name of this patient is Simmons and their medical record number their patient identifier in the system is 32345. And so as you can see I have this single JSON payload that showed up here of just relevant clinical information for my patient whose last name is Simmons all within a single response. So fantastic, right? Typically though when we see responses that look like this there is an assumption that this service is interacting with a single backend system and that single backend system is in charge of packaging that information up and returning it back to this caller. But in a smart data fabric architecture we're able to expand the scope to handle information across different, in this case, clinical applications. So how did this actually happen? Let's peel back another layer and really take a look at what happened in the background. What you're looking at here is our mission control center for our smart data fabric. On the left we have our APIs that allow users to interact with particular services. On the right we have our connections to our different data silos. And in the middle here we have our data fabric coordinator which is going to be in charge of this refinement and analysis those key pieces of our smart data fabric. So let's look back and think about the example we just showed. I received an inbound request for information for a patient whose last name is Simmons. My end user is requesting to connect to that service and that's happening here at my patient data retrieval API location. Users can define any number of different services and APIs depending on their use cases. And to that end we do also support full lifecycle API management within this platform. When you're dealing with APIs I always like to make a little shout out on this that you really want to make sure you have enough like a granular enough security model to handle and limit which APIs and which services a consumer can interact with. In this IRIS platform, which we're talking about today we have a very granular role-based security model that allows you to handle that, but it's really important in a smart data fabric to consider who's accessing your data and in what contact. >> Can I just interrupt you for a second? >> Yeah, please. >> So you were showing on the left hand side of the demo a couple of APIs. I presume that can be a very long list. I mean, what do you see as typical? >> I mean you can have hundreds of these APIs depending on what services an organization is serving up for their consumers. So yeah, we've seen hundreds of these services listed here. >> So my question is, obviously security is critical in the healthcare industry and API securities are really hot topic these days. How do you deal with that? >> Yeah, and I think API security is interesting 'cause it can happen at so many layers. So there's interactions with the API itself. So can I even see this API and leverage it? And then within an API call, you then have to deal with all right, which end points or what kind of interactions within that API am I allowed to do? What data am I getting back? And with healthcare data, the whole idea of consent to see certain pieces of data is critical. So the way that we handle that is, like I said, same thing at different layers. There is access to a particular API, which can happen within the IRIS product and also we see it happening with an API management layer, which has become a really hot topic with a lot of organizations. And then when it comes to data security, that really happens under the hood within your smart data fabric. So that role-based access control becomes very important in assigning, you know, roles and permissions to certain pieces of information. Getting that granular becomes the cornerstone of security. >> And that's been designed in, >> Absolutely, yes. it's not a bolt-on as they like to say. Okay, can we get into collect now? >> Of course, we're going to move on to the collection piece at this point in time, which involves pulling information from each of my different data silos to create an overall aggregated record. So commonly each data source requires a different method for establishing connections and collecting this information. So for instance, interactions with an EMR may require leveraging a standard healthcare messaging format like FIRE, interactions with a homegrown enterprise data warehouse for instance may use SQL for a cloud-based solutions managed by a vendor. They may only allow you to use web service calls to pull data. So it's really important that your data fabric platform that you're using has the flexibility to connect to all of these different systems and and applications. And I'm about to log out so I'm going to keep my session going here. So therefore it's incredibly important that your data fabric has the flexibility to connect to all these different kinds of applications and data sources and all these different kinds of formats and over all of these different kinds of protocols. So let's think back on our example here. I had four different applications that I was requesting information for to create that payload that we saw initially. Those are listed here under this operations section. So these are going out and connecting to downstream systems to pull information into my smart data fabric. What's great about the IRIS platform is it has an embedded interoperability platform. So there's all of these native adapters that can support these common connections that we see for different kinds of applications. So using REST or SOAP or SQL or FTP regardless of that protocol there's an adapter to help you work with that. And we also think of the types of formats that we typically see data coming in as, in healthcare we have H7, we have FIRE we have CCDs across the industry. JSON is, you know, really hitting a market strong now and XML, payloads, flat files. We need to be able to handle all of these different kinds of formats over these different kinds of protocols. So to illustrate that, if I click through these when I select a particular connection on the right side panel I'm going to see the different settings that are associated with that particular connection that allows me to collect information back into my smart data fabric. In this scenario, my connection to my chart script application in this example communicates over a SOAP connection. When I'm grabbing information from my clinical risk grouping application I'm using a SQL based connection. When I'm connecting to my EMR I'm leveraging a standard healthcare messaging format known as FIRE, which is a rest based protocol. And then when I'm working with my health record management system I'm leveraging a standard HTTP adapter. So you can see how we can be flexible when dealing with these different kinds of applications and systems. And then it becomes important to be able to validate that you've established those connections correctly and be able to do it in a reliable and quick way. Because if you think about it, you could have hundreds of these different kinds of applications built out and you want to make sure that you're maintaining and understanding those connections. So I can actually go ahead and test one of these applications and put in, for instance my patient's last name and their MRN and make sure that I'm actually getting data back from that system. So it's a nice little sanity check as we're building out that data fabric to ensure that we're able to establish these connections appropriately. So turnkey adapters are fantastic, as you can see we're leveraging them all here, but sometimes these connections are going to require going one step further and building something really specific for an application. So let's, why don't we go one step further here and talk about doing something custom or doing something innovative. And so it's important for users to have the ability to develop and go beyond what's an out of the box or black box approach to be able to develop things that are specific to their data fabric or specific to their particular connection. In this scenario, the IRIS data platform gives users access to the entire underlying code base. So you cannot, you not only get an opportunity to view how we're establishing these connections or how we're building out these processes but you have the opportunity to inject your own kind of processing your own kinds of pipelines into this. So as an example, you can leverage any number of different programming languages right within this pipeline. And so I went ahead and I injected Python. So Python is a very up and coming language, right? We see more and more developers turning towards Python to do their development. So it's important that your data fabric supports those kinds of developers and users that have standardized on these kinds of programming languages. This particular script here, as you can see actually calls out to our turnkey adapters. So we see a combination of out of the box code that is provided in this data fabric platform from IRIS combined with organization specific or user specific customizations that are included in this Python method. So it's a nice little combination of how do we bring the developer experience in and mix it with out of the box capabilities that we can provide in a smart data fabric. >> Wow. >> Yeah, I'll pause. >> It's a lot here. You know, actually, if I could >> I can pause. >> If I just want to sort of play that back. So we went through the connect and the collect phase. >> And the collect, yes, we're going into refine. So it's a good place to stop. >> Yeah, so before we get there, so we heard a lot about fine grain security, which is crucial. We heard a lot about different data types, multiple formats. You've got, you know the ability to bring in different dev tools. We heard about FIRE, which of course big in healthcare. >> Absolutely. >> And that's the standard and then SQL for traditional kind of structured data and then web services like HTTP you mentioned. And so you have a rich collection of capabilities within this single platform. >> Absolutely, and I think that's really important when you're dealing with a smart data fabric because what you're effectively doing is you're consolidating all of your processing, all of your collection into a single platform. So that platform needs to be able to handle any number of different kinds of scenarios and technical challenges. So you've got to pack that platform with as many of these features as you can to consolidate that processing. >> All right, so now we're going into refine. >> We're going into refinement, exciting. So how do we actually do refinement? Where does refinement happen and how does this whole thing end up being performant? Well the key to all of that is this SDF coordinator or stands for smart data fabric coordinator. And what this particular process is doing is essentially orchestrating all of these calls to all of these different downstream systems. It's aggregating, it's collecting that information it's aggregating it and it's refining it into that single payload that we saw get returned to the user. So really this coordinator is the main event when it comes to our data fabric. And in the IRIS platform we actually allow users to build these coordinators using web-based tool sets to make it intuitive. So we can take a sneak peek at what that looks like and as you can see it follows a flow chart like structure. So there's a start, there is an end and then there are these different arrows that point to different activities throughout the business process. And so there's all these different actions that are being taken within our coordinator. You can see an action for each of the calls to each of our different data sources to go retrieve information. And then we also have the sync call at the end that is in charge of essentially making sure that all of those responses come back before we package them together and send them out. So this becomes really crucial when we're creating that data fabric. And you know, this is a very simple data fabric example where we're just grabbing data and we're consolidating it together. But you can have really complex orchestrators and coordinators that do any number of different things. So for instance, I could inject SQL Logic into this or SQL code, I can have conditional logic, I can do looping, I can do error trapping and handling. So we're talking about a whole number of different features that can be included in this coordinator. So like I said, we have a really very simple process here that's just calling out, grabbing all those different data elements from all those different data sources and consolidating it. We'll look back at this coordinator in a second when we introduce or we make this data fabric a bit smarter and we start introducing that analytics piece to it. So this is in charge of the refinement. And so at this point in time we've looked at connections, collections, and refinements. And just to summarize what we've seen 'cause I always like to go back and take a look at everything that we've seen. We have our initial API connection we have our connections to our individual data sources and we have our coordinators there in the middle that are in charge of collecting the data and refining it into a single payload. As you can imagine, there's a lot going on behind the scenes of a smart data fabric, right? There's all these different processes that are interacting. So it's really important that your smart data fabric platform has really good traceability, really good logging 'cause you need to be able to know, you know, if there was an issue, where did that issue happen, in which connected process and how did it affect the other processes that are related to it. In IRIS, we have this concept called a visual trace. And what our clients use this for is basically to be able to step through the entire history of a request from when it initially came into the smart data fabric to when data was sent back out from that smart data fabric. So I didn't record the time but I bet if you recorded the time it was this time that we sent that request in. And you can see my patient's name and their medical record number here and you can see that that instigated four different calls to four different systems and they're represented by these arrows going out. So we sent something to chart script to our health record management system, to our clinical risk grouping application into my EMR through their FIRE server. So every request, every outbound application gets a request and we pull back all of those individual pieces of information from all of those different systems and we bundle them together. And for my FIRE lovers, here's our FIRE bundle that we got back from our FIRE server. So this is a really good way of being able to validate that I am appropriately grabbing the data from all these different applications and then ultimately consolidating it into one payload. Now we change this into a JSON format before we deliver it, but this is those data elements brought together. And this screen would also be used for being able to see things like error trapping or errors that were thrown alerts, warnings, developers might put log statements in just to validate that certain pieces of code are executing. So this really becomes the one stop shop for understanding what's happening behind the scenes with your data fabric. >> Etcher, who did what, when, where what did the machine do? What went wrong and where did that go wrong? >> Exactly. >> Right in your fingertips. >> Right, and I'm a visual person so a bunch of log files to me is not the most helpful. Well, being able to see this happened at this time in this location gives me that understanding I need to actually troubleshoot a problem. >> This business orchestration piece, can you say a little bit more about that? How people are using it? What's the business impact of the business orchestration? >> The business orchestration, especially in the smart data fabric is really that crucial part of being able to create a smart data fabric. So think of your business orchestrator as doing the heavy lifting of any kind of processing that involves data, right? It's bringing data in, it's analyzing that information, it's transforming that data, in a format that your consumer's not going to understand it's doing any additional injection of custom logic. So really your coordinator or that orchestrator that sits in the middle is the brains behind your smart data fabric. >> And this is available today? This all works? >> It's all available today. Yeah, it all works. And we have a number of clients that are using this technology to support these kinds of use cases. >> Awesome demo. Anything else you want to show us? >> Well we can keep going. 'Cause right now, I mean we can, oh, we're at 18 minutes. God help us. You can cut some of this. (laughs) I have a lot to say, but really this is our data fabric. The core competency of IRIS is making it smart, right? So I won't spend too much time on this but essentially if we go back to our coordinator here we can see here's that original that pipeline that we saw where we're pulling data from all these different systems and we're collecting it and we're sending it out. But then we see two more at the end here which involves getting a readmission prediction and then returning a prediction. So we can not only deliver data back as part of a smart data fabric but we can also deliver insights back to users and consumers based on data that we've aggregated as part of a smart data fabric. So in this scenario, we're actually taking all that data that we just looked at and we're running it through a machine learning model that exists within the smart data fabric pipeline and producing a readmission score to determine if this particular patient is at risk for readmission within the next 30 days. Which is a typical problem that we see in the healthcare space. So what's really exciting about what we're doing in the IRIS world is we're bringing analytics close to the data with integrated ML. So in this scenario we're actually creating the model, training the model, and then executing the model directly within the IRIS platform. So there's no shuffling of data, there's no external connections to make this happen. And it doesn't really require having a PhD in data science to understand how to do that. It leverages all really basic SQL like syntax to be able to construct and execute these predictions. So it's going one step further than the traditional data fabric example to introduce this ability to define actionable insights to our users based on the data that we've brought together. >> Well that readmission probability is huge. >> Yes. >> Right, because it directly affects the cost of for the provider and the patient, you know. So if you can anticipate the probability of readmission and either do things at that moment or you know, as an outpatient perhaps to minimize the probability then that's huge. That drops right to the bottom line. >> Absolutely, absolutely. And that really brings us from that data fabric to that smart data fabric at the end of the day which is what makes this so exciting. >> Awesome demo. >> Thank you. >> Fantastic people, are you cool? If people want to get in touch with you? >> Oh yes, absolutely. So you can find me on LinkedIn, Jessica Jowdy and we'd love to hear from you. I always love talking about this topic, so would be happy to engage on that. >> Great stuff, thank you Jess, appreciate it. >> Thank you so much. >> Okay, don't go away because in the next segment we're going to dig into the use cases where data fabric is driving business value. Stay right there.

(bright music) >> Hello everyone and welcome back to AWS re:Invent, here in wonderful Las Vegas, Nevada. We're theCUBE. I am Savannah Peterson. Joined with my co-host, Dave Vellante. Day four, you look great. Your voice has come back somehow. >> Yeah, a little bit. I don't know how. I took last night off. You guys, I know, were out partying all night, but - >> I don't know what you're talking about. (Dave laughing) >> Well, you were celebrating John's birthday. John Furrier's birthday today. >> Yes, happy birthday John! >> He's on his way to England. >> Yeah. >> To attend his nephew's wedding. Awesome family. And so good luck, John. I hope you feel better, he's got a little cold. >> I know, good luck to the newlyweds. I love this. I know we're both really excited for our next guest, so I'm going to bring out, Lena Smart from MongoDB. Thank you so much for being here. >> Thank you for having me. >> How's the show going for you? >> Good. It's been a long week. And I just, not much voice left, so. >> We'll be gentle on you. >> I'll give you what's left of it. >> All right, we'll take that. >> Okay. >> You had a fireside chat, at the show? >> Lena: I did. >> Can you tell us a little bit about that? >> So we were talking about the Rise, The developer is a platform. In this massive theater. I thought it would be like an intimate, you know, fireside chat. I keep believing them when they say to me come and do these talks, it'll be intimate. And you turn up and there's a stage and a theater and it's like, oh my god. But it was really interesting. It was well attended. Got some really good questions at the end as well. Lots of follow up, which was interesting. And it was really just about, you know, how we've brought together this developer platform that's got our integrated services. It's just what developers want, it gives them time to innovate and disrupt, rather than worry about the minutia of management. >> Savannah: Do the cool stuff. >> Exactly. >> Yeah, so you know Lena, it's funny that you're saying that oh wow, the lights came on and it was this big thing. When when we were at re:Inforced, Lena was on stage and it was so funny, Lena, you were self deprecating like making jokes about the audience. >> Savannah: (indistinct) >> It was hilarious. And so, but it was really endearing to the audience and so we were like - >> Lena: It was terrifying. >> You got huge props for that, I'll tell you. >> Absolutely terrifying. Because they told me I wouldn't see anyone. Because we did the rehearsal the day before, and they were like, it's just going to be like - >> Sometimes it just looks like blackness out there. >> Yeah, yeah. It wasn't, they lied. I could see eyeballs. It was terrifying. >> Would you rather know that going in though? Or is it better to be, is ignorance bliss in that moment? >> Ignorance is bliss. >> Yeah, yeah yeah. >> Good call Savannah, right? Yeah, just go. >> The older I get, the more I'm just, I'm on the ignorance is bliss train. I just, I don't need to know anything that's going to hurt my soul. >> Exactly. >> One of the things that you mentioned, and this has actually been a really frequent theme here on the show this week, is you said that this has been a transformative year for developers. >> Lena: Yeah. >> What did you mean by that? >> So I think developers are starting to come to the fore, if you like, the fore. And I'm not in any way being deprecating about developers 'cause I love them. >> Savannah: I think everyone here does. >> I was married to one, I live with one now. It's like, they follow me everywhere. They don't. But, I think they, this is my opinion obviously but I think that we're seeing more and more the value that developers bring to the table. They're not just code geeks anymore. They're not just code monkeys, you know, churning out lines and lines of code. Some of the most interesting discussions I've had this week have been with developers. And that's why I'm so pleased that our developer data platform is going to give these folks back time, so that they can go and innovate. And do super interesting things and do the next big thing. It was interesting, I was talking to Mary, our comms person earlier and she had said that Dave I guess, my boss, was on your show - >> Dave: Yeah, he was over here last night. >> Yeah. And he was saying that two thirds of the companies that had been mentioned so far, within the whole gamut of this conference use MongoDB. And so take that, extrapolate that, of all the developers >> Wow. >> who are there. I know, isn't that awesome? >> That's awesome. Congrats on that, that's like - >> Did I hear that right now? >> I know, I just had that moment. >> I know she just told me, I'm like, really? That's - >> That's so cool. >> 'Cause the first thing I thought of was then, oh my god, how many developers are we reaching then? 'Cause they're the ones. I mean, it's kind of interesting. So my job has kind of grown from, over the years, being the security geek in the back room that nobody talks to, to avoiding me in the lift, to I've got a seat at the table now. We meet with the board. And I think that I can see that that's where the developer mindset is moving towards. It's like, give us the right tools and we'll change your world. >> And let the human capital go back to doing the fun stuff and not just the maintenance stuff. >> And, but then you say that, you can't have everything automated. I get that automation is also the buzzword of the week. And I get that, trust me. Someone has to write the code to do the automation. >> Savannah: Right. >> So, so yeah, definitely give these people back time, so that they can work on ML, AI, choose your buzzword. You know, by giving people things like queriable encryption for example, you're going to free up a whole bunch of head space. They don't have to worry about their data being, you know harvested from memory or harvested while at rest or in motion. And it's like, okay, I don't have to worry about that now, let me go do something fun. >> How about the role of the developer as it relates to SecOps, right? They're being asked to do a lot. You and I talked about this at re:Inforce. You seem to have a pretty good handle on it. Like a lot of companies I think are struggling with it. I mean, the other thing you said said to me is you don't have a lack of talent at Mongo, right? 'Cause you're Mongo. But a lot of companies do. But a lot of the developers, you know we were just talking about this earlier with Capgemini, the developer metrics or the application development team's metrics might not be aligned with the CSO's metrics. How, what are you seeing there? What, how do you deal with it within Mongo? What do you advise your customers? >> So in terms of internal, I work very closely with our development group. So I work with Tara Hernandez, who's our new VP of developer productivity. And she and her team are very much interested in making developers more productive. That's her job. And so we get together because sometimes security can definitely be seen as a blocker. You know, funnily enough, I actually had a Slack that I had to respond to three seconds before I come on here. And it was like, help, we need some help getting this application through procurement, because blah, blah, blah. And it's weird the kind of change, the shift in mindset. Whereas before they might have gone to procurement or HR or someone to ask for this. Now they're coming to the CSO. 'Cause they know if I say yes, it'll go through. >> Talk about social engineering. >> Exactly. >> You were talking about - >> But turn it around though. If I say no, you know, I don't like to say no. I prefer to be the CSO that says yes, but. And so that's what we've done. We've definitely got that culture of ask, we'll tell you the risks, and then you can go away and be innovative and do what you need to do. And we basically do the same with our customers. Here's what you can do. Our application is secure out of the box. Here's how we can help you make it even more, you know, streamlined or bespoke to what you need. >> So mobile was a big inflection point, you know, I dunno, it seems like forever ago. >> 2007. >> 2007. Yeah, iPhone came out in 2007. >> You remember your first iPhone? >> Dave: Yeah. >> Yeah? Same. >> Yeah. It was pretty awesome, actually. >> Yeah, I do too. >> Yeah, I was on the train to Boston going up to see some friends at MIT on the consortium that I worked with. And I had, it was the wee one, 'member? But you thought it was massive. >> Oh, it felt - >> It felt big. And I remember I was sitting on the train to Boston it was like the Estella and there was these people, these two women sitting beside me. And they were all like glam, like you and unlike me. >> Dave: That's awesome. >> And they, you could see them like nudging each other. And I'm being like, I'm just sitting like this. >> You're chilling. >> Like please look at my phone, come on just look at it. Ask me about it. And eventually I'm like - >> You're baiting them. >> nonchalantly laid it on the table. And you know, I'm like, and they're like, is that an iPhone? And I'm like, yeah, you want to see it? >> I thought you'd never ask. >> I know. And I really played with it. And I showed them all the cool stuff, and they're like, oh we're going to buy iPhones. And so I should have probably worked for Apple, but I didn't. >> I was going to say, where was your referral kickback on that? Especially - >> It was a little like Tesla, right? When you first, we first saw Tesla, it was Ray Wong, you know, Ray? From Pasadena? >> It really was a moment and going from the Blackberry keyboard to that - >> He's like want to see my car? And I'm like oh yeah sure, what's the big deal? >> Yeah, then you see it and you're like, ooh. >> Yeah, that really was such a pivotal moment. >> Anyway, so we lost a track, 2007. >> Yeah, what were we talking about? 2007 mobile. >> Mobile. >> Key inflection point, is where you got us here. Thank you. >> I gotchu Dave, I gotchu. >> Bring us back here. My mind needs help right now. Day four. Okay, so - >> We're all getting here on day four, we're - >> I'm socially engineering you to end this, so I can go to bed and die quietly. That's what me and Mary are, we're counting down the minutes. >> Holy. >> That's so sick. >> You're breaking my heart right now. I love it. I'm with you, sis, I'm with you. >> So I dunno where I was, really where I was going with this, but, okay, there's - >> 2007. Three things happened. >> Another inflection point. Okay yeah, tell us what happened. But no, tell us that, but then - >> AWS, clones, 2006. >> Well 2006, 2007. Right, okay. >> 2007, the iPhone, the world blew up. So you've already got this platform ready to take all this data. >> Dave: Right. >> You've got this little slab of gorgeousness called the iPhone, ready to give you all that data. And then MongoDB pops up, it's like, woo-hoo. But what we could offer was, I mean back then was awesome, but it was, we knew that we would have to iterate and grow and grow and grow. So that was kind of the three things that came together in 2007. >> Yeah, and then Cloud came in big time, and now you've got this platform. So what's the next inflection point do you think? >> Oh... >> Good question, Dave. >> Don't even ask me that. >> I mean, is it Edge? Is it IOT? Is there another disruptor out there? >> I think it's going to be artificial intelligence. >> Dave: Is it AI? >> I mean I don't know enough about it to talk about it, to any level, so don't ask me any questions about it. >> This is like one of those ignorance is bliss moments. It feels right. >> Yeah. >> Well, does it scare you, from a security perspective? Or? >> Great question, Dave. >> Yeah, it scares me more from a humanity standpoint. Like - >> More than social scared you? 'Cause social was so benign when it started. >> Oh it was - >> You're like, oh - I remember, >> It was like a yearbook. I was on the Estella and we were - >> Shout out to Amtrak there. >> I was with, we were starting basically a wikibond, it was an open source. >> Yeah, yeah. >> Kind of, you know, technology community. And we saw these and we were like enamored of Facebook. And there were these two young kids on the train, and we were at 'em, we were picking the brain. Do you like Facebook? "I love Facebook." They're like "oh, Facebook's unbelievable." Now, kids today, "I hate Facebook," right? So, but social at the beginning it was kind of, like I say, benign and now everybody's like - >> Savannah: We didn't know what we were getting into. >> Right. >> I know. >> Exactly. >> Can you imagine if you could have seen into the future 20 years ago? Well first of all, we'd have all bought Facebook and Apple stock. >> Savannah: Right. >> And Tesla stock. But apart from, but yeah apart from that. >> Okay, so what about Quantum? Does that scare you at all? >> I think the only thing that scares me about Quantum is we have all this security in place today. And I'm not an expert in Quantum, but we have all this security in place that's securing what we have today. And my worry is, in 10 years, is it still going to be secure? 'Cause we're still going to be using that data in some way, shape, or form. And my question is to the quantum geniuses out there, what do we do in 10 years like to retrofit the stuff? >> Dave: Like a Y2K moment? >> Kind of. Although I think Y2K is coming in 2038, isn't it? When the Linux date flips. I'll be off the grid by then, I'll be living in Scotland. >> Somebody else's problem. >> Somebody else's problem. I'll be with the sheep in Glasgow, in Scotland. >> Y2K was a boondoggle for tech, right? >> What a farce. I mean, that whole - >> I worked in the power industry in Y2K. That was a nightmare. >> Dave: Oh I bet. >> Savannah: Oh my God. >> Yeah, 'cause we just assumed that the world was going to stop and there been no power, and we had nuclear power plants. And it's like holy moly. Yeah. >> More than moly. >> I was going to say, you did a good job holding that other word in. >> I think I was going to, in case my mom hears this. >> I grew up near Diablo Canyon in, in California. So you were, I mean we were legitimately worried that that exactly was going to happen. And what about the waste? And yeah it was chaos. We've covered a lot. >> Well, what does worry you? Like, it is culture? Is it - >> Why are you trying to freak her out? >> No, no, because it's a CSO, trying to get inside the CSO's head. >> You don't think I have enough to worry about? You want to keep piling on? >> Well if it's not Quantum, you know? Maybe it's spiders or like - >> Oh but I like spiders, well spiders are okay. I don't like bridges, that's my biggest fear. Bridges. >> Seriously? >> And I had to drive over the Tappan Zee bridge, which is one of the longest, for 17 years, every day, twice. The last time I drove over it, I was crying my heart out, and happy as anything. >> Stay out of Oakland. >> I've never driven over it since. Stay out of where? >> Stay out of Oakland. >> I'm staying out of anywhere that's got lots of water. 'Cause it'll have bridges. >> Savannah: Well it's good we're here in the desert. >> Exactly. So what scares me? Bridges, there you go. >> Yeah, right. What? >> Well wait a minute. So if I'm bridging technology, is that the scary stuff? >> Oh God, that was not - >> Was it really bad? >> It was really bad. >> Wow. Wow, the puns. >> There's a lot of seems in those bridges. >> It is lit on theCUBE A floor, we are all struggling. I'm curious because I've seen, your team is all over the place here on the show, of course. Your booth has been packed the whole time. >> Lena: Yes. >> The fingerprint. Talk to me about your shirt. >> So, this was designed by my team in house. It is the most wanted swag in the company, because only my security people wear it. So, we make it like, yeah, you could maybe have one, if this turns out well. >> I feel like we're on the right track. >> Dave: If it turns out well. >> Yeah, I just love it. It's so, it's just brilliant. I mean, it's the leaf, it's a fingerprint. It's just brilliant. >> That's why I wanted to call it out. You know, you see a lot of shirts, a lot of swag shirts. Some are really unfortunately sad, or not funny, >> They are. >> or they're just trying too hard. Now there's like, with this one, I thought oh I bet that's clever. >> Lena: It is very cool. Yes, I love it. >> I saw a good one yesterday. >> Yeah? >> We fix shit, 'member? >> Oh yeah, yeah. >> That was pretty good. >> I like when they're >> That's a pretty good one. >> just straightforward, like that, yeah yeah. >> But the only thing with this is when you're say in front of a green screen, you look as though you've got no tummy. >> A portal through your body. >> And so, when we did our first - >> That's a really good point, actually. >> Yeah, it's like the black hole to nothingless. And I'm like wow, that's my soul. >> I was just going to say, I don't want to see my soul like that. I don't want to know. >> But we had to do like, it was just when the pandemic first started, so we had to do our big presentation live announcement from home. And so they shipped us all this camera equipment for home and thank God my partner knows how that works, so he set it all up. And then he had me test with a green screen, and he's like, you have no tummy. I'm like, what the hell are you talking about? He's like, come and see. It's like this, I dunno what it was. So I had to actually go upstairs and felt tip with a magic marker and make it black. >> Wow. >> So that was why I did for two hours on a Friday, yeah. >> Couldn't think of another alternative, huh? >> Well no, 'cause I'm myopic when it comes to marketing and I knew I had to keep the tshirt on, and I just did that. >> Yeah. >> In hindsight, yes I could have worn an "I Fix Shit" tshirt, but I don't think my husband would've been very happy. I secure shit? >> There you go, yeah. >> There you go. >> Over to you, Savannah. >> I was going to say, I got acquainted, I don't know if I can say this, but I'm going to say it 'cause we're here right now. I got acquainted with theCUBE, wearing a shirt that said "Unfuck Kubernetes," 'cause it was a marketing campaign that I was running for one of my clients at Kim Con last year. >> That's so good. >> Yeah, so - >> Oh my God. I'll give you one of these if you get me one of those. >> I can, we can do a swapskee. We can absolutely. >> We need a few edits on this film, on the file. >> Lena: Okay, this is nothing - >> We're fallin' off the wheel. Okay, on that note, I'm going to bring us to our challenge that we discussed, before we got started on this really diverse discussion that we have had in the last 15 minutes. We've covered everything from felt tip markers to nuclear power plants. >> To the darkness of my soul. >> To the darkness of all of our souls. >> All of our souls, yes. >> Which is perhaps a little too accurate, especially at this stage in the conference. You've obviously seen a lot Lena, and you've been rockin' it, I know John was in your suite up here, at at at the Venetian. What's your 30 second hot take? Most important story, coming out of the show or for you all at Mongo this year? >> Genuinely, it was when I learned that two-thirds of the customers that had been mentioned, here, are MongoDB customers. And that just exploded in my head. 'Cause now I'm thinking of all the numbers and the metrics and how we can use that. And I just think it's amazing, so. >> Yeah, congratulations on that. That's awesome. >> Yeah, I thought it was amazing. >> And it makes sense actually, 'cause Mongo so easy to use. We were talking about Tengen. >> We knew you when, I feel that's our like, we - >> Yeah, but it's true. And so, Mongo was just really easy to use. And people are like, ah, it doesn't scale. It's like, turns out it actually does scale. >> Lena: Turns out, it scales pretty well. >> Well Lena, without question, this is my favorite conversation of the show so far. >> Thank you. >> Thank you so much for joining us. >> Thank you very much for having me. >> Dave: Great to see you. >> It's always a pleasure. >> Dave: Thanks Lena. >> Thank you. >> And thank you all, tuning in live, for tolerating wherever we take these conversations. >> Dave: Whatever that was. >> I bet you weren't ready for this one, folks. We're at AWS re:Invent in Las Vegas, Nevada. With Dave Vellante, I'm Savannah Peterson. You're washing theCUBE, the leader for high tech coverage.

(soft electronic music) >> Okay, we're back with Tim Yocum who is the Director of Engineering at InfluxData. Tim, welcome, good to see you. >> Good to see you, thanks for having me. >> You're really welcome. Listen, we've been covering opensource software on theCUBE for more than a decade and we've kind of watched the innovation from the big data ecosystem, the cloud is being built out on opensource, mobile, social platforms, key databases, and of course, InfluxDB. And InfluxData has been a big consumer and crontributor of opensource software. So my question to you is where have you seen the biggest bang for the buck from opensource software? >> So yeah, you know, Influx really, we thrive at the intersection of commercial services and opensource software, so OSS keeps us on the cutting edge. We benefit from OSS in delivering our own service from our core storage engine technologies to web services, templating engines. Our team stays lean and focused because we build on proven tools. We really build on the shoulders of giants. And like you've mentioned, even better, we contribute a lot back to the projects that we use, as well as our own product InfluxDB. >> But I got to ask you, Tim, because one of the challenge that we've seen, in particular, you saw this in the heyday of Hadoop, the innovations come so fast and furious, and as a software company, you got to place bets, you got to commit people, and sometimes those bets can be risky and not pay off. So how have you managed this challenge? >> Oh, it moves fast, yeah. That's a benefit, though, because the community moves so quickly that today's hot technology can be tomorrow's dinosaur. And what we tend to do is we fail fast and fail often; we try a lot of things. You know, you look at Kubernetes, for example. That ecosystem is driven by thousands of intelligent developers, engineers, builders. They're adding value every day, so we have to really keep up with that. And as the stack changes, we try different technologies, we try different methods. And at the end of the day, we come up with a better platform as a result of just the constant change in the environment. It is a challenge for us, but it's something that we just do every day. >> So we have a survey partner down in New York City called Enterprise Technology Research, ETR, and they do these quarterly surveys of about 1500 CIOs, IT practitioners, and they really have a good pulse on what's happening with spending. And the data shows that containers generally, but specifically Kubernetes, is one of the areas that is kind of, it's been off the charts and seen the most significant adoption and velocity particularly along with cloud, but really, Kubernetes is just, you know, still up and to the right consistently, even with the macro headwinds and all of the other stuff that we're sick of talking about. So what do you do with Kubernetes in the platform? >> Yeah, it's really central to our ability to run the product. When we first started out, we were just on AWS and the way we were running was a little bit like containers junior. Now we're running Kubernetes everywhere at AWS, Azure, Google cloud. It allows us to have a consistent experience across three different cloud providers and we can manage that in code. So our developers can focus on delivering services not trying to learn the intricacies of Amazon, Azure, and Google, and figure out how to deliver services on those three clouds with all of their differences. >> Just a followup on that, is it now, so I presume it sounds like there's a PaaS layer there to allow you guys to have a consistent experience across clouds and out to the edge, wherever. Is that correct? >> Yeah, so we've basically built more or less platform engineering is this the new, hot phrase. Kubernetes has made a lot of things easy for us because we've built a platform that our developers can lean on and they only have to learn one way of deploying their application, managing their application. And so that just gets all of the underlying infrastructure out of the way and lets them focus on delivering Influx cloud. >> And I know I'm taking a little bit of a tangent, but is that, I'll call it a PaaS layer, if I can use that term, are there specific attributes to InfluxDB or is it kind of just generally off-the-shelf PaaS? Is there any purpose built capability there that is value-add or is it pretty much generic? >> So we really build, we look at things with a build versus buy, through a build versus buy lens. Some things we want to leverage, cloud provider services, for instance, POSTGRES databases for metadata, perhaps. Get that off of our plate, let someone else run that. We're going to deploy a platform that our engineers can deliver on, that has consistency, that is all generated from code. that we can, as an SRE group, as an OPS team, that we can manage with very few people, really, and we can stamp out clusters across multiple regions in no time. >> So sometimes you build, sometimes you buy it. How do you make those decisions and what does that mean for the platform and for customers? >> Yeah, so what we're doing is, it's like everybody else will do. We're looking for trade-offs that make sense. We really want to protect our customers' data, so we look for services that support our own software with the most up-time reliability and durability we can get. Some things are just going to be easier to have a cloud provider take care of on our behalf. We make that transparent for our own team and of course, for our customers; you don't even see that. But we don't want to try to reinvent the wheel, like I had mentioned with SQL datasource for metadata, perhaps. Let's build on top of what of these three large cloud providers have already perfected and we can then focus on our platform engineering and we can help our developers then focus on the InfluxData software, the Influx cloud software. >> So take it to the customer level. What does it mean for them, what's the value that they're going to get out of all these innovations that we've been talking about today, and what can they expect in the future? >> So first of all, people who use the OSS product are really going to be at home on our cloud platform. You can run it on your desktop machine, on a single server, what have you, but then you want to scale up. We have some 270 terabytes of data across over four billion series keys that people have stored, so there's a proven ability to scale. Now in terms of the opensource software and how we've developed the platform, you're getting highly available, high cardinality time-series platform. We manage it and really, as I had mentioned earlier, we can keep up with the state of the art. We keep reinventing, we keep deploying things in realtime. We deploy to our platform every day, repeatedly, all the time. And it's that continuous deployment that allow us to continue testing things in flight, rolling things out that change, new features, better ways of doing deployments, safer ways of doing deployments. All of that happens behind the scenes and like we had mentioned earllier, Kubernetes, I mean, that allows us to get that done. We couldn't do it without having that platform as a base layer for us to then put our software on. So we iterate quickly. When you're on the Influx cloud platform, you really are able to take advantage of new features immediately. We roll things out every day and as those things go into production, you have the ability to use them. And so in the then, we want you to focus on getting actual insights from your data instead of running infrastructure, you know, let us do that for you. >> That makes sense. Are the innovations that we're talking about in the evolution of InfluxDB, do you see that as sort of a natural evolution for existing customers? Is it, I'm sure the answer is both, but is it opening up new territory for customers? Can you add some color to that? >> Yeah, it really is. It's a little bit of both. Any engineer will say, "Well it depends." So cloud-native technologies are really the hot thing, IoT, industrial IoT especially. People want to just shove tons of data out there and be able to do queries immediately and they don't want to manage infrastructure. What we've started to see are people that use the cloud service as their datastore backbone and then they use edge computing with our OSS product to ingest data from say, multiple production lines, and down-sample that data, send the rest of that data off to Influx cloud where the heavy processing takes place. So really, us being in all the different clouds and iterating on that, and being in all sorts of different regions, allows for people to really get out of the business of trying to manage that big data, have us take care of that. And, of course, as we change the platform, endusers benefit from that immediately. >> And so obviously you've taken away a lot of the heavy lifting for the infrastructure. Would you say the same things about security, especially as you go out to IoT at the edge? How should we be thinking about the value that you bring from a security perspective? >> We take security super seriously. It's built into our DNA. We do a lot of work to ensure that our platform is secure, that the data that we store is kept private. It's, of course, always a concern, you see in the news all the time, companies being compromised. That's something that you can have an entire team working on which we do, to make sure that the data that you have, whether it's in transit, whether it's at rest is always kept secure, is only viewable by you. You look at things like software bill of materials, if you're running this yourself, you have to go vet all sorts of different pieces of software and we do that, you know, as we use new tools. That's something, that's just part of our jobs to make sure that the platform that we're running has fully vetted software. And you know, with opensource especially, that's a lot of work, and so it's definitely new territory. Supply chain attacks are definitely happening at a higher clip that they used to but that is really just part of a day in the life for folks like us that are building platforms. >> And that's key, especially when you start getting into the, you know, that we talk about IoT and the operations technologies, the engineers running that infrastrucutre. You know, historically, as you know, Tim, they would air gap everything; that's how they kept it safe. But that's not feasible anymore. Everything's-- >> Can't do that. >> connected now, right? And so you've got to have a partner that is, again, take away that heavy lifting to R&D so you can focus on some of the other activities. All right, give us the last word and the key takeaways from your perspective. >> Well, you know, from my perspective, I see it as a two-lane approach, with Influx, with any time-series data. You've got a lot of stuff that you're going to run on-prem. What you had mentioned, air gapping? Sure, there's plenty of need for that. But at the end of the day, people that don't want to run big datacenters, people that want to entrust their data to a company that's got a full platform set up for them that they can build on, send that data over to the cloud. The cloud is not going away. I think a more hybrid approach is where the future lives and that's what we're prepared for. >> Tim, really appreciate you coming to the program. Great stuff, good to see you. >> Thanks very much, appreciate it. >> Okay in a moment, I'll be back to wrap up today's session. You're watching theCUBE. (soft electronic music)

>>Okay, we're back. I'm Dave Valante with The Cube and you're watching Evolving Influx DB into the smart data platform made possible by influx data. Anna East Otis Georgio is here. She's a developer advocate for influx data and we're gonna dig into the rationale and value contribution behind several open source technologies that Influx DB is leveraging to increase the granularity of time series analysis analysis and bring the world of data into realtime analytics. Anna is welcome to the program. Thanks for coming on. >>Hi, thank you so much. It's a pleasure to be here. >>Oh, you're very welcome. Okay, so IO X is being touted as this next gen open source core for Influx db. And my understanding is that it leverages in memory, of course for speed. It's a kilo store, so it gives you compression efficiency, it's gonna give you faster query speeds, it gonna use store files and object storages. So you got very cost effective approach. Are these the salient points on the platform? I know there are probably dozens of other features, but what are the high level value points that people should understand? >>Sure, that's a great question. So some of the main requirements that IOCs is trying to achieve and some of the most impressive ones to me, the first one is that it aims to have no limits on cardinality and also allow you to write any kind of event data that you want, whether that's lift tag or a field. It also wants to deliver the best in class performance on analytics queries. In addition to our already well served metrics queries, we also wanna have operator control over memory usage. So you should be able to define how much memory is used for buffering caching and query processing. Some other really important parts is the ability to have bulk data export and import, super useful. Also, broader ecosystem compatibility where possible we aim to use and embrace emerging standards in the data analytics ecosystem and have compatibility with things like sql, Python, and maybe even pandas in the future. >>Okay, so a lot there. Now we talked to Brian about how you're using Rust and and which is not a new programming language and of course we had some drama around Russ during the pandemic with the Mozilla layoffs, but the formation of the Russ Foundation really addressed any of those concerns. You got big guns like Amazon and Google and Microsoft throwing their collective weights behind it. It's really, adoption is really starting to get steep on the S-curve. So lots of platforms, lots of adoption with rust, but why rust as an alternative to say c plus plus for example? >>Sure, that's a great question. So Rust was chosen because of his exceptional performance and rebi reliability. So while rust is synt tactically similar to c c plus plus and it has similar performance, it also compiles to a native code like c plus plus. But unlike c plus plus, it also has much better memory safety. So memory safety is protection against bugs or security vulnerabilities that lead to excessive memory usage or memory leaks. And rust achieves this memory safety due to its like innovative type system. Additionally, it doesn't allow for dangling pointers and dangling pointers are the main classes of errors that lead to exploitable security vulnerabilities in languages like c plus plus. So Russ like helps meet that requirement of having no limits on card for example, because it's, we're also using the Russ implementation of Apache Arrow and this control over memory and also Russ, Russ Russ's packaging system called crates IO offers everything that you need out of the box to have features like AY and a weight to fixed race conditions to protect against buffering overflows and to ensure thread safe ay caching structures as well. So essentially it's just like has all the control, all the fine grain control, you need to take advantage of memory and all your resources as well as possible so that you can handle those really, really high ity use cases. >>Yeah, and the more I learned about the the new engine and the, and the platform IOCs et cetera, you know, you, you see things like, you know, the old days not even to even today you do a lot of garbage collection in these, in these systems and there's an inverse, you know, impact relative to performance. So it looks like you're really, you know, the community is modernizing the platform, but I wanna talk about Apache Arrow for a moment. It's designed to address the constraints that are associated with analyzing large data sets. We, we know that, but please explain why, what, what is Arrow and and what does it bring to Influx db? >>Sure, yeah. So Arrow is a, a framework for defining in memory calmer data and so much of the efficiency and performance of IOCs comes from taking advantage of calmer data structures. And I will, if you don't mind, take a moment to kind of illustrate why calmer data structures are so valuable. Let's pretend that we are gathering field data about the temperature in our room and also maybe the temperature of our stove. And in our table we have those two temperature values as well as maybe a measurement value, timestamp value, maybe some other tag values that describe what room and what house, et cetera we're getting this data from. And so you can picture this table where we have like two rows with the two temperature values for both our room and the stove. Well usually our room temperature is regulated so those values don't change very often. >>So when you have calm oriented st calm oriented storage, essentially you take each row, each column and group it together. And so if that's the case and you're just taking temperature values from the room and a lot of those temperature values are the same, then you'll, you might be able to imagine how equal values will then neighbor each other and when they neighbor each other in the storage format. This provides a really perfect opportunity for cheap compression. And then this cheap compression enables high cardinality use cases. It also enables for faster scan rates. So if you wanna define like the min and max value of the temperature in the room across a thousand different points, you only have to get those a thousand different points in order to answer that question and you have those immediately available to you. But let's contrast this with a row oriented storage solution instead so that we can understand better the benefits of calmer oriented storage. >>So if you had a row oriented storage, you'd first have to look at every field like the temperature in, in the room and the temperature of the stove. You'd have to go across every tag value that maybe describes where the room is located or what model the stove is. And every timestamp you'd then have to pluck out that one temperature value that you want at that one times stamp and do that for every single row. So you're scanning across a ton more data and that's why row oriented doesn't provide the same efficiency as calmer and Apache Arrow is in memory calmer data, calmer data fit framework. So that's where a lot of the advantages come >>From. Okay. So you've basically described like a traditional database, a row approach, but I've seen like a lot of traditional databases say, okay, now we've got, we can handle colo format versus what you're talking about is really, you know, kind of native it, is it not as effective as the, is the form not as effective because it's largely a, a bolt on? Can you, can you like elucidate on that front? >>Yeah, it's, it's not as effective because you have more expensive compression and because you can't scan across the values as quickly. And so those are, that's pretty much the main reasons why, why RO row oriented storage isn't as efficient as calm, calmer oriented storage. >>Yeah. Got it. So let's talk about Arrow data fusion. What is data fusion? I know it's written in rust, but what does it bring to to the table here? >>Sure. So it's an extensible query execution framework and it uses Arrow as its in memory format. So the way that it helps influx DB IOx is that okay, it's great if you can write unlimited amount of cardinality into influx cbis, but if you don't have a query engine that can successfully query that data, then I don't know how much value it is for you. So data fusion helps enable the, the query process and transformation of that data. It also has a PANDAS API so that you could take advantage of PDA's data frames as well and all of the machine learning tools associated with pandas. >>Okay. You're also leveraging par K in the platform course. We heard a lot about Par K in the middle of the last decade cuz as a storage format to improve on Hadoop column stores. What are you doing with Par K and why is it important? >>Sure. So Par K is the calm oriented durable file format. So it's important because it'll enable bulk import and bulk export. It has compatibility with Python and pandas so it supports a broader ecosystem. Parque files also take very little disc disc space and they're faster to scan because again they're column oriented in particular, I think PAR K files are like 16 times cheaper than CSV files, just as kind of a point of reference. And so that's essentially a lot of the, the benefits of par k. >>Got it. Very popular. So and these, what exactly is influx data focusing on as a committer to these projects? What is your focus? What's the value that you're bringing to the community? >>Sure. So Influx DB first has contributed a lot of different, different things to the Apache ecosystem. For example, they contribute an implementation of Apache Arrow and go and that will support clearing with flux. Also, there has been a quite a few contributions to data fusion for things like memory optimization and supportive additional SQL features like support for timestamp, arithmetic and support for exist clauses and support for memory control. So yeah, Influx has contributed a a lot to the Apache ecosystem and continues to do so. And I think kind of the idea here is that if you can improve these upstream projects and then the long term strategy here is that the more you contribute and build those up, then the more you will perpetuate that cycle of improvement and the more we will invest in our own project as well. So it's just that kind of symbiotic relationship and appreciation of the open source community. >>Yeah. Got it. You got that virtuous cycle going, the people call it the flywheel. Give us your last thoughts and kind of summarize, you know, where what, what the big takeaways are from your perspective. >>So I think the big takeaway is that influx data is doing a lot of really exciting things with Influx DB IOCs and I really encourage if you are interested in learning more about the technologies that Influx is leveraging to produce IOCs, the challenges associated with it and all of the hard work questions and I just wanna learn more, then I would encourage you to go to the monthly tech talks and community office hours and they are on every second Wednesday of the month at 8:30 AM Pacific time. There's also a community forums and a community Slack channel. Look for the influx D DB underscore IAC channel specifically to learn more about how to join those office hours and those monthly tech tech talks as well as ask any questions they have about IOCs, what to expect and what you'd like to learn more about. I as a developer advocate, I wanna answer your questions. So if there's a particular technology or stack that you wanna dive deeper into and want more explanation about how influx TB leverages it to build IOCs, I will be really excited to produce content on that topic for you. >>Yeah, that's awesome. You guys have a really rich community, collaborate with your peers, solve problems, and you guys super responsive, so really appreciate that. All right, thank you so much and East for explaining all this open source stuff to the audience and why it's important to the future of data. >>Thank you. I really appreciate it. >>All right, you're very welcome. Okay, stay right there and in a moment I'll be back with Tim Yokum. He's the director of engineering for Influx Data and we're gonna talk about how you update a SaaS engine while the plane is flying at 30,000 feet. You don't wanna miss this.

>>This past May, The Cube in collaboration with Influx data shared with you the latest innovations in Time series databases. We talked at length about why a purpose built time series database for many use cases, was a superior alternative to general purpose databases trying to do the same thing. Now, you may, you may remember the time series data is any data that's stamped in time, and if it's stamped, it can be analyzed historically. And when we introduced the concept to the community, we talked about how in theory, those time slices could be taken, you know, every hour, every minute, every second, you know, down to the millisecond and how the world was moving toward realtime or near realtime data analysis to support physical infrastructure like sensors and other devices and IOT equipment. A time series databases have had to evolve to efficiently support realtime data in emerging use cases in iot T and other use cases. >>And to do that, new architectural innovations have to be brought to bear. As is often the case, open source software is the linchpin to those innovations. Hello and welcome to Evolving Influx DB into the smart Data platform, made possible by influx data and produced by the Cube. My name is Dave Valante and I'll be your host today. Now, in this program, we're going to dig pretty deep into what's happening with Time series data generally, and specifically how Influx DB is evolving to support new workloads and demands and data, and specifically around data analytics use cases in real time. Now, first we're gonna hear from Brian Gilmore, who is the director of IOT and emerging technologies at Influx Data. And we're gonna talk about the continued evolution of Influx DB and the new capabilities enabled by open source generally and specific tools. And in this program, you're gonna hear a lot about things like Rust, implementation of Apache Arrow, the use of par k and tooling such as data fusion, which powering a new engine for Influx db. >>Now, these innovations, they evolve the idea of time series analysis by dramatically increasing the granularity of time series data by compressing the historical time slices, if you will, from, for example, minutes down to milliseconds. And at the same time, enabling real time analytics with an architecture that can process data much faster and much more efficiently. Now, after Brian, we're gonna hear from Anna East Dos Georgio, who is a developer advocate at In Flux Data. And we're gonna get into the why of these open source capabilities and how they contribute to the evolution of the Influx DB platform. And then we're gonna close the program with Tim Yokum, he's the director of engineering at Influx Data, and he's gonna explain how the Influx DB community actually evolved the data engine in mid-flight and which decisions went into the innovations that are coming to the market. Thank you for being here. We hope you enjoy the program. Let's get started. Okay, we're kicking things off with Brian Gilmore. He's the director of i t and emerging Technology at Influx State of Bryan. Welcome to the program. Thanks for coming on. >>Thanks Dave. Great to be here. I appreciate the time. >>Hey, explain why Influx db, you know, needs a new engine. Was there something wrong with the current engine? What's going on there? >>No, no, not at all. I mean, I think it's, for us, it's been about staying ahead of the market. I think, you know, if we think about what our customers are coming to us sort of with now, you know, related to requests like sql, you know, query support, things like that, we have to figure out a way to, to execute those for them in a way that will scale long term. And then we also, we wanna make sure we're innovating, we're sort of staying ahead of the market as well and sort of anticipating those future needs. So, you know, this is really a, a transparent change for our customers. I mean, I think we'll be adding new capabilities over time that sort of leverage this new engine, but you know, initially the customers who are using us are gonna see just great improvements in performance, you know, especially those that are working at the top end of the, of the workload scale, you know, the massive data volumes and things like that. >>Yeah, and we're gonna get into that today and the architecture and the like, but what was the catalyst for the enhancements? I mean, when and how did this all come about? >>Well, I mean, like three years ago we were primarily on premises, right? I mean, I think we had our open source, we had an enterprise product, you know, and, and sort of shifting that technology, especially the open source code base to a service basis where we were hosting it through, you know, multiple cloud providers. That was, that was, that was a long journey I guess, you know, phase one was, you know, we wanted to host enterprise for our customers, so we sort of created a service that we just managed and ran our enterprise product for them. You know, phase two of this cloud effort was to, to optimize for like multi-tenant, multi-cloud, be able to, to host it in a truly like sass manner where we could use, you know, some type of customer activity or consumption as the, the pricing vector, you know, And, and that was sort of the birth of the, of the real first influx DB cloud, you know, which has been really successful. >>We've seen, I think, like 60,000 people sign up and we've got tons and tons of, of both enterprises as well as like new companies, developers, and of course a lot of home hobbyists and enthusiasts who are using out on a, on a daily basis, you know, and having that sort of big pool of, of very diverse and very customers to chat with as they're using the product, as they're giving us feedback, et cetera, has has, you know, pointed us in a really good direction in terms of making sure we're continuously improving that and then also making these big leaps as we're doing with this, with this new engine. >>Right. So you've called it a transparent change for customers, so I'm presuming it's non-disruptive, but I really wanna understand how much of a pivot this is and what, what does it take to make that shift from, you know, time series, you know, specialist to real time analytics and being able to support both? >>Yeah, I mean, it's much more of an evolution, I think, than like a shift or a pivot. You know, time series data is always gonna be fundamental and sort of the basis of the solutions that we offer our customers, and then also the ones that they're building on the sort of raw APIs of our platform themselves. You know, the time series market is one that we've worked diligently to lead. I mean, I think when it comes to like metrics, especially like sensor data and app and infrastructure metrics, if we're being honest though, I think our, our user base is well aware that the way we were architected was much more towards those sort of like backwards looking historical type analytics, which are key for troubleshooting and making sure you don't, you know, run into the same problem twice. But, you know, we had to ask ourselves like, what can we do to like better handle those queries from a performance and a, and a, you know, a time to response on the queries, and can we get that to the point where the results sets are coming back so quickly from the time of query that we can like limit that window down to minutes and then seconds. >>And now with this new engine, we're really starting to talk about a query window that could be like returning results in, in, you know, milliseconds of time since it hit the, the, the ingest queue. And that's, that's really getting to the point where as your data is available, you can use it and you can query it, you can visualize it, and you can do all those sort of magical things with it, you know? And I think getting all of that to a place where we're saying like, yes to the customer on, you know, all of the, the real time queries, the, the multiple language query support, but, you know, it was hard, but we're now at a spot where we can start introducing that to, you know, a a limited number of customers, strategic customers and strategic availability zones to start. But you know, everybody over time. >>So you're basically going from what happened to in, you can still do that obviously, but to what's happening now in the moment? >>Yeah, yeah. I mean, if you think about time, it's always sort of past, right? I mean, like in the moment right now, whether you're talking about like a millisecond ago or a minute ago, you know, that's, that's pretty much right now, I think for most people, especially in these use cases where you have other sort of components of latency induced by the, by the underlying data collection, the architecture, the infrastructure, the, you know, the, the devices and you know, the sort of highly distributed nature of all of this. So yeah, I mean, getting, getting a customer or a user to be able to use the data as soon as it is available is what we're after here. >>I always thought, you know, real, I always thought of real time as before you lose the customer, but now in this context, maybe it's before the machine blows up. >>Yeah, it's, it's, I mean it is operationally or operational real time is different, you know, and that's one of the things that really triggered us to know that we were, we were heading in the right direction, is just how many sort of operational customers we have. You know, everything from like aerospace and defense. We've got companies monitoring satellites, we've got tons of industrial users, users using us as a processes storing on the plant floor, you know, and, and if we can satisfy their sort of demands for like real time historical perspective, that's awesome. I think what we're gonna do here is we're gonna start to like edge into the real time that they're used to in terms of, you know, the millisecond response times that they expect of their control systems. Certainly not their, their historians and databases. >>I, is this available, these innovations to influx DB cloud customers only who can access this capability? >>Yeah. I mean, commercially and today, yes. You know, I think we want to emphasize that's a, for now our goal is to get our latest and greatest and our best to everybody over time. Of course. You know, one of the things we had to do here was like we double down on sort of our, our commitment to open source and availability. So like anybody today can take a look at the, the libraries in on our GitHub and, you know, can ex inspect it and even can try to, you know, implement or execute some of it themselves in their own infrastructure. You know, we are, we're committed to bringing our sort of latest and greatest to our cloud customers first for a couple of reasons. Number one, you know, there are big workloads and they have high expectations of us. I think number two, it also gives us the opportunity to monitor a little bit more closely how it's working, how they're using it, like how the system itself is performing. >>And so just, you know, being careful, maybe a little cautious in terms of, of, of how big we go with this right away. Just sort of both limits, you know, the risk of, of, you know, any issues that can come with new software rollouts. We haven't seen anything so far, but also it does give us the opportunity to have like meaningful conversations with a small group of users who are using the products, but once we get through that and they give us two thumbs up on it, it'll be like, open the gates and let everybody in. It's gonna be exciting time for the whole ecosystem. >>Yeah, that makes a lot of sense. And you can do some experimentation and, you know, using the cloud resources. Let's dig into some of the architectural and technical innovations that are gonna help deliver on this vision. What, what should we know there? >>Well, I mean, I think foundationally we built the, the new core on Rust. You know, this is a new very sort of popular systems language, you know, it's extremely efficient, but it's also built for speed and memory safety, which goes back to that us being able to like deliver it in a way that is, you know, something we can inspect very closely, but then also rely on the fact that it's going to behave well. And if it does find error conditions, I mean, we, we've loved working with Go and, you know, a lot of our libraries will continue to, to be sort of implemented in Go, but you know, when it came to this particular new engine, you know, that power performance and stability rust was critical. On top of that, like, we've also integrated Apache Arrow and Apache Parque for persistence. I think for anybody who's really familiar with the nuts and bolts of our backend and our TSI and our, our time series merged Trees, this is a big break from that, you know, arrow on the sort of in MI side and then Par K in the on disk side. >>It, it allows us to, to present, you know, a unified set of APIs for those really fast real time inquiries that we talked about, as well as for very large, you know, historical sort of bulk data archives in that PARQUE format, which is also cool because there's an entire ecosystem sort of popping up around Parque in terms of the machine learning community, you know, and getting that all to work, we had to glue it together with aero flight. That's sort of what we're using as our, our RPC component. You know, it handles the orchestration and the, the transportation of the Coer data. Now we're moving to like a true Coer database model for this, this version of the engine, you know, and it removes a lot of overhead for us in terms of having to manage all that serialization, the deserialization, and, you know, to that again, like blurring that line between real time and historical data. It's, you know, it's, it's highly optimized for both streaming micro batch and then batches, but true streaming as well. >>Yeah. Again, I mean, it's funny you mentioned Rust. It is, it's been around for a long time, but it's popularity is, is, you know, really starting to hit that steep part of the S-curve. And, and we're gonna dig into to more of that, but give us any, is there anything else that we should know about Bryan? Give us the last word? >>Well, I mean, I think first I'd like everybody sort of watching just to like, take a look at what we're offering in terms of early access in beta programs. I mean, if, if, if you wanna participate or if you wanna work sort of in terms of early access with the, with the new engine, please reach out to the team. I'm sure you know, there's a lot of communications going out and, you know, it'll be highly featured on our, our website, you know, but reach out to the team, believe it or not, like we have a lot more going on than just the new engine. And so there are also other programs, things we're, we're offering to customers in terms of the user interface, data collection and things like that. And, you know, if you're a customer of ours and you have a sales team, a commercial team that you work with, you can reach out to them and see what you can get access to because we can flip a lot of stuff on, especially in cloud through feature flags. >>But if there's something new that you wanna try out, we'd just love to hear from you. And then, you know, our goal would be that as we give you access to all of these new cool features that, you know, you would give us continuous feedback on these products and services, not only like what you need today, but then what you'll need tomorrow to, to sort of build the next versions of your business. Because, you know, the whole database, the ecosystem as it expands out into to, you know, this vertically oriented stack of cloud services and enterprise databases and edge databases, you know, it's gonna be what we all make it together, not just, you know, those of us who were employed by Influx db. And then finally, I would just say please, like watch in ice in Tim's sessions, Like these are two of our best and brightest. They're totally brilliant, completely pragmatic, and they are most of all customer obsessed, which is amazing. And there's no better takes, like honestly on the, the sort of technical details of this, then there's, especially when it comes to like the value that these investments will, will bring to our customers and our communities. So encourage you to, to, you know, pay more attention to them than you did to me, for sure. >>Brian Gilmore, great stuff. Really appreciate your time. Thank you. >>Yeah, thanks Dave. It was awesome. Look forward to it. >>Yeah, me too. Looking forward to see how the, the community actually applies these new innovations and goes, goes beyond just the historical into the real time, really hot area. As Brian said in a moment, I'll be right back with Anna East Dos Georgio to dig into the critical aspects of key open source components of the Influx DB engine, including Rust, Arrow, Parque, data fusion. Keep it right there. You don't want to miss this.

>>This past May, The Cube in collaboration with Influx data shared with you the latest innovations in Time series databases. We talked at length about why a purpose built time series database for many use cases, was a superior alternative to general purpose databases trying to do the same thing. Now, you may, you may remember the time series data is any data that's stamped in time, and if it's stamped, it can be analyzed historically. And when we introduced the concept to the community, we talked about how in theory, those time slices could be taken, you know, every hour, every minute, every second, you know, down to the millisecond and how the world was moving toward realtime or near realtime data analysis to support physical infrastructure like sensors and other devices and IOT equipment. A time series databases have had to evolve to efficiently support realtime data in emerging use cases in iot T and other use cases. >>And to do that, new architectural innovations have to be brought to bear. As is often the case, open source software is the linchpin to those innovations. Hello and welcome to Evolving Influx DB into the smart Data platform, made possible by influx data and produced by the Cube. My name is Dave Valante and I'll be your host today. Now in this program we're going to dig pretty deep into what's happening with Time series data generally, and specifically how Influx DB is evolving to support new workloads and demands and data, and specifically around data analytics use cases in real time. Now, first we're gonna hear from Brian Gilmore, who is the director of IOT and emerging technologies at Influx Data. And we're gonna talk about the continued evolution of Influx DB and the new capabilities enabled by open source generally and specific tools. And in this program you're gonna hear a lot about things like Rust, implementation of Apache Arrow, the use of par k and tooling such as data fusion, which powering a new engine for Influx db. >>Now, these innovations, they evolve the idea of time series analysis by dramatically increasing the granularity of time series data by compressing the historical time slices, if you will, from, for example, minutes down to milliseconds. And at the same time, enabling real time analytics with an architecture that can process data much faster and much more efficiently. Now, after Brian, we're gonna hear from Anna East Dos Georgio, who is a developer advocate at In Flux Data. And we're gonna get into the why of these open source capabilities and how they contribute to the evolution of the Influx DB platform. And then we're gonna close the program with Tim Yokum, he's the director of engineering at Influx Data, and he's gonna explain how the Influx DB community actually evolved the data engine in mid-flight and which decisions went into the innovations that are coming to the market. Thank you for being here. We hope you enjoy the program. Let's get started. Okay, we're kicking things off with Brian Gilmore. He's the director of i t and emerging Technology at Influx State of Bryan. Welcome to the program. Thanks for coming on. >>Thanks Dave. Great to be here. I appreciate the time. >>Hey, explain why Influx db, you know, needs a new engine. Was there something wrong with the current engine? What's going on there? >>No, no, not at all. I mean, I think it's, for us, it's been about staying ahead of the market. I think, you know, if we think about what our customers are coming to us sort of with now, you know, related to requests like sql, you know, query support, things like that, we have to figure out a way to, to execute those for them in a way that will scale long term. And then we also, we wanna make sure we're innovating, we're sort of staying ahead of the market as well and sort of anticipating those future needs. So, you know, this is really a, a transparent change for our customers. I mean, I think we'll be adding new capabilities over time that sort of leverage this new engine, but you know, initially the customers who are using us are gonna see just great improvements in performance, you know, especially those that are working at the top end of the, of the workload scale, you know, the massive data volumes and things like that. >>Yeah, and we're gonna get into that today and the architecture and the like, but what was the catalyst for the enhancements? I mean, when and how did this all come about? >>Well, I mean, like three years ago we were primarily on premises, right? I mean, I think we had our open source, we had an enterprise product, you know, and, and sort of shifting that technology, especially the open source code base to a service basis where we were hosting it through, you know, multiple cloud providers. That was, that was, that was a long journey I guess, you know, phase one was, you know, we wanted to host enterprise for our customers, so we sort of created a service that we just managed and ran our enterprise product for them. You know, phase two of this cloud effort was to, to optimize for like multi-tenant, multi-cloud, be able to, to host it in a truly like sass manner where we could use, you know, some type of customer activity or consumption as the, the pricing vector, you know, And, and that was sort of the birth of the, of the real first influx DB cloud, you know, which has been really successful. >>We've seen, I think like 60,000 people sign up and we've got tons and tons of, of both enterprises as well as like new companies, developers, and of course a lot of home hobbyists and enthusiasts who are using out on a, on a daily basis, you know, and having that sort of big pool of, of very diverse and very customers to chat with as they're using the product, as they're giving us feedback, et cetera, has has, you know, pointed us in a really good direction in terms of making sure we're continuously improving that and then also making these big leaps as we're doing with this, with this new engine. >>Right. So you've called it a transparent change for customers, so I'm presuming it's non-disruptive, but I really wanna understand how much of a pivot this is and what, what does it take to make that shift from, you know, time series, you know, specialist to real time analytics and being able to support both? >>Yeah, I mean, it's much more of an evolution, I think, than like a shift or a pivot. You know, time series data is always gonna be fundamental and sort of the basis of the solutions that we offer our customers, and then also the ones that they're building on the sort of raw APIs of our platform themselves. You know, the time series market is one that we've worked diligently to lead. I mean, I think when it comes to like metrics, especially like sensor data and app and infrastructure metrics, if we're being honest though, I think our, our user base is well aware that the way we were architected was much more towards those sort of like backwards looking historical type analytics, which are key for troubleshooting and making sure you don't, you know, run into the same problem twice. But, you know, we had to ask ourselves like, what can we do to like better handle those queries from a performance and a, and a, you know, a time to response on the queries, and can we get that to the point where the results sets are coming back so quickly from the time of query that we can like limit that window down to minutes and then seconds. >>And now with this new engine, we're really starting to talk about a query window that could be like returning results in, in, you know, milliseconds of time since it hit the, the, the ingest queue. And that's, that's really getting to the point where as your data is available, you can use it and you can query it, you can visualize it, and you can do all those sort of magical things with it, you know? And I think getting all of that to a place where we're saying like, yes to the customer on, you know, all of the, the real time queries, the, the multiple language query support, but, you know, it was hard, but we're now at a spot where we can start introducing that to, you know, a a limited number of customers, strategic customers and strategic availability zones to start. But you know, everybody over time. >>So you're basically going from what happened to in, you can still do that obviously, but to what's happening now in the moment? >>Yeah, yeah. I mean if you think about time, it's always sort of past, right? I mean, like in the moment right now, whether you're talking about like a millisecond ago or a minute ago, you know, that's, that's pretty much right now, I think for most people, especially in these use cases where you have other sort of components of latency induced by the, by the underlying data collection, the architecture, the infrastructure, the, you know, the, the devices and you know, the sort of highly distributed nature of all of this. So yeah, I mean, getting, getting a customer or a user to be able to use the data as soon as it is available is what we're after here. >>I always thought, you know, real, I always thought of real time as before you lose the customer, but now in this context, maybe it's before the machine blows up. >>Yeah, it's, it's, I mean it is operationally or operational real time is different, you know, and that's one of the things that really triggered us to know that we were, we were heading in the right direction, is just how many sort of operational customers we have. You know, everything from like aerospace and defense. We've got companies monitoring satellites, we've got tons of industrial users, users using us as a processes storing on the plant floor, you know, and, and if we can satisfy their sort of demands for like real time historical perspective, that's awesome. I think what we're gonna do here is we're gonna start to like edge into the real time that they're used to in terms of, you know, the millisecond response times that they expect of their control systems, certainly not their, their historians and databases. >>I, is this available, these innovations to influx DB cloud customers only who can access this capability? >>Yeah. I mean commercially and today, yes. You know, I think we want to emphasize that's a, for now our goal is to get our latest and greatest and our best to everybody over time. Of course. You know, one of the things we had to do here was like we double down on sort of our, our commitment to open source and availability. So like anybody today can take a look at the, the libraries in on our GitHub and, you know, can ex inspect it and even can try to, you know, implement or execute some of it themselves in their own infrastructure. You know, we are, we're committed to bringing our sort of latest and greatest to our cloud customers first for a couple of reasons. Number one, you know, there are big workloads and they have high expectations of us. I think number two, it also gives us the opportunity to monitor a little bit more closely how it's working, how they're using it, like how the system itself is performing. >>And so just, you know, being careful, maybe a little cautious in terms of, of, of how big we go with this right away, just sort of both limits, you know, the risk of, of, you know, any issues that can come with new software rollouts. We haven't seen anything so far, but also it does give us the opportunity to have like meaningful conversations with a small group of users who are using the products, but once we get through that and they give us two thumbs up on it, it'll be like, open the gates and let everybody in. It's gonna be exciting time for the whole ecosystem. >>Yeah, that makes a lot of sense. And you can do some experimentation and, you know, using the cloud resources. Let's dig into some of the architectural and technical innovations that are gonna help deliver on this vision. What, what should we know there? >>Well, I mean, I think foundationally we built the, the new core on Rust. You know, this is a new very sort of popular systems language, you know, it's extremely efficient, but it's also built for speed and memory safety, which goes back to that us being able to like deliver it in a way that is, you know, something we can inspect very closely, but then also rely on the fact that it's going to behave well. And if it does find error conditions, I mean we, we've loved working with Go and, you know, a lot of our libraries will continue to, to be sort of implemented in Go, but you know, when it came to this particular new engine, you know, that power performance and stability rust was critical. On top of that, like, we've also integrated Apache Arrow and Apache Parque for persistence. I think for anybody who's really familiar with the nuts and bolts of our backend and our TSI and our, our time series merged Trees, this is a big break from that, you know, arrow on the sort of in MI side and then Par K in the on disk side. >>It, it allows us to, to present, you know, a unified set of APIs for those really fast real time inquiries that we talked about, as well as for very large, you know, historical sort of bulk data archives in that PARQUE format, which is also cool because there's an entire ecosystem sort of popping up around Parque in terms of the machine learning community, you know, and getting that all to work, we had to glue it together with aero flight. That's sort of what we're using as our, our RPC component. You know, it handles the orchestration and the, the transportation of the Coer data. Now we're moving to like a true Coer database model for this, this version of the engine, you know, and it removes a lot of overhead for us in terms of having to manage all that serialization, the deserialization, and, you know, to that again, like blurring that line between real time and historical data. It's, you know, it's, it's highly optimized for both streaming micro batch and then batches, but true streaming as well. >>Yeah. Again, I mean, it's funny you mentioned Rust. It is, it's been around for a long time, but it's popularity is, is you know, really starting to hit that steep part of the S-curve. And, and we're gonna dig into to more of that, but give us any, is there anything else that we should know about Bryan? Give us the last word? >>Well, I mean, I think first I'd like everybody sort of watching just to like take a look at what we're offering in terms of early access in beta programs. I mean, if, if, if you wanna participate or if you wanna work sort of in terms of early access with the, with the new engine, please reach out to the team. I'm sure you know, there's a lot of communications going out and you know, it'll be highly featured on our, our website, you know, but reach out to the team, believe it or not, like we have a lot more going on than just the new engine. And so there are also other programs, things we're, we're offering to customers in terms of the user interface, data collection and things like that. And, you know, if you're a customer of ours and you have a sales team, a commercial team that you work with, you can reach out to them and see what you can get access to because we can flip a lot of stuff on, especially in cloud through feature flags. >>But if there's something new that you wanna try out, we'd just love to hear from you. And then, you know, our goal would be that as we give you access to all of these new cool features that, you know, you would give us continuous feedback on these products and services, not only like what you need today, but then what you'll need tomorrow to, to sort of build the next versions of your business. Because you know, the whole database, the ecosystem as it expands out into to, you know, this vertically oriented stack of cloud services and enterprise databases and edge databases, you know, it's gonna be what we all make it together, not just, you know, those of us who were employed by Influx db. And then finally I would just say please, like watch in ICE in Tim's sessions, like these are two of our best and brightest, They're totally brilliant, completely pragmatic, and they are most of all customer obsessed, which is amazing. And there's no better takes, like honestly on the, the sort of technical details of this, then there's, especially when it comes to like the value that these investments will, will bring to our customers and our communities. So encourage you to, to, you know, pay more attention to them than you did to me, for sure. >>Brian Gilmore, great stuff. Really appreciate your time. Thank you. >>Yeah, thanks Dave. It was awesome. Look forward to it. >>Yeah, me too. Looking forward to see how the, the community actually applies these new innovations and goes, goes beyond just the historical into the real time really hot area. As Brian said in a moment, I'll be right back with Anna East dos Georgio to dig into the critical aspects of key open source components of the Influx DB engine, including Rust, Arrow, Parque, data fusion. Keep it right there. You don't wanna miss this >>Time series Data is everywhere. The number of sensors, systems and applications generating time series data increases every day. All these data sources producing so much data can cause analysis paralysis. Influx DB is an entire platform designed with everything you need to quickly build applications that generate value from time series data influx. DB Cloud is a serverless solution, which means you don't need to buy or manage your own servers. There's no need to worry about provisioning because you only pay for what you use. Influx DB Cloud is fully managed so you get the newest features and enhancements as they're added to the platform's code base. It also means you can spend time building solutions and delivering value to your users instead of wasting time and effort managing something else. Influx TVB Cloud offers a range of security features to protect your data, multiple layers of redundancy ensure you don't lose any data access controls ensure that only the people who should see your data can see it. >>And encryption protects your data at rest and in transit between any of our regions or cloud providers. InfluxDB uses a single API across the entire platform suite so you can build on open source, deploy to the cloud and then then easily query data in the cloud at the edge or on prem using the same scripts. And InfluxDB is schemaless automatically adjusting to changes in the shape of your data without requiring changes in your application. Logic. InfluxDB Cloud is production ready from day one. All it needs is your data and your imagination. Get started today@influxdata.com slash cloud. >>Okay, we're back. I'm Dave Valante with a Cube and you're watching evolving Influx DB into the smart data platform made possible by influx data. Anna ETOs Georgio is here, she's a developer advocate for influx data and we're gonna dig into the rationale and value contribution behind several open source technologies that Influx DB is leveraging to increase the granularity of time series analysis analysis and bring the world of data into real-time analytics and is welcome to the program. Thanks for coming on. >>Hi, thank you so much. It's a pleasure to be here. >>Oh, you're very welcome. Okay, so IX is being touted as this next gen open source core for Influx db. And my understanding is that it leverages in memory of course for speed. It's a kilo store, so it gives you a compression efficiency, it's gonna give you faster query speeds, you store files and object storage, so you got very cost effective approach. Are these the salient points on the platform? I know there are probably dozens of other features, but what are the high level value points that people should understand? >>Sure, that's a great question. So some of the main requirements that IOx is trying to achieve and some of the most impressive ones to me, the first one is that it aims to have no limits on cardinality and also allow you to write any kind of event data that you want, whether that's live tag or a field. It also wants to deliver the best in class performance on analytics queries. In addition to our already well served metrics queries, we also wanna have operator control over memory usage. So you should be able to define how much memory is used for buffering caching and query processing. Some other really important parts is the ability to have bulk data export and import super useful. Also broader ecosystem compatibility where possible we aim to use and embrace emerging standards in the data analytics ecosystem and have compatibility with things like sql, Python, and maybe even pandas in the future. >>Okay, so lot there. Now we talked to Brian about how you're using Rust and which is not a new programming language and of course we had some drama around Rust during the pandemic with the Mozilla layoffs, but the formation of the Rust Foundation really addressed any of those concerns. You got big guns like Amazon and Google and Microsoft throwing their collective weights behind it. It's really, the adoption is really starting to get steep on the S-curve. So lots of platforms, lots of adoption with rust, but why rust as an alternative to say c plus plus for example? >>Sure, that's a great question. So Russ was chosen because of his exceptional performance and reliability. So while Russ is synt tactically similar to c plus plus and it has similar performance, it also compiles to a native code like c plus plus. But unlike c plus plus, it also has much better memory safety. So memory safety is protection against bugs or security vulnerabilities that lead to excessive memory usage or memory leaks. And rust achieves this memory safety due to its like innovative type system. Additionally, it doesn't allow for dangling pointers. And dangling pointers are the main classes of errors that lead to exploitable security vulnerabilities in languages like c plus plus. So Russ like helps meet that requirement of having no limits on ality, for example, because it's, we're also using the Russ implementation of Apache Arrow and this control over memory and also Russ Russ's packaging system called crates IO offers everything that you need out of the box to have features like AY and a weight to fix race conditions, to protection against buffering overflows and to ensure thread safe async cashing structures as well. So essentially it's just like has all the control, all the fine grain control, you need to take advantage of memory and all your resources as well as possible so that you can handle those really, really high ity use cases. >>Yeah, and the more I learn about the, the new engine and, and the platform IOCs et cetera, you know, you, you see things like, you know, the old days not even to even today you do a lot of garbage collection in these, in these systems and there's an inverse, you know, impact relative to performance. So it looks like you really, you know, the community is modernizing the platform, but I wanna talk about Apache Arrow for a moment. It it's designed to address the constraints that are associated with analyzing large data sets. We, we know that, but please explain why, what, what is Arrow and and what does it bring to Influx db? >>Sure, yeah. So Arrow is a, a framework for defining in memory calmer data. And so much of the efficiency and performance of IOx comes from taking advantage of calmer data structures. And I will, if you don't mind, take a moment to kind of of illustrate why column or data structures are so valuable. Let's pretend that we are gathering field data about the temperature in our room and also maybe the temperature of our stove. And in our table we have those two temperature values as well as maybe a measurement value, timestamp value, maybe some other tag values that describe what room and what house, et cetera we're getting this data from. And so you can picture this table where we have like two rows with the two temperature values for both our room and the stove. Well usually our room temperature is regulated so those values don't change very often. >>So when you have calm oriented st calm oriented storage, essentially you take each row, each column and group it together. And so if that's the case and you're just taking temperature values from the room and a lot of those temperature values are the same, then you'll, you might be able to imagine how equal values will then enable each other and when they neighbor each other in the storage format, this provides a really perfect opportunity for cheap compression. And then this cheap compression enables high cardinality use cases. It also enables for faster scan rates. So if you wanna define like the men and max value of the temperature in the room across a thousand different points, you only have to get those a thousand different points in order to answer that question and you have those immediately available to you. But let's contrast this with a row oriented storage solution instead so that we can understand better the benefits of calmer oriented storage. >>So if you had a row oriented storage, you'd first have to look at every field like the temperature in, in the room and the temperature of the stove. You'd have to go across every tag value that maybe describes where the room is located or what model the stove is. And every timestamp you'd then have to pluck out that one temperature value that you want at that one time stamp and do that for every single row. So you're scanning across a ton more data and that's why Rowe Oriented doesn't provide the same efficiency as calmer and Apache Arrow is in memory calmer data, commoner data fit framework. So that's where a lot of the advantages come >>From. Okay. So you basically described like a traditional database, a row approach, but I've seen like a lot of traditional database say, okay, now we've got, we can handle colo format versus what you're talking about is really, you know, kind of native i, is it not as effective? Is the, is the foreman not as effective because it's largely a, a bolt on? Can you, can you like elucidate on that front? >>Yeah, it's, it's not as effective because you have more expensive compression and because you can't scan across the values as quickly. And so those are, that's pretty much the main reasons why, why RO row oriented storage isn't as efficient as calm, calmer oriented storage. Yeah. >>Got it. So let's talk about Arrow Data Fusion. What is data fusion? I know it's written in Rust, but what does it bring to the table here? >>Sure. So it's an extensible query execution framework and it uses Arrow as it's in memory format. So the way that it helps in influx DB IOCs is that okay, it's great if you can write unlimited amount of cardinality into influx Cbis, but if you don't have a query engine that can successfully query that data, then I don't know how much value it is for you. So Data fusion helps enable the, the query process and transformation of that data. It also has a PANDAS API so that you could take advantage of PANDAS data frames as well and all of the machine learning tools associated with Pandas. >>Okay. You're also leveraging Par K in the platform cause we heard a lot about Par K in the middle of the last decade cuz as a storage format to improve on Hadoop column stores. What are you doing with Parque and why is it important? >>Sure. So parque is the column oriented durable file format. So it's important because it'll enable bulk import, bulk export, it has compatibility with Python and Pandas, so it supports a broader ecosystem. Par K files also take very little disc disc space and they're faster to scan because again, they're column oriented in particular, I think PAR K files are like 16 times cheaper than CSV files, just as kind of a point of reference. And so that's essentially a lot of the, the benefits of par k. >>Got it. Very popular. So and he's, what exactly is influx data focusing on as a committer to these projects? What is your focus? What's the value that you're bringing to the community? >>Sure. So Influx DB first has contributed a lot of different, different things to the Apache ecosystem. For example, they contribute an implementation of Apache Arrow and go and that will support clearing with flux. Also, there has been a quite a few contributions to data fusion for things like memory optimization and supportive additional SQL features like support for timestamp, arithmetic and support for exist clauses and support for memory control. So yeah, Influx has contributed a a lot to the Apache ecosystem and continues to do so. And I think kind of the idea here is that if you can improve these upstream projects and then the long term strategy here is that the more you contribute and build those up, then the more you will perpetuate that cycle of improvement and the more we will invest in our own project as well. So it's just that kind of symbiotic relationship and appreciation of the open source community. >>Yeah. Got it. You got that virtuous cycle going, the people call the flywheel. Give us your last thoughts and kind of summarize, you know, where what, what the big takeaways are from your perspective. >>So I think the big takeaway is that influx data is doing a lot of really exciting things with Influx DB IOx and I really encourage, if you are interested in learning more about the technologies that Influx is leveraging to produce IOCs, the challenges associated with it and all of the hard work questions and you just wanna learn more, then I would encourage you to go to the monthly Tech talks and community office hours and they are on every second Wednesday of the month at 8:30 AM Pacific time. There's also a community forums and a community Slack channel look for the influx DDB unders IAC channel specifically to learn more about how to join those office hours and those monthly tech tech talks as well as ask any questions they have about iacs, what to expect and what you'd like to learn more about. I as a developer advocate, I wanna answer your questions. So if there's a particular technology or stack that you wanna dive deeper into and want more explanation about how INFLUX DB leverages it to build IOCs, I will be really excited to produce content on that topic for you. >>Yeah, that's awesome. You guys have a really rich community, collaborate with your peers, solve problems, and, and you guys super responsive, so really appreciate that. All right, thank you so much Anise for explaining all this open source stuff to the audience and why it's important to the future of data. >>Thank you. I really appreciate it. >>All right, you're very welcome. Okay, stay right there and in a moment I'll be back with Tim Yoakum, he's the director of engineering for Influx Data and we're gonna talk about how you update a SAS engine while the plane is flying at 30,000 feet. You don't wanna miss this. >>I'm really glad that we went with InfluxDB Cloud for our hosting because it has saved us a ton of time. It's helped us move faster, it's saved us money. And also InfluxDB has good support. My name's Alex Nada. I am CTO at Noble nine. Noble Nine is a platform to measure and manage service level objectives, which is a great way of measuring the reliability of your systems. You can essentially think of an slo, the product we're providing to our customers as a bunch of time series. So we need a way to store that data and the corresponding time series that are related to those. The main reason that we settled on InfluxDB as we were shopping around is that InfluxDB has a very flexible query language and as a general purpose time series database, it basically had the set of features we were looking for. >>As our platform has grown, we found InfluxDB Cloud to be a really scalable solution. We can quickly iterate on new features and functionality because Influx Cloud is entirely managed, it probably saved us at least a full additional person on our team. We also have the option of running InfluxDB Enterprise, which gives us the ability to even host off the cloud or in a private cloud if that's preferred by a customer. Influx data has been really flexible in adapting to the hosting requirements that we have. They listened to the challenges we were facing and they helped us solve it. As we've continued to grow, I'm really happy we have influx data by our side. >>Okay, we're back with Tim Yokum, who is the director of engineering at Influx Data. Tim, welcome. Good to see you. >>Good to see you. Thanks for having me. >>You're really welcome. Listen, we've been covering open source software in the cube for more than a decade, and we've kind of watched the innovation from the big data ecosystem. The cloud has been being built out on open source, mobile, social platforms, key databases, and of course influx DB and influx data has been a big consumer and contributor of open source software. So my question to you is, where have you seen the biggest bang for the buck from open source software? >>So yeah, you know, influx really, we thrive at the intersection of commercial services and open, so open source software. So OSS keeps us on the cutting edge. We benefit from OSS in delivering our own service from our core storage engine technologies to web services temping engines. Our, our team stays lean and focused because we build on proven tools. We really build on the shoulders of giants and like you've mentioned, even better, we contribute a lot back to the projects that we use as well as our own product influx db. >>You know, but I gotta ask you, Tim, because one of the challenge that that we've seen in particular, you saw this in the heyday of Hadoop, the, the innovations come so fast and furious and as a software company you gotta place bets, you gotta, you know, commit people and sometimes those bets can be risky and not pay off well, how have you managed this challenge? >>Oh, it moves fast. Yeah, that, that's a benefit though because it, the community moves so quickly that today's hot technology can be tomorrow's dinosaur. And what we, what we tend to do is, is we fail fast and fail often. We try a lot of things. You know, you look at Kubernetes for example, that ecosystem is driven by thousands of intelligent developers, engineers, builders, they're adding value every day. So we have to really keep up with that. And as the stack changes, we, we try different technologies, we try different methods, and at the end of the day, we come up with a better platform as a result of just the constant change in the environment. It is a challenge for us, but it's, it's something that we just do every day. >>So we have a survey partner down in New York City called Enterprise Technology Research etr, and they do these quarterly surveys of about 1500 CIOs, IT practitioners, and they really have a good pulse on what's happening with spending. And the data shows that containers generally, but specifically Kubernetes is one of the areas that has kind of, it's been off the charts and seen the most significant adoption and velocity particularly, you know, along with cloud. But, but really Kubernetes is just, you know, still up until the right consistently even with, you know, the macro headwinds and all, all of the stuff that we're sick of talking about. But, so what are you doing with Kubernetes in the platform? >>Yeah, it, it's really central to our ability to run the product. When we first started out, we were just on AWS and, and the way we were running was, was a little bit like containers junior. Now we're running Kubernetes everywhere at aws, Azure, Google Cloud. It allows us to have a consistent experience across three different cloud providers and we can manage that in code so our developers can focus on delivering services, not trying to learn the intricacies of Amazon, Azure, and Google and figure out how to deliver services on those three clouds with all of their differences. >>Just to follow up on that, is it, no. So I presume it's sounds like there's a PAs layer there to allow you guys to have a consistent experience across clouds and out to the edge, you know, wherever is that, is that correct? >>Yeah, so we've basically built more or less platform engineering, This is the new hot phrase, you know, it, it's, Kubernetes has made a lot of things easy for us because we've built a platform that our developers can lean on and they only have to learn one way of deploying their application, managing their application. And so that, that just gets all of the underlying infrastructure out of the way and, and lets them focus on delivering influx cloud. >>Yeah, and I know I'm taking a little bit of a tangent, but is that, that, I'll call it a PAs layer if I can use that term. Is that, are there specific attributes to Influx db or is it kind of just generally off the shelf paths? You know, are there, is, is there any purpose built capability there that, that is, is value add or is it pretty much generic? >>So we really build, we, we look at things through, with a build versus buy through a, a build versus by lens. Some things we want to leverage cloud provider services, for instance, Postgres databases for metadata, perhaps we'll get that off of our plate, let someone else run that. We're going to deploy a platform that our engineers can, can deliver on that has consistency that is, is all generated from code that we can as a, as an SRE group, as an ops team, that we can manage with very few people really, and we can stamp out clusters across multiple regions and in no time. >>So how, so sometimes you build, sometimes you buy it. How do you make those decisions and and what does that mean for the, for the platform and for customers? >>Yeah, so what we're doing is, it's like everybody else will do, we're we're looking for trade offs that make sense. You know, we really want to protect our customers data. So we look for services that support our own software with the most uptime, reliability, and durability we can get. Some things are just going to be easier to have a cloud provider take care of on our behalf. We make that transparent for our own team. And of course for customers you don't even see that, but we don't want to try to reinvent the wheel, like I had mentioned with SQL data stores for metadata, perhaps let's build on top of what of these three large cloud providers have already perfected. And we can then focus on our platform engineering and we can have our developers then focus on the influx data, software, influx, cloud software. >>So take it to the customer level, what does it mean for them? What's the value that they're gonna get out of all these innovations that we've been been talking about today and what can they expect in the future? >>So first of all, people who use the OSS product are really gonna be at home on our cloud platform. You can run it on your desktop machine, on a single server, what have you, but then you want to scale up. We have some 270 terabytes of data across, over 4 billion series keys that people have stored. So there's a proven ability to scale now in terms of the open source, open source software and how we've developed the platform. You're getting highly available high cardinality time series platform. We manage it and, and really as, as I mentioned earlier, we can keep up with the state of the art. We keep reinventing, we keep deploying things in real time. We deploy to our platform every day repeatedly all the time. And it's that continuous deployment that allows us to continue testing things in flight, rolling things out that change new features, better ways of doing deployments, safer ways of doing deployments. >>All of that happens behind the scenes. And like we had mentioned earlier, Kubernetes, I mean that, that allows us to get that done. We couldn't do it without having that platform as a, as a base layer for us to then put our software on. So we, we iterate quickly. When you're on the, the Influx cloud platform, you really are able to, to take advantage of new features immediately. We roll things out every day and as those things go into production, you have, you have the ability to, to use them. And so in the end we want you to focus on getting actual insights from your data instead of running infrastructure, you know, let, let us do that for you. So, >>And that makes sense, but so is the, is the, are the innovations that we're talking about in the evolution of Influx db, do, do you see that as sort of a natural evolution for existing customers? I, is it, I'm sure the answer is both, but is it opening up new territory for customers? Can you add some color to that? >>Yeah, it really is it, it's a little bit of both. Any engineer will say, well, it depends. So cloud native technologies are, are really the hot thing. Iot, industrial iot especially, people want to just shove tons of data out there and be able to do queries immediately and they don't wanna manage infrastructure. What we've started to see are people that use the cloud service as their, their data store backbone and then they use edge computing with R OSS product to ingest data from say, multiple production lines and downsample that data, send the rest of that data off influx cloud where the heavy processing takes place. So really us being in all the different clouds and iterating on that and being in all sorts of different regions allows for people to really get out of the, the business of man trying to manage that big data, have us take care of that. And of course as we change the platform end users benefit from that immediately. And, >>And so obviously taking away a lot of the heavy lifting for the infrastructure, would you say the same thing about security, especially as you go out to IOT and the Edge? How should we be thinking about the value that you bring from a security perspective? >>Yeah, we take, we take security super seriously. It, it's built into our dna. We do a lot of work to ensure that our platform is secure, that the data we store is, is kept private. It's of course always a concern. You see in the news all the time, companies being compromised, you know, that's something that you can have an entire team working on, which we do to make sure that the data that you have, whether it's in transit, whether it's at rest, is always kept secure, is only viewable by you. You know, you look at things like software, bill of materials, if you're running this yourself, you have to go vet all sorts of different pieces of software. And we do that, you know, as we use new tools. That's something that, that's just part of our jobs to make sure that the platform that we're running it has, has fully vetted software and, and with open source especially, that's a lot of work. And so it's, it's definitely new territory. Supply chain attacks are, are definitely happening at a higher clip than they used to, but that is, that is really just part of a day in the, the life for folks like us that are, are building platforms. >>Yeah, and that's key. I mean especially when you start getting into the, the, you know, we talk about IOT and the operations technologies, the engineers running the, that infrastructure, you know, historically, as you know, Tim, they, they would air gap everything. That's how they kept it safe. But that's not feasible anymore. Everything's >>That >>Connected now, right? And so you've gotta have a partner that is again, take away that heavy lifting to r and d so you can focus on some of the other activities. Right. Give us the, the last word and the, the key takeaways from your perspective. >>Well, you know, from my perspective I see it as, as a a two lane approach with, with influx, with Anytime series data, you know, you've got a lot of stuff that you're gonna run on-prem, what you had mentioned, air gaping. Sure there's plenty of need for that, but at the end of the day, people that don't want to run big data centers, people that want torus their data to, to a company that's, that's got a full platform set up for them that they can build on, send that data over to the cloud, the cloud is not going away. I think more hybrid approach is, is where the future lives and that's what we're prepared for. >>Tim, really appreciate you coming to the program. Great stuff. Good to see you. >>Thanks very much. Appreciate it. >>Okay, in a moment I'll be back to wrap up. Today's session, you're watching The Cube. >>Are you looking for some help getting started with InfluxDB Telegraph or Flux Check >>Out Influx DB University >>Where you can find our entire catalog of free training that will help you make the most of your time series data >>Get >>Started for free@influxdbu.com. >>We'll see you in class. >>Okay, so we heard today from three experts on time series and data, how the Influx DB platform is evolving to support new ways of analyzing large data sets very efficiently and effectively in real time. And we learned that key open source components like Apache Arrow and the Rust Programming environment Data fusion par K are being leveraged to support realtime data analytics at scale. We also learned about the contributions in importance of open source software and how the Influx DB community is evolving the platform with minimal disruption to support new workloads, new use cases, and the future of realtime data analytics. Now remember these sessions, they're all available on demand. You can go to the cube.net to find those. Don't forget to check out silicon angle.com for all the news related to things enterprise and emerging tech. And you should also check out influx data.com. There you can learn about the company's products. You'll find developer resources like free courses. You could join the developer community and work with your peers to learn and solve problems. And there are plenty of other resources around use cases and customer stories on the website. This is Dave Valante. Thank you for watching Evolving Influx DB into the smart data platform, made possible by influx data and brought to you by the Cube, your leader in enterprise and emerging tech coverage.

>>This past May, The Cube in collaboration with Influx data shared with you the latest innovations in Time series databases. We talked at length about why a purpose built time series database for many use cases, was a superior alternative to general purpose databases trying to do the same thing. Now, you may, you may remember the time series data is any data that's stamped in time, and if it's stamped, it can be analyzed historically. And when we introduced the concept to the community, we talked about how in theory, those time slices could be taken, you know, every hour, every minute, every second, you know, down to the millisecond and how the world was moving toward realtime or near realtime data analysis to support physical infrastructure like sensors and other devices and IOT equipment. A time series databases have had to evolve to efficiently support realtime data in emerging use cases in iot T and other use cases. >>And to do that, new architectural innovations have to be brought to bear. As is often the case, open source software is the linchpin to those innovations. Hello and welcome to Evolving Influx DB into the smart Data platform, made possible by influx data and produced by the Cube. My name is Dave Valante and I'll be your host today. Now in this program we're going to dig pretty deep into what's happening with Time series data generally, and specifically how Influx DB is evolving to support new workloads and demands and data, and specifically around data analytics use cases in real time. Now, first we're gonna hear from Brian Gilmore, who is the director of IOT and emerging technologies at Influx Data. And we're gonna talk about the continued evolution of Influx DB and the new capabilities enabled by open source generally and specific tools. And in this program you're gonna hear a lot about things like Rust, implementation of Apache Arrow, the use of par k and tooling such as data fusion, which powering a new engine for Influx db. >>Now, these innovations, they evolve the idea of time series analysis by dramatically increasing the granularity of time series data by compressing the historical time slices, if you will, from, for example, minutes down to milliseconds. And at the same time, enabling real time analytics with an architecture that can process data much faster and much more efficiently. Now, after Brian, we're gonna hear from Anna East Dos Georgio, who is a developer advocate at In Flux Data. And we're gonna get into the why of these open source capabilities and how they contribute to the evolution of the Influx DB platform. And then we're gonna close the program with Tim Yokum, he's the director of engineering at Influx Data, and he's gonna explain how the Influx DB community actually evolved the data engine in mid-flight and which decisions went into the innovations that are coming to the market. Thank you for being here. We hope you enjoy the program. Let's get started. Okay, we're kicking things off with Brian Gilmore. He's the director of i t and emerging Technology at Influx State of Bryan. Welcome to the program. Thanks for coming on. >>Thanks Dave. Great to be here. I appreciate the time. >>Hey, explain why Influx db, you know, needs a new engine. Was there something wrong with the current engine? What's going on there? >>No, no, not at all. I mean, I think it's, for us, it's been about staying ahead of the market. I think, you know, if we think about what our customers are coming to us sort of with now, you know, related to requests like sql, you know, query support, things like that, we have to figure out a way to, to execute those for them in a way that will scale long term. And then we also, we wanna make sure we're innovating, we're sort of staying ahead of the market as well and sort of anticipating those future needs. So, you know, this is really a, a transparent change for our customers. I mean, I think we'll be adding new capabilities over time that sort of leverage this new engine, but you know, initially the customers who are using us are gonna see just great improvements in performance, you know, especially those that are working at the top end of the, of the workload scale, you know, the massive data volumes and things like that. >>Yeah, and we're gonna get into that today and the architecture and the like, but what was the catalyst for the enhancements? I mean, when and how did this all come about? >>Well, I mean, like three years ago we were primarily on premises, right? I mean, I think we had our open source, we had an enterprise product, you know, and, and sort of shifting that technology, especially the open source code base to a service basis where we were hosting it through, you know, multiple cloud providers. That was, that was, that was a long journey I guess, you know, phase one was, you know, we wanted to host enterprise for our customers, so we sort of created a service that we just managed and ran our enterprise product for them. You know, phase two of this cloud effort was to, to optimize for like multi-tenant, multi-cloud, be able to, to host it in a truly like sass manner where we could use, you know, some type of customer activity or consumption as the, the pricing vector, you know, And, and that was sort of the birth of the, of the real first influx DB cloud, you know, which has been really successful. >>We've seen, I think like 60,000 people sign up and we've got tons and tons of, of both enterprises as well as like new companies, developers, and of course a lot of home hobbyists and enthusiasts who are using out on a, on a daily basis, you know, and having that sort of big pool of, of very diverse and very customers to chat with as they're using the product, as they're giving us feedback, et cetera, has has, you know, pointed us in a really good direction in terms of making sure we're continuously improving that and then also making these big leaps as we're doing with this, with this new engine. >>Right. So you've called it a transparent change for customers, so I'm presuming it's non-disruptive, but I really wanna understand how much of a pivot this is and what, what does it take to make that shift from, you know, time series, you know, specialist to real time analytics and being able to support both? >>Yeah, I mean, it's much more of an evolution, I think, than like a shift or a pivot. You know, time series data is always gonna be fundamental and sort of the basis of the solutions that we offer our customers, and then also the ones that they're building on the sort of raw APIs of our platform themselves. You know, the time series market is one that we've worked diligently to lead. I mean, I think when it comes to like metrics, especially like sensor data and app and infrastructure metrics, if we're being honest though, I think our, our user base is well aware that the way we were architected was much more towards those sort of like backwards looking historical type analytics, which are key for troubleshooting and making sure you don't, you know, run into the same problem twice. But, you know, we had to ask ourselves like, what can we do to like better handle those queries from a performance and a, and a, you know, a time to response on the queries, and can we get that to the point where the results sets are coming back so quickly from the time of query that we can like limit that window down to minutes and then seconds. >>And now with this new engine, we're really starting to talk about a query window that could be like returning results in, in, you know, milliseconds of time since it hit the, the, the ingest queue. And that's, that's really getting to the point where as your data is available, you can use it and you can query it, you can visualize it, and you can do all those sort of magical things with it, you know? And I think getting all of that to a place where we're saying like, yes to the customer on, you know, all of the, the real time queries, the, the multiple language query support, but, you know, it was hard, but we're now at a spot where we can start introducing that to, you know, a a limited number of customers, strategic customers and strategic availability zones to start. But you know, everybody over time. >>So you're basically going from what happened to in, you can still do that obviously, but to what's happening now in the moment? >>Yeah, yeah. I mean if you think about time, it's always sort of past, right? I mean, like in the moment right now, whether you're talking about like a millisecond ago or a minute ago, you know, that's, that's pretty much right now, I think for most people, especially in these use cases where you have other sort of components of latency induced by the, by the underlying data collection, the architecture, the infrastructure, the, you know, the, the devices and you know, the sort of highly distributed nature of all of this. So yeah, I mean, getting, getting a customer or a user to be able to use the data as soon as it is available is what we're after here. >>I always thought, you know, real, I always thought of real time as before you lose the customer, but now in this context, maybe it's before the machine blows up. >>Yeah, it's, it's, I mean it is operationally or operational real time is different, you know, and that's one of the things that really triggered us to know that we were, we were heading in the right direction, is just how many sort of operational customers we have. You know, everything from like aerospace and defense. We've got companies monitoring satellites, we've got tons of industrial users, users using us as a processes storing on the plant floor, you know, and, and if we can satisfy their sort of demands for like real time historical perspective, that's awesome. I think what we're gonna do here is we're gonna start to like edge into the real time that they're used to in terms of, you know, the millisecond response times that they expect of their control systems, certainly not their, their historians and databases. >>I, is this available, these innovations to influx DB cloud customers only who can access this capability? >>Yeah. I mean commercially and today, yes. You know, I think we want to emphasize that's a, for now our goal is to get our latest and greatest and our best to everybody over time. Of course. You know, one of the things we had to do here was like we double down on sort of our, our commitment to open source and availability. So like anybody today can take a look at the, the libraries in on our GitHub and, you know, can ex inspect it and even can try to, you know, implement or execute some of it themselves in their own infrastructure. You know, we are, we're committed to bringing our sort of latest and greatest to our cloud customers first for a couple of reasons. Number one, you know, there are big workloads and they have high expectations of us. I think number two, it also gives us the opportunity to monitor a little bit more closely how it's working, how they're using it, like how the system itself is performing. >>And so just, you know, being careful, maybe a little cautious in terms of, of, of how big we go with this right away, just sort of both limits, you know, the risk of, of, you know, any issues that can come with new software rollouts. We haven't seen anything so far, but also it does give us the opportunity to have like meaningful conversations with a small group of users who are using the products, but once we get through that and they give us two thumbs up on it, it'll be like, open the gates and let everybody in. It's gonna be exciting time for the whole ecosystem. >>Yeah, that makes a lot of sense. And you can do some experimentation and, you know, using the cloud resources. Let's dig into some of the architectural and technical innovations that are gonna help deliver on this vision. What, what should we know there? >>Well, I mean, I think foundationally we built the, the new core on Rust. You know, this is a new very sort of popular systems language, you know, it's extremely efficient, but it's also built for speed and memory safety, which goes back to that us being able to like deliver it in a way that is, you know, something we can inspect very closely, but then also rely on the fact that it's going to behave well. And if it does find error conditions, I mean we, we've loved working with Go and, you know, a lot of our libraries will continue to, to be sort of implemented in Go, but you know, when it came to this particular new engine, you know, that power performance and stability rust was critical. On top of that, like, we've also integrated Apache Arrow and Apache Parque for persistence. I think for anybody who's really familiar with the nuts and bolts of our backend and our TSI and our, our time series merged Trees, this is a big break from that, you know, arrow on the sort of in MI side and then Par K in the on disk side. >>It, it allows us to, to present, you know, a unified set of APIs for those really fast real time inquiries that we talked about, as well as for very large, you know, historical sort of bulk data archives in that PARQUE format, which is also cool because there's an entire ecosystem sort of popping up around Parque in terms of the machine learning community, you know, and getting that all to work, we had to glue it together with aero flight. That's sort of what we're using as our, our RPC component. You know, it handles the orchestration and the, the transportation of the Coer data. Now we're moving to like a true Coer database model for this, this version of the engine, you know, and it removes a lot of overhead for us in terms of having to manage all that serialization, the deserialization, and, you know, to that again, like blurring that line between real time and historical data. It's, you know, it's, it's highly optimized for both streaming micro batch and then batches, but true streaming as well. >>Yeah. Again, I mean, it's funny you mentioned Rust. It is, it's been around for a long time, but it's popularity is, is you know, really starting to hit that steep part of the S-curve. And, and we're gonna dig into to more of that, but give us any, is there anything else that we should know about Bryan? Give us the last word? >>Well, I mean, I think first I'd like everybody sort of watching just to like take a look at what we're offering in terms of early access in beta programs. I mean, if, if, if you wanna participate or if you wanna work sort of in terms of early access with the, with the new engine, please reach out to the team. I'm sure you know, there's a lot of communications going out and you know, it'll be highly featured on our, our website, you know, but reach out to the team, believe it or not, like we have a lot more going on than just the new engine. And so there are also other programs, things we're, we're offering to customers in terms of the user interface, data collection and things like that. And, you know, if you're a customer of ours and you have a sales team, a commercial team that you work with, you can reach out to them and see what you can get access to because we can flip a lot of stuff on, especially in cloud through feature flags. >>But if there's something new that you wanna try out, we'd just love to hear from you. And then, you know, our goal would be that as we give you access to all of these new cool features that, you know, you would give us continuous feedback on these products and services, not only like what you need today, but then what you'll need tomorrow to, to sort of build the next versions of your business. Because you know, the whole database, the ecosystem as it expands out into to, you know, this vertically oriented stack of cloud services and enterprise databases and edge databases, you know, it's gonna be what we all make it together, not just, you know, those of us who were employed by Influx db. And then finally I would just say please, like watch in ICE in Tim's sessions, like these are two of our best and brightest, They're totally brilliant, completely pragmatic, and they are most of all customer obsessed, which is amazing. And there's no better takes, like honestly on the, the sort of technical details of this, then there's, especially when it comes to like the value that these investments will, will bring to our customers and our communities. So encourage you to, to, you know, pay more attention to them than you did to me, for sure. >>Brian Gilmore, great stuff. Really appreciate your time. Thank you. >>Yeah, thanks Dave. It was awesome. Look forward to it. >>Yeah, me too. Looking forward to see how the, the community actually applies these new innovations and goes, goes beyond just the historical into the real time really hot area. As Brian said in a moment, I'll be right back with Anna East dos Georgio to dig into the critical aspects of key open source components of the Influx DB engine, including Rust, Arrow, Parque, data fusion. Keep it right there. You don't wanna miss this >>Time series Data is everywhere. The number of sensors, systems and applications generating time series data increases every day. All these data sources producing so much data can cause analysis paralysis. Influx DB is an entire platform designed with everything you need to quickly build applications that generate value from time series data influx. DB Cloud is a serverless solution, which means you don't need to buy or manage your own servers. There's no need to worry about provisioning because you only pay for what you use. Influx DB Cloud is fully managed so you get the newest features and enhancements as they're added to the platform's code base. It also means you can spend time building solutions and delivering value to your users instead of wasting time and effort managing something else. Influx TVB Cloud offers a range of security features to protect your data, multiple layers of redundancy ensure you don't lose any data access controls ensure that only the people who should see your data can see it. >>And encryption protects your data at rest and in transit between any of our regions or cloud providers. InfluxDB uses a single API across the entire platform suite so you can build on open source, deploy to the cloud and then then easily query data in the cloud at the edge or on prem using the same scripts. And InfluxDB is schemaless automatically adjusting to changes in the shape of your data without requiring changes in your application. Logic. InfluxDB Cloud is production ready from day one. All it needs is your data and your imagination. Get started today@influxdata.com slash cloud. >>Okay, we're back. I'm Dave Valante with a Cube and you're watching evolving Influx DB into the smart data platform made possible by influx data. Anna ETOs Georgio is here, she's a developer advocate for influx data and we're gonna dig into the rationale and value contribution behind several open source technologies that Influx DB is leveraging to increase the granularity of time series analysis analysis and bring the world of data into real-time analytics and is welcome to the program. Thanks for coming on. >>Hi, thank you so much. It's a pleasure to be here. >>Oh, you're very welcome. Okay, so IX is being touted as this next gen open source core for Influx db. And my understanding is that it leverages in memory of course for speed. It's a kilo store, so it gives you a compression efficiency, it's gonna give you faster query speeds, you store files and object storage, so you got very cost effective approach. Are these the salient points on the platform? I know there are probably dozens of other features, but what are the high level value points that people should understand? >>Sure, that's a great question. So some of the main requirements that IOx is trying to achieve and some of the most impressive ones to me, the first one is that it aims to have no limits on cardinality and also allow you to write any kind of event data that you want, whether that's live tag or a field. It also wants to deliver the best in class performance on analytics queries. In addition to our already well served metrics queries, we also wanna have operator control over memory usage. So you should be able to define how much memory is used for buffering caching and query processing. Some other really important parts is the ability to have bulk data export and import super useful. Also broader ecosystem compatibility where possible we aim to use and embrace emerging standards in the data analytics ecosystem and have compatibility with things like sql, Python, and maybe even pandas in the future. >>Okay, so lot there. Now we talked to Brian about how you're using Rust and which is not a new programming language and of course we had some drama around Rust during the pandemic with the Mozilla layoffs, but the formation of the Rust Foundation really addressed any of those concerns. You got big guns like Amazon and Google and Microsoft throwing their collective weights behind it. It's really, the adoption is really starting to get steep on the S-curve. So lots of platforms, lots of adoption with rust, but why rust as an alternative to say c plus plus for example? >>Sure, that's a great question. So Russ was chosen because of his exceptional performance and reliability. So while Russ is synt tactically similar to c plus plus and it has similar performance, it also compiles to a native code like c plus plus. But unlike c plus plus, it also has much better memory safety. So memory safety is protection against bugs or security vulnerabilities that lead to excessive memory usage or memory leaks. And rust achieves this memory safety due to its like innovative type system. Additionally, it doesn't allow for dangling pointers. And dangling pointers are the main classes of errors that lead to exploitable security vulnerabilities in languages like c plus plus. So Russ like helps meet that requirement of having no limits on ality, for example, because it's, we're also using the Russ implementation of Apache Arrow and this control over memory and also Russ Russ's packaging system called crates IO offers everything that you need out of the box to have features like AY and a weight to fix race conditions, to protection against buffering overflows and to ensure thread safe async cashing structures as well. So essentially it's just like has all the control, all the fine grain control, you need to take advantage of memory and all your resources as well as possible so that you can handle those really, really high ity use cases. >>Yeah, and the more I learn about the, the new engine and, and the platform IOCs et cetera, you know, you, you see things like, you know, the old days not even to even today you do a lot of garbage collection in these, in these systems and there's an inverse, you know, impact relative to performance. So it looks like you really, you know, the community is modernizing the platform, but I wanna talk about Apache Arrow for a moment. It it's designed to address the constraints that are associated with analyzing large data sets. We, we know that, but please explain why, what, what is Arrow and and what does it bring to Influx db? >>Sure, yeah. So Arrow is a, a framework for defining in memory calmer data. And so much of the efficiency and performance of IOx comes from taking advantage of calmer data structures. And I will, if you don't mind, take a moment to kind of of illustrate why column or data structures are so valuable. Let's pretend that we are gathering field data about the temperature in our room and also maybe the temperature of our stove. And in our table we have those two temperature values as well as maybe a measurement value, timestamp value, maybe some other tag values that describe what room and what house, et cetera we're getting this data from. And so you can picture this table where we have like two rows with the two temperature values for both our room and the stove. Well usually our room temperature is regulated so those values don't change very often. >>So when you have calm oriented st calm oriented storage, essentially you take each row, each column and group it together. And so if that's the case and you're just taking temperature values from the room and a lot of those temperature values are the same, then you'll, you might be able to imagine how equal values will then enable each other and when they neighbor each other in the storage format, this provides a really perfect opportunity for cheap compression. And then this cheap compression enables high cardinality use cases. It also enables for faster scan rates. So if you wanna define like the men and max value of the temperature in the room across a thousand different points, you only have to get those a thousand different points in order to answer that question and you have those immediately available to you. But let's contrast this with a row oriented storage solution instead so that we can understand better the benefits of calmer oriented storage. >>So if you had a row oriented storage, you'd first have to look at every field like the temperature in, in the room and the temperature of the stove. You'd have to go across every tag value that maybe describes where the room is located or what model the stove is. And every timestamp you'd then have to pluck out that one temperature value that you want at that one time stamp and do that for every single row. So you're scanning across a ton more data and that's why Rowe Oriented doesn't provide the same efficiency as calmer and Apache Arrow is in memory calmer data, commoner data fit framework. So that's where a lot of the advantages come >>From. Okay. So you basically described like a traditional database, a row approach, but I've seen like a lot of traditional database say, okay, now we've got, we can handle colo format versus what you're talking about is really, you know, kind of native i, is it not as effective? Is the, is the foreman not as effective because it's largely a, a bolt on? Can you, can you like elucidate on that front? >>Yeah, it's, it's not as effective because you have more expensive compression and because you can't scan across the values as quickly. And so those are, that's pretty much the main reasons why, why RO row oriented storage isn't as efficient as calm, calmer oriented storage. Yeah. >>Got it. So let's talk about Arrow Data Fusion. What is data fusion? I know it's written in Rust, but what does it bring to the table here? >>Sure. So it's an extensible query execution framework and it uses Arrow as it's in memory format. So the way that it helps in influx DB IOCs is that okay, it's great if you can write unlimited amount of cardinality into influx Cbis, but if you don't have a query engine that can successfully query that data, then I don't know how much value it is for you. So Data fusion helps enable the, the query process and transformation of that data. It also has a PANDAS API so that you could take advantage of PANDAS data frames as well and all of the machine learning tools associated with Pandas. >>Okay. You're also leveraging Par K in the platform cause we heard a lot about Par K in the middle of the last decade cuz as a storage format to improve on Hadoop column stores. What are you doing with Parque and why is it important? >>Sure. So parque is the column oriented durable file format. So it's important because it'll enable bulk import, bulk export, it has compatibility with Python and Pandas, so it supports a broader ecosystem. Par K files also take very little disc disc space and they're faster to scan because again, they're column oriented in particular, I think PAR K files are like 16 times cheaper than CSV files, just as kind of a point of reference. And so that's essentially a lot of the, the benefits of par k. >>Got it. Very popular. So and he's, what exactly is influx data focusing on as a committer to these projects? What is your focus? What's the value that you're bringing to the community? >>Sure. So Influx DB first has contributed a lot of different, different things to the Apache ecosystem. For example, they contribute an implementation of Apache Arrow and go and that will support clearing with flux. Also, there has been a quite a few contributions to data fusion for things like memory optimization and supportive additional SQL features like support for timestamp, arithmetic and support for exist clauses and support for memory control. So yeah, Influx has contributed a a lot to the Apache ecosystem and continues to do so. And I think kind of the idea here is that if you can improve these upstream projects and then the long term strategy here is that the more you contribute and build those up, then the more you will perpetuate that cycle of improvement and the more we will invest in our own project as well. So it's just that kind of symbiotic relationship and appreciation of the open source community. >>Yeah. Got it. You got that virtuous cycle going, the people call the flywheel. Give us your last thoughts and kind of summarize, you know, where what, what the big takeaways are from your perspective. >>So I think the big takeaway is that influx data is doing a lot of really exciting things with Influx DB IOx and I really encourage, if you are interested in learning more about the technologies that Influx is leveraging to produce IOCs, the challenges associated with it and all of the hard work questions and you just wanna learn more, then I would encourage you to go to the monthly Tech talks and community office hours and they are on every second Wednesday of the month at 8:30 AM Pacific time. There's also a community forums and a community Slack channel look for the influx DDB unders IAC channel specifically to learn more about how to join those office hours and those monthly tech tech talks as well as ask any questions they have about iacs, what to expect and what you'd like to learn more about. I as a developer advocate, I wanna answer your questions. So if there's a particular technology or stack that you wanna dive deeper into and want more explanation about how INFLUX DB leverages it to build IOCs, I will be really excited to produce content on that topic for you. >>Yeah, that's awesome. You guys have a really rich community, collaborate with your peers, solve problems, and, and you guys super responsive, so really appreciate that. All right, thank you so much Anise for explaining all this open source stuff to the audience and why it's important to the future of data. >>Thank you. I really appreciate it. >>All right, you're very welcome. Okay, stay right there and in a moment I'll be back with Tim Yoakum, he's the director of engineering for Influx Data and we're gonna talk about how you update a SAS engine while the plane is flying at 30,000 feet. You don't wanna miss this. >>I'm really glad that we went with InfluxDB Cloud for our hosting because it has saved us a ton of time. It's helped us move faster, it's saved us money. And also InfluxDB has good support. My name's Alex Nada. I am CTO at Noble nine. Noble Nine is a platform to measure and manage service level objectives, which is a great way of measuring the reliability of your systems. You can essentially think of an slo, the product we're providing to our customers as a bunch of time series. So we need a way to store that data and the corresponding time series that are related to those. The main reason that we settled on InfluxDB as we were shopping around is that InfluxDB has a very flexible query language and as a general purpose time series database, it basically had the set of features we were looking for. >>As our platform has grown, we found InfluxDB Cloud to be a really scalable solution. We can quickly iterate on new features and functionality because Influx Cloud is entirely managed, it probably saved us at least a full additional person on our team. We also have the option of running InfluxDB Enterprise, which gives us the ability to even host off the cloud or in a private cloud if that's preferred by a customer. Influx data has been really flexible in adapting to the hosting requirements that we have. They listened to the challenges we were facing and they helped us solve it. As we've continued to grow, I'm really happy we have influx data by our side. >>Okay, we're back with Tim Yokum, who is the director of engineering at Influx Data. Tim, welcome. Good to see you. >>Good to see you. Thanks for having me. >>You're really welcome. Listen, we've been covering open source software in the cube for more than a decade, and we've kind of watched the innovation from the big data ecosystem. The cloud has been being built out on open source, mobile, social platforms, key databases, and of course influx DB and influx data has been a big consumer and contributor of open source software. So my question to you is, where have you seen the biggest bang for the buck from open source software? >>So yeah, you know, influx really, we thrive at the intersection of commercial services and open, so open source software. So OSS keeps us on the cutting edge. We benefit from OSS in delivering our own service from our core storage engine technologies to web services temping engines. Our, our team stays lean and focused because we build on proven tools. We really build on the shoulders of giants and like you've mentioned, even better, we contribute a lot back to the projects that we use as well as our own product influx db. >>You know, but I gotta ask you, Tim, because one of the challenge that that we've seen in particular, you saw this in the heyday of Hadoop, the, the innovations come so fast and furious and as a software company you gotta place bets, you gotta, you know, commit people and sometimes those bets can be risky and not pay off well, how have you managed this challenge? >>Oh, it moves fast. Yeah, that, that's a benefit though because it, the community moves so quickly that today's hot technology can be tomorrow's dinosaur. And what we, what we tend to do is, is we fail fast and fail often. We try a lot of things. You know, you look at Kubernetes for example, that ecosystem is driven by thousands of intelligent developers, engineers, builders, they're adding value every day. So we have to really keep up with that. And as the stack changes, we, we try different technologies, we try different methods, and at the end of the day, we come up with a better platform as a result of just the constant change in the environment. It is a challenge for us, but it's, it's something that we just do every day. >>So we have a survey partner down in New York City called Enterprise Technology Research etr, and they do these quarterly surveys of about 1500 CIOs, IT practitioners, and they really have a good pulse on what's happening with spending. And the data shows that containers generally, but specifically Kubernetes is one of the areas that has kind of, it's been off the charts and seen the most significant adoption and velocity particularly, you know, along with cloud. But, but really Kubernetes is just, you know, still up until the right consistently even with, you know, the macro headwinds and all, all of the stuff that we're sick of talking about. But, so what are you doing with Kubernetes in the platform? >>Yeah, it, it's really central to our ability to run the product. When we first started out, we were just on AWS and, and the way we were running was, was a little bit like containers junior. Now we're running Kubernetes everywhere at aws, Azure, Google Cloud. It allows us to have a consistent experience across three different cloud providers and we can manage that in code so our developers can focus on delivering services, not trying to learn the intricacies of Amazon, Azure, and Google and figure out how to deliver services on those three clouds with all of their differences. >>Just to follow up on that, is it, no. So I presume it's sounds like there's a PAs layer there to allow you guys to have a consistent experience across clouds and out to the edge, you know, wherever is that, is that correct? >>Yeah, so we've basically built more or less platform engineering, This is the new hot phrase, you know, it, it's, Kubernetes has made a lot of things easy for us because we've built a platform that our developers can lean on and they only have to learn one way of deploying their application, managing their application. And so that, that just gets all of the underlying infrastructure out of the way and, and lets them focus on delivering influx cloud. >>Yeah, and I know I'm taking a little bit of a tangent, but is that, that, I'll call it a PAs layer if I can use that term. Is that, are there specific attributes to Influx db or is it kind of just generally off the shelf paths? You know, are there, is, is there any purpose built capability there that, that is, is value add or is it pretty much generic? >>So we really build, we, we look at things through, with a build versus buy through a, a build versus by lens. Some things we want to leverage cloud provider services, for instance, Postgres databases for metadata, perhaps we'll get that off of our plate, let someone else run that. We're going to deploy a platform that our engineers can, can deliver on that has consistency that is, is all generated from code that we can as a, as an SRE group, as an ops team, that we can manage with very few people really, and we can stamp out clusters across multiple regions and in no time. >>So how, so sometimes you build, sometimes you buy it. How do you make those decisions and and what does that mean for the, for the platform and for customers? >>Yeah, so what we're doing is, it's like everybody else will do, we're we're looking for trade offs that make sense. You know, we really want to protect our customers data. So we look for services that support our own software with the most uptime, reliability, and durability we can get. Some things are just going to be easier to have a cloud provider take care of on our behalf. We make that transparent for our own team. And of course for customers you don't even see that, but we don't want to try to reinvent the wheel, like I had mentioned with SQL data stores for metadata, perhaps let's build on top of what of these three large cloud providers have already perfected. And we can then focus on our platform engineering and we can have our developers then focus on the influx data, software, influx, cloud software. >>So take it to the customer level, what does it mean for them? What's the value that they're gonna get out of all these innovations that we've been been talking about today and what can they expect in the future? >>So first of all, people who use the OSS product are really gonna be at home on our cloud platform. You can run it on your desktop machine, on a single server, what have you, but then you want to scale up. We have some 270 terabytes of data across, over 4 billion series keys that people have stored. So there's a proven ability to scale now in terms of the open source, open source software and how we've developed the platform. You're getting highly available high cardinality time series platform. We manage it and, and really as, as I mentioned earlier, we can keep up with the state of the art. We keep reinventing, we keep deploying things in real time. We deploy to our platform every day repeatedly all the time. And it's that continuous deployment that allows us to continue testing things in flight, rolling things out that change new features, better ways of doing deployments, safer ways of doing deployments. >>All of that happens behind the scenes. And like we had mentioned earlier, Kubernetes, I mean that, that allows us to get that done. We couldn't do it without having that platform as a, as a base layer for us to then put our software on. So we, we iterate quickly. When you're on the, the Influx cloud platform, you really are able to, to take advantage of new features immediately. We roll things out every day and as those things go into production, you have, you have the ability to, to use them. And so in the end we want you to focus on getting actual insights from your data instead of running infrastructure, you know, let, let us do that for you. So, >>And that makes sense, but so is the, is the, are the innovations that we're talking about in the evolution of Influx db, do, do you see that as sort of a natural evolution for existing customers? I, is it, I'm sure the answer is both, but is it opening up new territory for customers? Can you add some color to that? >>Yeah, it really is it, it's a little bit of both. Any engineer will say, well, it depends. So cloud native technologies are, are really the hot thing. Iot, industrial iot especially, people want to just shove tons of data out there and be able to do queries immediately and they don't wanna manage infrastructure. What we've started to see are people that use the cloud service as their, their data store backbone and then they use edge computing with R OSS product to ingest data from say, multiple production lines and downsample that data, send the rest of that data off influx cloud where the heavy processing takes place. So really us being in all the different clouds and iterating on that and being in all sorts of different regions allows for people to really get out of the, the business of man trying to manage that big data, have us take care of that. And of course as we change the platform end users benefit from that immediately. And, >>And so obviously taking away a lot of the heavy lifting for the infrastructure, would you say the same thing about security, especially as you go out to IOT and the Edge? How should we be thinking about the value that you bring from a security perspective? >>Yeah, we take, we take security super seriously. It, it's built into our dna. We do a lot of work to ensure that our platform is secure, that the data we store is, is kept private. It's of course always a concern. You see in the news all the time, companies being compromised, you know, that's something that you can have an entire team working on, which we do to make sure that the data that you have, whether it's in transit, whether it's at rest, is always kept secure, is only viewable by you. You know, you look at things like software, bill of materials, if you're running this yourself, you have to go vet all sorts of different pieces of software. And we do that, you know, as we use new tools. That's something that, that's just part of our jobs to make sure that the platform that we're running it has, has fully vetted software and, and with open source especially, that's a lot of work. And so it's, it's definitely new territory. Supply chain attacks are, are definitely happening at a higher clip than they used to, but that is, that is really just part of a day in the, the life for folks like us that are, are building platforms. >>Yeah, and that's key. I mean especially when you start getting into the, the, you know, we talk about IOT and the operations technologies, the engineers running the, that infrastructure, you know, historically, as you know, Tim, they, they would air gap everything. That's how they kept it safe. But that's not feasible anymore. Everything's >>That >>Connected now, right? And so you've gotta have a partner that is again, take away that heavy lifting to r and d so you can focus on some of the other activities. Right. Give us the, the last word and the, the key takeaways from your perspective. >>Well, you know, from my perspective I see it as, as a a two lane approach with, with influx, with Anytime series data, you know, you've got a lot of stuff that you're gonna run on-prem, what you had mentioned, air gaping. Sure there's plenty of need for that, but at the end of the day, people that don't want to run big data centers, people that want torus their data to, to a company that's, that's got a full platform set up for them that they can build on, send that data over to the cloud, the cloud is not going away. I think more hybrid approach is, is where the future lives and that's what we're prepared for. >>Tim, really appreciate you coming to the program. Great stuff. Good to see you. >>Thanks very much. Appreciate it. >>Okay, in a moment I'll be back to wrap up. Today's session, you're watching The Cube. >>Are you looking for some help getting started with InfluxDB Telegraph or Flux Check >>Out Influx DB University >>Where you can find our entire catalog of free training that will help you make the most of your time series data >>Get >>Started for free@influxdbu.com. >>We'll see you in class. >>Okay, so we heard today from three experts on time series and data, how the Influx DB platform is evolving to support new ways of analyzing large data sets very efficiently and effectively in real time. And we learned that key open source components like Apache Arrow and the Rust Programming environment Data fusion par K are being leveraged to support realtime data analytics at scale. We also learned about the contributions in importance of open source software and how the Influx DB community is evolving the platform with minimal disruption to support new workloads, new use cases, and the future of realtime data analytics. Now remember these sessions, they're all available on demand. You can go to the cube.net to find those. Don't forget to check out silicon angle.com for all the news related to things enterprise and emerging tech. And you should also check out influx data.com. There you can learn about the company's products. You'll find developer resources like free courses. You could join the developer community and work with your peers to learn and solve problems. And there are plenty of other resources around use cases and customer stories on the website. This is Dave Valante. Thank you for watching Evolving Influx DB into the smart data platform, made possible by influx data and brought to you by the Cube, your leader in enterprise and emerging tech coverage.

>> Okay, so we heard today from three experts on time series and data, how the InfluxDB platform is evolving to support new ways of analyzing large data sets very efficiently and effectively in realtime. And we learned that key open source components like Apache Arrow and the Rust Programming environment DataFusion parquet are being leveraged to support realtime data analytics at scale. We also learned about the contributions and importance of open source software and how the InfluxDB community is evolving the platform with minimal disruption to support new workloads, new use cases in the future of realtime data analytics. Now remember these sessions, they're all available on demand. You can go to thecube.net to find those. Don't forget to check out siliconangle.com for all the news related to things enterprise and emerging tech. And you should also check out influxdata.com. There you can learn about the company's products, you'll find developer resources like free courses, you can join the developer community and work with your peers to learn and solve problems, and there are plenty of other resources around use cases and customer stories on the website. This is Dave Vellante. Thank you for watching Evolving InfluxDB into the Smart Data Platform, made possible by InfluxData and brought to you by theCUBE, your leader in enterprise and emerging tech coverage.

>> This past May, the Cube, in collaboration with Influx Data shared with you the latest innovations in Time series databases. We talked at length about why a purpose-built time series database for many use cases, was a superior alternative to general purpose databases trying to do the same thing. Now, you may, you may remember that time series data is any data that's stamped in time and if it's stamped, it can be analyzed historically. And when we introduced the concept to the community we talked about how in theory those time slices could be taken, you know every hour, every minute, every second, you know, down to the millisecond and how the world was moving toward realtime or near realtime data analysis to support physical infrastructure like sensors, and other devices and IOT equipment. Time series databases have had to evolve to efficiently support realtime data in emerging use, use cases in IOT and other use cases. And to do that, new architectural innovations have to be brought to bear. As is often the case, open source software is the linchpin to those innovations. Hello and welcome to Evolving Influx DB into the Smart Data platform, made possible by influx data and produced by the cube. My name is Dave Vellante, and I'll be your host today. Now, in this program, we're going to dig pretty deep into what's happening with Time series data generally, and specifically how Influx DB is evolving to support new workloads and demands and data, and specifically around data analytics use cases in real time. Now, first we're going to hear from Brian Gilmore who is the director of IOT and emerging technologies at Influx Data. And we're going to talk about the continued evolution of Influx DB and the new capabilities enabled by open source generally and specific tools. And in this program, you're going to hear a lot about things like rust implementation of Apache Arrow, the use of Parquet and tooling such as data fusion, which are powering a new engine for Influx db. Now, these innovations, they evolve the idea of time series analysis by dramatically increasing the granularity of time series data by compressing the historical time slices if you will, from, for example minutes down to milliseconds. And at the same time, enabling real time analytics with an architecture that can process data much faster and much more efficiently. Now, after Brian, we're going to hear from Anais Dotis-Georgiou who is a developer advocate at Influx Data. And we're going to get into the "why's" of these open source capabilities, and how they contribute to the evolution of the Influx DB platform. And then we're going to close the program with Tim Yocum. He's the director of engineering at Influx Data, and he's going to explain how the Influx DB community actually evolved the data engine in mid-flight and which decisions went into the innovations that are coming to the market. Thank you for being here. We hope you enjoy the program. Let's get started.

(electronic music) >> Hello everybody, welcome back to Boston. This is Dave Vellante and you're watching theCUBE's continuous coverage of AWS re:Inforce 2022. We're here at the convention center in Boston where theCUBE got started in May of 2010. I'm really excited. Lena Smart is here, she's the chief information security officer at MongoDB rocket ship company We covered MongoDB World earlier this year, June, down in New York. Lena, thanks for coming to theCUBE. >> Thank you for having me. >> You're very welcome, I enjoyed your keynote yesterday. You had a big audience, I mean, this is a big deal. >> Yeah. >> This is the cloud security conference, AWS, putting its mark in the sand back in 2019. Of course, a couple of years of virtual, now back in Boston. You talked in your keynote about security, how it used to be an afterthought, used to be the responsibility of a small group of people. >> Yeah. >> You know, it used to be a bolt on. >> Yep. >> That's changed dramatically and that change has really accelerated through the pandemic. >> Yep. >> Just describe that change from your perspective. >> So when I started at MongoDB about three and a half years ago, we had a very strong security program, but it wasn't under one person. So I was their first CISO that they employed. And I brought together people who were already doing security and we employed people from outside the company as well. The person that I employed as my deputy is actually a third time returnee, I guess? So he's worked for, MongoDB be twice before, his name is Chris Sandalo, and having someone of that stature in the company is really helpful to build the security culture that I wanted. That's why I really wanted Chris to come back. He's technically brilliant, but he also knew all the people who'd been there for a while and having that person as a trusted second in command really, really helped me grow the team very quickly. I've already got a reputation as a strong female leader. He had a reputation as a strong technical leader. So us combined is like indestructible, we we're a great team. >> Is your scope of responsibility, obviously you're protecting Mongo, >> Yeah. >> How much of your role extends into the product? >> So we have a product security team that report into Sahir Azam, our chief product officer. I think you even spoke to him. >> Yeah, he's amazing. >> He's awesome, isn't he? He's just fabulous. And so his team, they've got security experts on our product side who are really kind of the customer facing. I'm also to a certain extent customer facing, but the product folks are the absolute experts. They will listen to what our customers need, what they want, and together we can then work out and translate that. I'm also responsible for governance risk and compliance. So there's a large portion of our customers that give us input via that program too. So there's a lot of avenues to allow us to facilitate change in the security field. And I think that's really important. We have to listen to what our customers want, but also internally. You know, what our internal groups need as well to help them grow. >> I remember last year, Re:invent 2021, I was watching a talk on security. It was the, I forget his name, but it was the individual who responsible for data center security. And one of the things he said was, you know, look it's not at the end of the day, the technology's important but it's not the technology. It's how you apply the tools and the practices and the culture- >> Right. That you build in the organization that will ultimately determine how successful you are at decreasing the ROI for the bad guys. >> Yes. >> Let's put it that way. So talk about the challenges of building that culture, how you go about that, and how you sustain that cultural aspect. >> So, I think having the security champion program, so that's just, it's like one of my babies, that and helping underrepresented groups in MongoDB kind of get on in the tech world are both really important to me. And so the security champion program is purely voluntary. We have over a hundred members. And these are people, there's no bar to join. You don't have to be technical. If you're an executive assistant who wants to learn more about security, like my assistant does, you're more than welcome. Up to, we actually people grade themselves, when they join us, we give them a little tick box. Like five is, I walk in security water. One is, I can spell security but I'd like to learn more. Mixing those groups together has been game changing for us. We now have over a hundred people who volunteer their time, with their supervisors permission, they help us with their phishing campaigns, testing AWS tool sets, testing things like queryable encryption. I mean, we have people who have such an in-depth knowledge in other areas of the business that I could never learn, no matter how much time I had. And so to have them- And we have people from product as security champions as well, and security, and legal, and HR, and every department is recognized. And I think almost every geographical location is also recognized. So just to have that scope and depth of people with long tenure in the company, technically brilliant, really want to understand how they can apply the cultural values that we live with each day to make our security program stronger. As I say, that's been a game changer for us. We use it as a feeder program. So we've had five people transfer from other departments into the security and GRC teams through this Champions program. >> Makes a lot of sense. You take somebody who walks on water in security, mix them with somebody who really doesn't know a lot about it but wants to learn and then can ask really basic questions, and then the experts can actually understand better how to communicate. >> Absolutely. >> To that you know that 101 level. >> It's absolutely true. Like my mom lives in her iPad. She worships her iPad. Unfortunately she thinks everything on it is true. And so for me to try and dumb it down, and she's not a dumb person, but for me to try and dumb down the message of most of it's rubbish, mom, Facebook is made up. It's just people telling stories. For me to try and get that over to- So she's a one, and I might be a five, that's hard. That's really hard. And so that's what we're doing in the office as well. It's like, if you can explain to my mother how not everything on the internet is true, we're golden. >> My mom, rest her soul, when she first got a- we got her a Macintosh, this was years and years and years ago, and we were trying to train her over the phone, and said, mom, just grab the mouse. And she's like, I don't like mice. (Lena laughs) There you go. I know, I know, Lena, what that's like. Years ago, it was early last decade, we started to think about, wow, security really has to become a board level item. >> Yeah. >> And it really wasn't- 2010, you know, for certain companies. But really, and so I had the pleasure of interviewing Dr. Robert Gates, who was the defense secretary. >> Yes. >> We had this conversation, and he sits on a number, or sat on a number of boards, probably still does, but he was adamant. Oh, absolutely. Here's how you know, here. This is the criticality. Now it's totally changed. >> Right. >> I mean, it's now a board level item. But how do you communicate to the C-Suite, the board? How often do you do that? What do you recommend is the right regime? And I know there's not any perfect- there's got to be situational, but how do you approach it? >> So I am extremely lucky. We have a very technical board. Our chairman of the board is Tom Killalea. You know, Amazon alum, I mean, just genius. And he, and the rest of the board, it's not like a normal board. Like I actually have the meeting on this coming Monday. So this weekend will be me reading as much stuff as I possibly can, trying to work out what questions they're going to ask me. And it's never a gotcha kind of thing. I've been at board meetings before where you almost feel personally attacked and that's not a good thing. Where, at MongoDB, you can see they genuinely want us to grow and mature. And so I actually meet with our board four times a year, just for security. So we set up our own security meeting just with board members who are specifically interested in security, which is all of them. And so this is actually off cadence. So I actually get their attention for at least an hour once a quarter, which is almost unheard of. And we actually use the AWS memo format. People have a chance to comment and read prior to the meeting. So they know what we're going to talk about and we know what their concerns are. And so you're not going in like, oh my gosh, what what's going to happen for this hour? We come prepared. We have statistics. We can show them where we're growing. We can show them where we need more growth and maturity. And I think having that level of just development of programs, but also the ear of the board has has helped me mature my role 10 times. And then also we have the chance to ask them, well what are your other CISOs doing? You know, they're members of other boards. So I can say to Dave, for example, you know, what's so-and-so doing at Datadog? Or Tom Killelea, what's the CISO of Capital One doing? And they help me make a lot of those connections as well. I mean, the CISO world is small and me being a female in the world with a Scottish accent, I'm probably more memorable than most. So it's like, oh yeah, that's the Irish girl. Yeah. She's Scottish, thank you. But they remember me and I can use that. And so just having all those mentors from the board level down, and obviously Dev is a huge, huge fan of security and GRC. It's no longer that box ticking exercise that I used to feel security was, you know, if you heated your SOC2 type two in FinTech, oh, you were good to go. You know, if you did a HERC set for the power industry. All right, right. You know, we can move on now. It's not that anymore. >> Right. It's every single day. >> Yeah. Of course. Dev is Dev at the Chario. Dev spelled D E V. I spell Dave differently. My Dave. But, Lena, it sounds like you present a combination of metrics, so, the board, you feel like that's appropriate to dig into the metrics. But also I'm presuming you're talking strategy, potentially, you know, gaps- >> Road roadmaps, the whole nine yards. Yep. >> What's the, you know, I look at the budget scenario. At the macro level, CIOs have told us, they came into the year saying, hey we're going to grow spending at the macro, around eight percent, eight and a half percent. That's dialed down a little bit post Ukraine and the whole recession and Fed tightening. So now they're down maybe around six percent. So not dramatically lower, but still. And they tell us security is still the number one priority. >> Yes. >> That's been the case for many, many quarters, and actually years, but you don't have an unlimited budget. >> Sure >> Right. It's not like, oh, here is an open checkbook. >> Right. >> Lena, so, how does Mongo balance that with the other priorities in the organization, obviously, you know, you got to spend money on product, you got to spend money and go to market. What's the climate like now, is it, you know continuing on in 2022 despite some of the macro concerns? Is it maybe tapping the brakes? What's the general sentiment? >> We would never tap the breaks. I mean, this is something that's- So my other half works in the finance industry still. So we have, you know, interesting discussions when it comes to geopolitics and financial politics and you know, Dev, the chairman of the board, all very technical people, get that security is going to be taken advantage of if we're seeing to be tapping the brakes. So it does kind of worry me when I hear other people are saying, oh, we're, you know, we're cutting back our budget. We are not. That being said, you also have to be fiscally responsible. I'm Scottish, we're cheap, really frugal with money. And so I always tell my team: treat this money as if it's your own. As if it's my money. And so when we're buying tool sets, I want to make sure that I'm talking to the CISO, or the CISO of the company that's supplying it, and saying are you giving me the really the best value? You know, how can we maybe even partner with you as a database platform? How could we partner with you, X company, to, you know, maybe we'll give you credits on our platform. If you look to moving to us and then we could have a partnership, and I mean, that's how some of this stuff builds, and so I've been pretty good at doing that. I enjoy doing that. But then also just in terms of being fiscally responsible, yeah, I get it. There's CISOs who have every tool that's out there because it's shiny and it's new and they know the board is never going to say no, but at some point, people will get wise to that and be like, I think we need a new CISO. So it's not like we're going to stop spending it. So we're going to get someone who actually knows how to budget and get us what the best value for money. And so that's always been my view is we're always going to be financed. We're always going to be financed well. But I need to keep showing that value for money. And we do that every board meeting, every Monday when I meet with my boss. I mean, I report to the CFO but I've got a dotted line to the CTO. So I'm, you know, I'm one of the few people at this level that's got my feet in both camps. You know budgets are talked at Dev's level. So, you know, it's really important that we get the spend right. >> And that value is essentially, as I was kind of alluding to before, it's decreasing the value equation for the hackers, for the adversary. >> Hopefully, yes. >> Right? Who's the- of course they're increasingly sophisticated. I want to ask you about your relationship with AWS in this context. It feels like, when I look around here, I think back to 2019, there was a lot of talk about the shared responsibility model. >> Yes. >> You know, AWS likes to educate people and back then it was like, okay, hey, by the way, you know you got to, you know, configure the S3 bucket properly. And then, oh, by the way, there's more than just, it's not just binary. >> Right, right. >> There's other factors involved. The application access and identity and things like that, et cetera, et cetera. So that was all kind of cool. But I feel like the cloud is becoming the first line of defense for the CISO but because of the shared responsibility model, CISO is now the second line of defense >> Yes. Does that change your role? Does it make it less complicated in a way? Maybe, you know, more complicated because you now got to get your DevSecOps team? The developers are now much more involved in security? How is that shifting, specifically in the context of your relationship with AWS? >> It's honestly not been that much of a shift. I mean, these guys are very proactive when it comes to where we are from the security standpoint. They listen to their customers as much as we do. So when we sit down with them, when I meet with Steve Schmidt or CJ or you know, our account manager, its not a conversation that's a surprise to me when I tell them this is what we need. They're like, yep, we're on that already. And so I think that relationship has been very proactive rather than reactive. And then in terms of MongoDB, as a tech company, security is always at the forefront. So it's not been a huge lift for me. It's really just been my time that I've taken to understand where DevSecOps is coming from. And you know, how far are we shifting left? Are we actually shifting right now? It's like, you know, get the balance, right? You can't be too much to one side. But I think in terms of where we're teaching the developers, you know, we are a company by developers for developers. So, we get it, we understand where they're coming from, and we try and be as proactive as AWS is. >> When you obviously the SolarWinds hack was a a major mile- I think in security, there's always something in the headlines- >> Yes. But when you think of things like, you know, Stuxnet, you know, Log4J, obviously Solarwinds and the whole supply chain infiltration and the bill of materials. As I said before, the adversary is extremely capable and sophisticated and you know, much more automated. It's always been automated attacks, but you know island hopping and infiltrating and self-forming malware and really sophisticated techniques. >> Yep. >> How are you thinking about that supply chain, bill of materials from inside Mongo and ultimately externally to your customers? >> So you've picked on my third favorite topic to talk about. So I came from the power industry before, so I've got a lot of experience with critical infrastructure. And that was really, I think, where a lot of the supply chain management rules and regulations came from. If you're building a turbine and the steel's coming from China, we would send people to China to make sure that the steel we were buying was the steel we were using. And so that became the H bomb. The hardware bill of materials, bad name. But, you know, we remember what it stood for. And then fast forward: President Biden's executive order. SBOs front and center, cloud first front and center. It's like, this is perfect. And so I was actually- I actually moderated a panel earlier this year at Homeland Security Week in DC, where we had a sneak CISA, So Dr. Allen Friedman from CISA, and also Patrick Weir from OWASP for the framework, CISA for the framework as well, and just the general guidance, and Snake for the front end. That was where my head was going. And MongoDB is the back-end database. And what we've done is we've taken our work with Snake and we now have a proof of concept for SBOs. And so I'm now trying to kind of package that, if you like, as a program and get the word out that SBOs shouldn't be something to be afraid of. If you want to do business with the government you're going to have to create one. We are offering a secure repository to store that data, the government could have access to that repository and see that data. So there's one source of truth. And so I think SBOs is going to be really interesting. I know that, you know, some of my peers are like, oh, it's just another box to tick. And I think it's more than that. I definitely- I've just, there's something percolating in the back of my mind that this is going to be big and we're going to be able to use it to hopefully not stop things like another Log4j, there's always going to be another Log4j, we know that. we don't know everything, the unknown unknown, but at least if we're prepared to go find stuff quicker than we were then before Log4j, I think having SBOs on hand, having that one source of truth, that one repository, I think is going to make it so much easier to find those things. >> Last question, what's the CISO's number one challenge? Either yours or the CISO, generally. >> Keeping up with the fire hose that is security. Like, what do you pick tomorrow? And if you pick the wrong thing, what's the impact? So that's why I'm always networking and talking to my peers. And, you know, we're sometimes like meerkats, you know. there's meerkats, you see like this, it's like, what do we talk about? But there's always something to talk about. And you just have to learn and keep learning. >> Last question, part B. As a hot technology company, that's, you know, rising star, you know not withstanding the tech lash and the stock market- >> Yeah. >> But Mongo's growing, you know, wonderfully. Do you find it easier to attract talent? Like many CISOs will say, you know, lack of talent is my biggest, biggest challenge. Do you find that that's not the challenge for you? >> Not at all. I think on two fronts, one, we have the champions program. So we've got a whole internal ecosystem who love working there. So the minute one of my jobs goes on the board, they get first dibs at it. So they'd already phoning their friends. So we've got, you know, there's ripple effects out from over a hundred people internally. You know, I think just having that, that's been a game changer. >> I was so looking forward to interviewing you, Lena, thanks so much for coming. >> Thank you, this was a pleasure. >> It was really great to have you. >> Thank you so much. Thank you. >> You're really welcome. All right, keep it right there. This is Dave Villante for theCUBE. We'll be right back at AWS Re:inforce22 right after this short break.

>> Announcer: From around the globe it's theCUBE with digital coverage of smart data marketplaces brought to you by Io Tahoe. >> Hi everybody, this is Dave Vellante. And welcome back. We've been talking about smart data. We've been hearing Io Tahoe talk about putting data to work and keep heart of building great data outcomes is the Cloud of course, and also Cloud native tooling. Stuti Deshpande is here. She's a partner solutions architect for Amazon Web Services and an expert in this area. Stuti, great to see you. Thanks so much for coming on theCUBE. >> Thank you so much for having me here. >> You're very welcome. So let's talk a little bit about Amazon. I mean, you have been on this machine learning journey for quite sometime. Take us through how this whole evolution has occurred in technology over the period of time. Since the Cloud really has been evolving. >> Amazon in itself is a company, an example of a company that has gotten through a multi year machine learning transformation to become the machine learning driven company that you see today. They have been improvising on original personalization model using robotics to all different women's centers, developing a forecasting system to predict the customer needs and improvising on that and reading customer expectations on convenience, fast delivery and speed, from developing natural language processing technology for end user infraction, to developing a groundbreaking technology such as Prime Air jobs to give packages to the customers. So our goal at Amazon With Services is to take this rich expertise and experience with machine learning technology across Amazon, and to work with thousands of customers and partners to handle this powerful technology into the hands of developers or data engineers of all levels. >> Great. So, okay. So if I'm a customer or a partner of AWS, give me the sales pitch on why I should choose you for machine learning. What are the benefits that I'm going to get specifically from AWS? >> Well, there are three main reasons why partners choose us. First and foremost, we provide the broadest and the deepest set of machine learning and AI services and features for your business. The velocity at which we innovate is truly unmatched. Over the last year, we launched 200 different services and features. So not only our pace is accelerating, but we provide fully managed services to our customers and partners who can easily build sophisticated AI driven applications and utilizing those fully managed services began build and train and deploy machine learning models, which is both valuable and differentiating. Secondly, we can accelerate the adoption of machine learning. So as I mentioned about fully managed services for machine learning, we have Amazon SageMaker. So SageMaker is a fully managed service that are any developer of any level or a data scientist can utilize to build complex machine learning, algorithms and models and deploy that at scale with very less effort and a very less cost. Before SageMaker, it used to take so much of time and expertise and specialization to build all these extensive models, but SageMaker, you can literally build any complex models within just a time of days or weeks. So to increase it option, AWS has acceleration programs just in a solution maps. And we also have education and training programs such as DeepRacer, which are enforces on enforcement learning and Embark, which actually help organization to adopt machine learning very readily. And we also support three major frameworks such as TensorFlow five charge, or they have separate teams who are dedicated to just focus on all these frameworks and improve the support of these frameworks for a wide variety of workloads. And finaly, we provide the most comprehensive platform that is optimized for machine learning. So when you think about machine learning, you need to have a data store where you can store your training sets, your test sets, which is highly reliable, highly scalable, and secure data store. Most of our customers want to store all of their data and any kind of data into a centralized repository that can be treated at the central source of fraud. And in this case from the Amazon Esri data store to build and endurance machine learning workflow. So we believe that we provide this capability of having the most comprehensive platform to build the machine learning workflow from internally. >> Great. Thank you for that. So I wanted, my next question is, this is a complicated situation for a lot of customers. You know, having the technology is one thing, but adoption is sort of everything. So I wonder if you could paint a picture for us and help us understand, how you're helping customers think about machine learning, thinking about that journey and maybe give us the context of what the ecosystem looks like? >> Sure. If someone can put up the belt, I would like to provide a picture representation of how AWS and fusion machine learning as three layers of stack. And moving on to next bill, I can talk about the bottom there. And bottom there as you can see over this screen, it's basically for advanced technologists advanced data scientists who are machine learning practitioners who work at the framework level. 90% of data scientists use multiple frameworks because multiple frameworks are adjusted and are suitable for multiple and different kinds of workloads. So at this layer, we provide support for all of the different types of frameworks. And the bottom layer is only for the advanced scientists and developers who are actually actually want to build, train and deploy these machine learning models by themselves and moving onto the next level, which is the middle layer. This layer is only suited for non-experts. So here we have SageMaker where it provides a fully managed service there you can build, tune, train and deploy your machine learning models at a very low cost and with very minimal efforts and at a higher scale, it removes all the complexity, heavy lifting and guesswork from this stage of machine learning and Amazon SageMaker has been the scene that will change. Many of our customers are actually standardizing on top off Amazon SageMaker. And then I'm moving on to the next layer, which is the top most layer. We call this as AI services because this may make the human recognition. So all of the services mentioned here such as Amazon Rekognition, which is basically a deep learning service optimized for image and video analysis. And then we have Amazon Polly, which can do the text to speech conversion and so on and so forth. So these are the AI services that can be embedded into the application so that the end user or the end customer can build AI driven applications. >> Love it. Okay. So you've got the experts at the bottom with the frameworks, the hardcore data scientists, you kind of get the self driving machine learning in the middle, and then you have all the ingredients. I'm like an AI chef or a machine learning chef. I can pull in vision, speech, chatbots, fraud detection, and sort of compile my own solutions that's cool. We hear a lot about SageMaker studio. I wonder if you could tell us a little bit more, can we double click a little bit on SageMaker? That seems to be a pretty important component of that stack that you just showed us. >> I think that was an absolutely very great summarization of all the different layers of machine unexpected. So thank you for providing the gist of that. Of course, I'll be really happy to talk about Amazon SageMaker because most of our customers are actually standardizing on top of SageMaker. That is spoken about how machine learning traditionally has so many complications and it's very complex and expensive and I traded process, which makes it even harder because they don't know integrated tools or if you do the traditional machine learning all kind of deployment, there are no integrated tools for the entire workflow process and deployment. And that is where SageMaker comes into the picture. SageMaker removes all the heaviness thing and complexities from each step of the deployment of machine learning workflow, how it solves our challenges by providing all of the different components that are optimized for every stage of the workflow into one single tool set. So that models get to production faster and with much less effort and at a lower cost. We really continue to add important (indistinct) leading to Amazon SageMaker. I think last year we announced 50 cubic litres in this far SageMaker being improvised it's features and functionalities. And I would love to call out a couple of those here, SageMaker notebooks, which are just one thing, the prominent notebooks that comes along with easy two instances, I'm sorry for quoting Jarvin here is Amazon Elastic Compute Instances. So you just need to have a one thing deployment and you have the entire SageMaker Notebook Interface, along with the Elastic Compute Instances running that gives you the faster time to production. If you're a machine, if you are a data scientist or a data engineer who worked extensively for machine learning, you must be aware about building training datasets is really complex. So there we have on his own ground truth, that is only for building machine learning training data sets, which can reduce your labeling cost by 70%. And if you perform machine learning and other model technology in general, there are some workflows where you need to do inferences. So there we have inference, Elastic Inference Incense, which you can reduce the cost by 75% by adding a little GP acceleration. Or you can reduce the cost by adding managed squad training, utilizing easy to spot instances. So there are multiple ways that you can reduce the costs and there are multiple ways there you can improvise and speed up your machine, learning deployment and workflow. >> So one of the things I love about, I mean, I'm a prime member who is not right. I love to shop at Amazon. And what I like about it is the consumer experience. It kind of helps me find things that maybe I wasn't aware of, maybe based on other patterns that are going on in the buying community with people that are similar. If I want to find a good book. It's always gives me great reviews and recommendations. So I'm wondering if that applies to sort of the tech world and machine learning, are you seeing any patterns emerge across the various use cases, you have such scale? What can you tell us about that? >> Sure. One of the battles that we have seen all the time is to build scalable layer for any kind of use case. So as I spoke before that as much, I'm really looking to put their data into a single set of depository where they have the single source of truth. So storing of data and any kind of data at any velocity into a single source of would actually help them build models who run on these data and get useful insights out of it. So when you speak about an entry and workflow, using Amazon SageMaker along bigger, scalable analytical tool is actually what we have seen as one of the factors where they can perform some analysis using Amazon SageMaker and build predictive models to say samples, if you want to take a healthcare use case. So they can build a predictive model that can victimize the readmissions of using Amazon SageMaker. So what I mean, to say is, by not moving data around and connecting different services to the same set of source of data, that's tumor avoid creating copies of data, which is very crucial when you are having training data set and test data sets with Amazon SageMaker. And it is highly important to consider this. So the pattern that we have seen is to utilize a central source of depository of data, which could be Amazon Extra. In this scenario, scalable analytical layer along with SageMaker. I would have to code at Intuit for a success story over here. I'm using sandwich, a Amazon SageMaker Intuit had reviews the machine learning deployment time by 90%. So I'm quoting here from six months to one week. And if you think about a healthcare industry, there hadn't been a shift from reactive to predictive care. So utilizing predictive models to accelerate research and discovery of new drugs and new treatments. And you've also observed that nurses were supported by AI tools increase their, their productivity has increased by 50%. I would like to say that one of our customers are really diving deep into the AWS portfolio of machine learning and AI services and including transcribed medical, where they are able to provide some insights so that their customers are getting benefits from them. Most of their customers are healthcare providers and they are able to give some into insights so that they can create some more personalized and improvise patient care. So there you have the end user benefits as well. One of the patterns that I have, I can speak about and what we have seen as well, appearing a predictive model with real time integration into healthcare records will actually help their healthcare provider customers for informed decision making and improvising the personalized patient care. >> That's a great example, several there. And I appreciate that. I mean, healthcare is one of those industries that is just so right for technology ingestion and transformation, that is a great example of how the cloud has really enabled really. I mean, I'm talking about major changes in healthcare with proactive versus reactive. We're talking about lower costs, better health, longer lives is really inspiring to see that evolve. We're going to watch it over the next several years. I wonder if we could close in the marketplace. I've had the pleasure of interviewing Dave McCann, a number of times. He and his team have built just an awesome capability for Amazon and its ecosystem. What about the data products, whether it's SageMaker or other data products in the marketplace, what can you tell us? >> Sure. Either of this market visits are interesting thing. So let me first talk about the AWS marketplace of what, AWS marketplace you can browse and search for hundreds of machine learning algorithms and machine learning, modern packages in a broad range of categories that this company provision, fixed analysis, voice answers, email, video, and it says predictive models and so on and so forth. And all of these models and algorithms can be deployed to a Jupiter notebook, which comes as part of the SageMaker that form. And you can integrate all of these different models and algorithms into our fully managed service, which is Amazon SageMaker to Jupiter notebooks, Sage maker, STK, and even command as well. And this experience is followed by either of those marketplace catalog and API. So you get the same benefits as any other marketplace products, the just seamless deployments and consolidate it. So you get the same benefits as the products and the invest marketplace for your machine learning algorithms and model packages. And this is really important because these can be darkly integrated into our SageMaker platform. And I don't even be honest about the data products as well. And I'm really happy to provide and code one of the example over here in the interest of cooler times and because we are in unprecedented times over here we collaborated with our partners to provide some data products. And one of them is data hub by tablet view that gives you the time series data of phases and depth data gathered from multiple trusted sources. And this is to provide better and informed knowledge so that everyone who was utilizing this product can make some informed decisions and help the community at the end. >> I love it. I love this concept of being able to access the data, algorithms, tooling. And it's not just about the data, it's being able to do something with the data and that we've been talking about injecting intelligence into those data marketplaces. That's what we mean by smart data marketplaces. Stuti Deshpande, thanks so much for coming to theCUBES here, sharing your knowledge and tell us a little bit about AWS. There's a pleasure having you. >> It's my pleasure too. Thank you so much for having me here. >> You're very welcome. And thank you for watching. Keep it right there. We will be right back right after this short break. (soft orchestral music)

>> Announcer: From around the globe it's theCUBE with digital coverage of smart data marketplaces brought to you by Io Tahoe. >> Hi everybody, this is Dave Vellante. And welcome back. We've been talking about smart data. We've been hearing Io Tahoe talk about putting data to work and keep heart of building great data outcomes is the Cloud of course, and also Cloud native tooling. Stuti Deshpande is here. She's a partner solutions architect for Amazon Web Services and an expert in this area. Stuti, great to see you. Thanks so much for coming on theCUBE. >> Thank you so much for having me here. >> You're very welcome. So let's talk a little bit about Amazon. I mean, you have been on this machine learning journey for quite sometime. Take us through how this whole evolution has occurred in technology over the period of time. Since the Cloud really has been evolving. >> Amazon in itself is a company, an example of a company that has gotten through a multi year machine learning transformation to become the machine learning driven company that you see today. They have been improvising on original personalization model using robotics to all different women's centers, developing a forecasting system to predict the customer needs and improvising on that and reading customer expectations on convenience, fast delivery and speed, from developing natural language processing technology for end user infraction, to developing a groundbreaking technology such as Prime Air jobs to give packages to the customers. So our goal at Amazon With Services is to take this rich expertise and experience with machine learning technology across Amazon, and to work with thousands of customers and partners to handle this powerful technology into the hands of developers or data engineers of all levels. >> Great. So, okay. So if I'm a customer or a partner of AWS, give me the sales pitch on why I should choose you for machine learning. What are the benefits that I'm going to get specifically from AWS? >> Well, there are three main reasons why partners choose us. First and foremost, we provide the broadest and the deepest set of machine learning and AI services and features for your business. The velocity at which we innovate is truly unmatched. Over the last year, we launched 200 different services and features. So not only our pace is accelerating, but we provide fully managed services to our customers and partners who can easily build sophisticated AI driven applications and utilizing those fully managed services began build and train and deploy machine learning models, which is both valuable and differentiating. Secondly, we can accelerate the adoption of machine learning. So as I mentioned about fully managed services for machine learning, we have Amazon SageMaker. So SageMaker is a fully managed service that are any developer of any level or a data scientist can utilize to build complex machine learning, algorithms and models and deploy that at scale with very less effort and a very less cost. Before SageMaker, it used to take so much of time and expertise and specialization to build all these extensive models, but SageMaker, you can literally build any complex models within just a time of days or weeks. So to increase it option, AWS has acceleration programs just in a solution maps. And we also have education and training programs such as DeepRacer, which are enforces on enforcement learning and Embark, which actually help organization to adopt machine learning very readily. And we also support three major frameworks that just tensive no charge, or they have separate teams who are dedicated to just focus on all these frameworks and improve the support of these frameworks for a wide variety of workloads. And finaly, we provide the most comprehensive platform that is optimized for machine learning. So when you think about machine learning, you need to have a data store where you can store your training sets, your test sets, which is highly reliable, highly scalable, and secure data store. Most of our customers want to store all of their data and any kind of data into a centralized repository that can be treated at the central source of fraud. And in this case from the Amazon Esri data store to build and endurance machine learning workflow. So we believe that we provide this capability of having the most comprehensive platform to build the machine learning workflow from internally. >> Great. Thank you for that. So I wanted, my next question is, this is a complicated situation for a lot of customers. You know, having the technology is one thing, but adoption is sort of everything. So I wonder if you could paint a picture for us and help us understand, how you're helping customers think about machine learning, thinking about that journey and maybe give us the context of what the ecosystem looks like? >> Sure. If someone can put up the belt, I would like to provide a picture representation of how AWS and fusion machine learning as three layers of stack. And moving on to next bill, I can talk about the bottom there. And bottom there as you can see over this screen, it's basically for advanced technologists advanced data scientists who are machine learning practitioners who work at the framework level. 90% of data scientists use multiple frameworks because multiple frameworks are adjusted and are suitable for multiple and different kinds of workloads. So at this layer, we provide support for all of the different types of frameworks. And the bottom layer is only for the advanced scientists and developers who are actually actually want to build, train and deploy these machine learning models by themselves and moving onto the next level, which is the middle layer. This layer is only suited for non-experts. So here we have seen Jamaica where it provides a fully managed service there you can build, tune, train and deploy your machine learning models at a very low cost and with very minimal efforts and at a higher scale, it removes all the complexity, having a thing and guess guesswork from this stage of machine learning and Amazon SageMaker has been the scene that will change. Many of our customers are actually standardizing on top off Amazon SageMaker. And then I'm moving on to the next layer, which is the top most layer. We call this as AI services because this may make the human recognition. So all of the services mentioned here such as Amazon Rekognition, which is basically a deep learning service optimized for image and video analysis. And then we have Amazon Polly, which can do the text to speech from Russian and so on and so forth. So these are the AI services that can be embedded into the application so that the end user or the end customer can build AI driven applications. >> Love it. Okay. So you've got the experts at the bottom with the frameworks, the hardcore data scientists, you kind of get the self driving machine learning in the middle, and then you have all the ingredients. I'm like an AI chef or a machine learning chef. I can pull in vision, speech, chatbots, fraud detection, and sort of compile my own solutions that's cool. We hear a lot about SageMaker studio. I wonder if you could tell us a little bit more, can we double click a little bit on SageMaker? That seems to be a pretty important component of that stack that you just showed us. >> I think that was an absolutely very great summarization of all the different layers of machine unexpected. So thank you for providing the gist of that. Of course, I'll be really happy to talk about Amazon SageMaker because most of our customers are actually standardizing on top of SageMaker. That is spoken about how machine learning traditionally has so many complications and it's very complex and expensive and I traded process, which makes it even harder because they don't know integrated tools or if you do the traditional machine learning all kind of deployment, there are no integrated tools for the entire workflow process and deployment. And that is where SageMaker comes into the picture. SageMaker removes all the heaviness thing and complexities from each step of the deployment of machine learning workflow, how it solves our challenges by providing all of the different components that are optimized for every stage of the workflow into one single tool set. So that models get to production faster and with much less effort and at a lower cost. We really continue to add important (indistinct) leading to Amazon SageMaker. I think last year we announced 50 cubic litres in this far SageMaker being improvised it's features and functionalities. And I would love to call out a couple of those here, SageMaker notebooks, which are just one thing, the prominent notebooks that comes along with easy two instances, I'm sorry for quoting Jarvin here is Amazon Elastic Compute Instances. So you just need to have a one thing deployment and you have the entire SageMaker Notebook Interface, along with the Elastic Compute Instances running that gives you the faster time to production. If you're a machine, if you are a data scientist or a data engineer who worked extensively for machine learning, you must be aware about building training datasets is really complex. So there we have on his own ground truth, that is only for building machine learning training data sets, which can reduce your labeling cost by 70%. And if you perform machine learning and other model technology in general, there are some workflows where you need to do inferences. So there we have inference, Elastic Inference Incense, which you can reduce the cost by 75% by adding a little GP acceleration. Or you can reduce the cost by adding managed squad training, utilizing easy to spot instances. So there are multiple ways that you can reduce the costs and there are multiple ways there you can improvise and speed up your machine, learning deployment and workflow. >> So one of the things I love about, I mean, I'm a prime member who is not right. I love to shop at Amazon. And what I like about it is the consumer experience. It kind of helps me find things that maybe I wasn't aware of, maybe based on other patterns that are going on in the buying community with people that are similar. If I want to find a good book. It's always gives me great reviews and recommendations. So I'm wondering if that applies to sort of the tech world and machine learning, are you seeing any patterns emerge across the various use cases, you have such scale? What can you tell us about that? >> Sure. One of the battles that we have seen all the time is to build scalable layer for any kind of use case. So as I spoke before that as much, I'm really looking to put their data into a single set of depository where they have the single source of truth. So storing of data and any kind of data at any velocity into a single source of would actually help them build models who run on these data and get useful insights out of it. So when you speak about an entry and workflow, using Amazon SageMaker along bigger, scalable analytical tool is actually what we have seen as one of the factors where they can perform some analysis using Amazon SageMaker and build predictive models to say samples, if you want to take a healthcare use case. So they can build a predictive model that can victimize the readmissions of using Amazon SageMaker. So what I mean, to say is, by not moving data around and connecting different services to the same set of source of data, that's tumor avoid creating copies of data, which is very crucial when you are having training data set and test data sets with Amazon SageMaker. And it is highly important to consider this. So the pattern that we have seen is to utilize a central source of depository of data, which could be Amazon Extra. In this scenario, scalable analytical layer along with SageMaker. I would have to code at Intuit for a success story over here. I'm using sandwich, a Amazon SageMaker Intuit had reviews the machine learning deployment time by 90%. So I'm quoting here from six months to one week. And if you think about a healthcare industry, there hadn't been a shift from reactive to predictive care. So utilizing predictive models to accelerate research and discovery of new drugs and new treatments. And you've also observed that nurses were supported by AI tools increase their, their productivity has increased by 50%. I would like to say that one of our customers are really diving deep into the AWS portfolio of machine learning and AI services and including transcribed medical, where they are able to provide some insights so that their customers are getting benefits from them. Most of their customers are healthcare providers and they are able to give some into insights so that they can create some more personalized and improvise patient care. So there you have the end user benefits as well. One of the patterns that I have, I can speak about and what we have seen as well, appearing a predictive model with real time integration into healthcare records will actually help their healthcare provider customers for informed decision making and improvising the personalized patient care. >> That's a great example, several there. And I appreciate that. I mean, healthcare is one of those industries that is just so right for technology ingestion and transformation, that is a great example of how the cloud has really enabled really. I mean, I'm talking about major changes in healthcare with proactive versus reactive. We're talking about lower costs, better health, longer lives is really inspiring to see that evolve. We're going to watch it over the next several years. I wonder if we could close in the marketplace. I've had the pleasure of interviewing Dave McCann, a number of times. He and his team have built just an awesome capability for Amazon and its ecosystem. What about the data products, whether it's SageMaker or other data products in the marketplace, what can you tell us? >> Sure. Either of this market visits are interesting thing. So let me first talk about the AWS marketplace of what, AWS marketplace you can browse and search for hundreds of machine learning algorithms and machine learning, modern packages in a broad range of categories that this company provision, fixed analysis, voice answers, email, video, and it says predictive models and so on and so forth. And all of these models and algorithms can be deployed to a Jupiter notebook, which comes as part of the SageMaker that form. And you can integrate all of these different models and algorithms into our fully managed service, which is Amazon SageMaker to Jupiter notebooks, Sage maker, STK, and even command as well. And this experience is followed by either of those marketplace catalog and API. So you get the same benefits as any other marketplace products, the just seamless deployments and consolidate it. So you get the same benefits as the products and the invest marketplace for your machine learning algorithms and model packages. And this is really important because these can be darkly integrated into our SageMaker platform. And I don't even be honest about the data products as well. And I'm really happy to provide and code one of the example over here in the interest of cooler times and because we are in unprecedented times over here we collaborated with our partners to provide some data products. And one of them is data hub by tablet view that gives you the time series data of phases and depth data gathered from multiple trusted sources. And this is to provide better and informed knowledge so that everyone who was utilizing this product can make some informed decisions and help the community at the end. >> I love it. I love this concept of being able to access the data, algorithms, tooling. And it's not just about the data, it's being able to do something with the data and that we've been talking about injecting intelligence into those data marketplaces. That's what we mean by smart data marketplaces. Stuti Deshpande, thanks so much for coming to theCUBES here, sharing your knowledge and tell us a little bit about AWS. There's a pleasure having you. >> It's my pleasure too. Thank you so much for having me here. >> You're very welcome. And thank you for watching. Keep it right there. We will be right back right after this short break. (soft orchestral music)

>>from around the globe. It's the Cube with digital coverage of data automated and event. Siri's Brought to You by Iot Tahoe Welcome, everyone to the second episode in our data automated Siri's made possible with support from Iot Tahoe. Today we're gonna drill into the data lifecycle, meaning the sequence of stages that data travels through from creation to consumption to archive. The problem, as we discussed in our last episode, is that data pipelines, they're complicated, They're cumbersome, that disjointed, and they involve highly manual processes. Ah, smart data lifecycle uses automation and metadata to approve agility, performance, data quality and governance and ultimately reduce costs and time to outcomes. Now, in today's session will define the data lifecycle in detail and provide perspectives on what makes a data lifecycle smart and importantly, how to build smarts into your processes. In a moment, we'll be back with Adam Worthington from ethos to kick things off, and then we'll go into an export power panel to dig into the tech behind smart data life cycles, and it will hop into the crowdchat and give you a chance to ask questions. So stay right there. You're watching the cube innovation impact influence. Welcome >>to the Cube disruptors. Developers and practitioners learn from the voices of leaders who share their personal insights from the hottest digital events around the globe. Enjoy the best this community has to offer on the Cube, your global leader. >>High tech digital coverage. Okay, we're back with Adam Worthington. Adam, good to see you. How are things across the pond? >>Thank you, I'm sure. >>Okay, so let's let's set it up. Tell us about yourself. What? Your role is a CTO and >>automatically. As you said, we found a way to have a pretty in company ourselves that we're in our third year on. Do we specialize in emerging disruptive technologies within the infrastructure? That's the kind of cloud space on my phone is the technical lead. So I kind of my job to be an expert in all of the technologies that we work with, which can be a bit of a challenge if you have a huge for phone is one of the reasons, like deliberately focusing on on also kind of pieces a successful validation and evaluation of new technologies. >>So you guys really technology experts, data experts and probably also expert in process and delivering customer outcomes. Right? >>That's a great word there, Dave Outcomes. That's a lot of what I like to speak to customers about. >>Let's talk about smart data, you know, when you when you throw in terms like this is it kind of can feel buzz, wordy. But what are the critical aspects of so called smart data? >>Help to step back a little bit, seen a little bit more in terms of kind of where I can see the types of problems I saw. I'm really an infrastructure solution architect trace on and what I kind of benefit we organically. But over time my personal framework, I focused on three core design principal simplicity, flexibility, inefficient, whatever it was designing. And obviously they need different things, depending on what the technology area is working with. But that's a pretty good. So they're the kind of areas that a smart approach to data will directly address. Reducing silos that comes from simplifying, so moving away from conflict of infrastructure, reducing the amount of copies of data that we have across the infrastructure and reducing the amount of application environments that need different areas so smarter get with data in my eyes anyway, the further we moved away from this. >>But how does it work? I mean, how do you know what's what's involved in injecting smarts into your data lifecycle? >>I think one of my I actually did not ready, but generally one of my favorite quotes from the French lost a mathematician, Blaise Pascal. He said, If I get this right, I have written a short letter, but I didn't have time. But Israel, I love that quite for lots of reasons >>why >>direct application in terms of what we're talking about, it is actually really complicated. These developers technology capabilities to make things simple, more directly meet the needs of the business. So you provide self service capabilities that they just need to stop driving. I mean, making data on infrastructure makes the business users using >>your job. Correct me. If I'm wrong is to kind of put that all together in a solution and then help the customer realize that we talked about earlier that business out. >>Yeah, enough if they said in understanding both sides so that it keeps us on our ability to deliver on exactly what you just said is big experts in the capabilities and new a better way to do things but also having the kind of the business understanding to be able to ask the right questions. That's how new a better price is. Positions another area that I really like his stuff with their platforms. You can do more with less. And that's not just about using data redundancy. That's about creating application environments, that conservative and then the infrastructure to service different requirements that are able to use the random Io thing without getting too kind of low level as well as the sequential. So what that means is you don't necessarily have to move data from application environment a do one thing related, and then move it to the application environment. Be that environment free terms of an analytics on the left Right works. Both keep the data where it is, use it or different different requirements within the infrastructure and again do more with less. And what that does is not just about simplicity and efficiency. It significantly reduces the time to value of that as well. >>Do you have examples that you can share with us even if they're anonymous customers that you work with that are maybe a little further down on the journey. Or maybe not >>looking at the you mentioned data protection earlier. So another organization This is a project which is just kind of hearing confessions moment, huge organization. They're literally petabytes of data that was servicing their back up in archive. And what they have is not just this realization they have combined. I think I different that they have dependent on the what area of infrastructure they were backing up, whether it was virtualization, that was different because they were backing up PC's June 6th. They're backing up another database environment, using something else in the cloud knowledge bases approach that we recommended to work with them on. They were able to significantly reduce complexity and reduce the amount of time that it systems of what they were able to achieve and what this is again. One of the clients have They've gone above the threshold of being able to back up for that. >>Adam, give us the final thoughts, bring us home. In this segment, >>the family built something we didn't particularly such on, that I think it is really barely hidden. It is spoken about as much as I think it is, that agile approaches to infrastructure we're going to be touched on there could be complicated on the lack of it efficient, the impact, a user's ability to be agile. But what you find with traditional approaches and you already touched on some of the kind of benefits new approaches there. It's often very prescriptive, designed for a particular as the infrastructure environment, the way that it served up the users in kind of a packaged. Either way, it means that they need to use it in that whatever wave in data bases, that kind of service of as it comes in from a flexibility standpoint. But for this platform approach, which is the right way to address technology in my eyes enables, it's the infrastructure to be used. Flexible piece of it, the business users of the data users what we find this capability into their innovating in the way they use that on the White House. I bring benefits. This is a platform to prescriptive, and they are able to do that. What you're doing with these new approaches is all of the metrics that we touched on and pass it from a cost standpoint from a visibility standpoint, but what it means is that the innovators in the business want really, is to really understand what they're looking to achieve and now have to to innovate with us. Now, I think I've started to see that with projects season places. If you do it in the right way, you articulate the capability and empower the business users in the right ways. Very significantly. Better position. The advantages on really matching significantly bigger than their competition. Yeah, >>Super Adam in a really exciting space. And we spent the last 10 years gathering all this data, you know, trying to slog through it and figure it out. And now, with the tools that we have and the automation capabilities, it really is a new era of innovation and insights. So, Adam or they didn't thanks so much for coming on the Cube and participating in this program. >>Exciting times with that. Thank you very much Today. >>Now we're going to go into the power panel and go deeper into the technologies that enable smart data life cycles. Stay right there. You're watching the cube. Are >>you interested in test driving? The i o ta ho platform Kickstart the benefits of data automation for your business through the Iot Labs program. Ah, flexible, scalable sandbox environment on the cloud of your choice with set up a service and support provided by Iot. Top. Click on the Link and connect with the data engineer to learn more and see Iot Tahoe in action. >>Welcome back, everybody to the power panel driving business performance with smart data life cycles. Leicester Waters is here. He's the chief technology officer from Iot Tahoe. He's joined by Patrick Smith, who was field CTO from pure storage. And is that data? Who's a system engineering manager at KohI City? Gentlemen, good to see you. Thanks so much for coming on this panel. >>Thank you. >>Let's start with Lester. I wonder if each of you could just give us a quick overview of your role. And what's the number one problem that you're focused on solving for your customers? Let's start with Lester Fleet. >>Yes, I'm Lost Waters, chief technology officer for Iot Tahoe and really the number one problem that we're trying to solve for our customers is to understand, help them understand what they have, because if they don't understand what they have in terms of their data. They can't manage it. They can't control it. The cap monitor. They can't ensure compliance. So really, that's finding all you can about your data that you have. And building a catalog that could be readily consumed by the entire business is what we do. >>Patrick Field, CTO in your title That says to me, You're talking to customers all the time, so you got a good perspective on it. Give us your take on things here. >>Yeah, absolutely. So my patches in here on day talkto customers and prospects in lots of different verticals across the region. And as they look at their environments and their data landscape, they're faced with massive growth in the data that they're trying to analyze and demands to be able to get insight our stuff and to deliver better business value faster than they've ever had to do in the past. So >>got it. And is that of course, Kohi City. You're like the new kid on the block. You guys were really growing rapidly created this whole notion of data management, backup and and beyond. But I'm assistant system engineering manager. What are you seeing from from from customers your role and the number one problem that you're solving. >>Yeah, sure. So the number one problem I see time and again speaking with customers. It's around data fragmentation. So do two things like organic growth, even maybe budgetary limitations. Infrastructure has grown over time very piecemeal, and it's highly distributed internally. And just to be clear, you know, when I say internally, that >>could be >>that it's on multiple platforms or silos within an on Prem infrastructure that it also does extend to the cloud as well. >>Right Cloud is cool. Everybody wants to be in the cloud, right? So you're right, It creates, Ah, maybe unintended consequences. So let's start with the business outcome and kind of try to work backwards to people you know. They want to get more insights from data they want to have. Ah, Mawr efficient data lifecycle. But so let's let me start with you were thinking about like the North Star for creating data driven cultures. You know, what is the North Star or customers >>here? I think the North Star, in a nutshell, is driving value from your data. Without question, I mean way, differentiate ourselves these days by even nuances in our data now, underpinning that, there's a lot of things that have to happen to make that work out. Well, you know, for example, making sure you adequately protect your data, you know? Do you have a good You have a good storage sub system? Do you have a good backup and recovery point objectives? Recovery time objective. How do you Ah, are you fully compliant? Are you ensuring that you're taking all the boxes? There's a lot of regulations these days in terms with respect to compliance, data retention, data, privacy and so forth. Are you taking those boxes? Are you being efficient with your, uh, your your your data? You know, In other words, I think there's a statistic that someone mentioned me the other day that 53% of all businesses have between three and 15 copies of the same data. So you know, finding and eliminating does is it is part of the part of the problem is when you do a chase, >>um, I I like to think of you're right, no doubt, business value and and a lot of that comes from reducing the end in cycle times. But anything that you guys would would add to that. Patrick, Maybe start with Patrick. >>Yeah, I think I think in value from your data really hits on tips on what everyone wants to achieve. But I think there are a couple of key steps in doing that. First of all, is getting access to the data and asked that, Really, it's three big problems, firstly, working out what you've got. Secondly, looking at what? After working on what you've got, how to get access to it? Because it's all very well knowing that you've got some data. But if you can't get access to it either because of privacy reasons, security reasons, then that's a big challenge. And then finally, once you've got access to the data making sure that you can process that data in a timely manner >>for me, you know it would be that an organization has got a really good global view of all of its data. It understands the data flow and dependencies within their infrastructure, understands that precise legal and compliance requirements, and you had the ability to action changes or initiatives within their environment to give the fun. But with a cloud like agility. Um, you know, and that's no easy feat, right? That is hard work. >>Okay, so we've we've talked about. The challenge is in some of the objectives, but there's a lot of blockers out there, and I want to understand how you guys are helping remove them. So So, Lester. But what do you see as some of the big blockers in terms of people really leaning in? So this smart data lifecycle >>yeah, Silos is is probably one of the biggest one I see in business is yes, it's it's my data, not your data. Lots of lots of compartmentalization. Breaking that down is one of the one of the challenges. And having the right tools to help you do that is only part of the solution. There's obviously a lot of cultural things that need to take place Teoh to break down those silos and work together. If you can identify where you have redundant data across your enterprise, you might be able to consolidate those. >>So, Patrick, so one of the blockers that I see is legacy infrastructure, technical debt, sucking all the budget you got. You know, too many people have having to look after, >>as you look at the infrastructure that supports people's data landscapes today for primarily legacy reasons. The infrastructure itself is siloed. So you have different technologies with different underlying hardware and different management methodologies that they're there for good reason, because historically you have to have specific fitness, the purpose for different data requirements. And that's one of the challenges that we tackled head on a pure with with the flash blade technology and the concept of the data, a platform that can deliver in different characteristics for the different workloads. But from a consistent data platform >>now is that I want to go to you because, you know, in the world in your world, which to me goes beyond backup. And one of the challenges is, you know, they say backup is one thing. Recovery is everything, but as well. The the CFO doesn't want to pay for just protection, and one of things that I like about what you guys have done is you. You broadened the perspective to get more value out of your what was once seen as an insurance policy. >>I do see one of the one of the biggest blockers as the fact that the task at hand can, you know, can be overwhelming for customers. But the key here is to remember that it's not an overnight change. It's not, you know, a flick of a switch. It's something that can be tackled in a very piecemeal manner on. Absolutely. Like you said, You know, reduction in TCO and being able to leverage the data for other purposes is a key driver for this. So, you know, this can be this can be resolved. It would be very, you know, pretty straightforward. It can be quite painless as well. Same goes for unstructured data, which is very complex to manage. And, you know, we've all heard the stats from the the analysts. You know, data obviously is growing at an extremely rapid rate, but actually, when you look at that, you know how is actually growing. 80% of that growth is actually in unstructured data, and only 20% of that growth is in unstructured data. S o. You know, these are quick win areas that customers can realize immediate tco improvement and increased agility as well >>paint a picture of this guy that you could bring up the life cycle. You know what you can see here is you've got this this cycle, the data lifecycle and what we're wanting to do is inject intelligence or smarts into this, like like life cycles. You see, you start with ingestion or creation of data. You're you're storing it. You got to put it somewhere, right? You gotta classify it. You got to protect it. And then, of course, you want to reduce the copies, make it, you know, efficient on. And then you want to prepare it so that businesses can actually sumit. And then you've got clients and governance and privacy issues, and I wonder if we could start with you. Lester, this is, you know, the picture of the life cycle. What role does automation play in terms of injecting smarts into the lifecycle? >>Automation is key here, especially from the discover it catalog and classify perspective. I've seen companies where they geo and will take and dump their all of their database scheme is into a spreadsheet so that they can sit down and manually figure out what attributes 37 means for a column names, Uh, and that's that's only the tip of the iceberg. So being able to do automatically detect what you have automatically deduced where what's consuming the data, you know, upstream and downstream. Being able to understand all of the things related to the lifecycle of your data. Back up archive deletion. It is key. And so we're having having good tool. IShares is very >>important. So, Patrick, obviously you participate in the store piece of this picture s I wonder if you could talk more specifically about that. But I'm also interested in how you effect the whole system view the the end end cycle time. >>Yeah, I think Leicester kind of hit the nail on the head in terms of the importance of automation because the data volumes are just just so massive. Now that you can, you can you can effectively manage or understand or catalog your data without automation. Once you understand the data and the value of the data, then that's where you can work out where the data needs to be at any point in >>time, right? So pure and kohi city obviously partner to do that and of course, is that you guys were part of the protect you certainly part of the retain. But Also, you provide data management capabilities and analytics. I wonder if you could add some color there. >>Yeah, absolutely. So, like you said, you know, we focused pretty heavily on data protection. Is just one of our one of our areas on that infrastructure. It is just sitting there, really? Can, you know, with the legacy infrastructure, It's just sitting there, you know, consuming power, space cooling and pretty inefficient. And what, if anything, that protest is a key part of that. If I If I have a modern data platform such as, you know, the cohesive data platform, I can actually do a lot of analytics on that through application. So we have a marketplace for APS. >>I wonder if we could talk about metadata. It's It's increasingly important. Metadata is data about the data, but Leicester maybe explain why it's so important and what role it plays in terms of creating smart data lifecycle. A >>lot of people think it's just about the data itself, but there's a lot of extended characteristics about your data. So so imagine if or my data life cycle I can communicate with the backup system from Kohi City and find out when the last time that data was backed up or where is backed up to. I can communicate exchange data with pure storage and find out what two years? And is the data at the right tier commensurate with its use level pointed out and being able to share that metadata across systems? I think that's the direction that we're going in right now. We're at the stage where just identifying the metadata and trying to bring it together and catalog the next stage will be OK using the AP eyes it that that we have between our systems can't communicate and share that data and build good solutions for customers to use. >>It's a huge point that you just made. I mean, you know, 10 years ago, automating classification was the big problem, and it was machine intelligence, you know, obviously attacking that, But your point about as machines start communicating to each other and you start, it's cloud to cloud. There's all kinds of metadata, uh, kind of new meta data that's being created. I often joke that someday there's gonna be more metadata than data, so that brings us to cloud and that I'd like to start with you. >>You know, I do think, you know, having the cloud is a great thing. And it has got its role to play, and you can have many different permutations and iterations of how you use it on. Um, you know, I may have sort of mentioned previously. You know, I've seen customers go into the cloud very, very quickly, and actually recently, they're starting to remove workloads from the cloud. And the reason why this happens is that, you know, Cloud has got its role to play, but it's not right for absolutely everything, especially in their current form as well. A good analogy I like to use on this may sound a little bit cliche, but you know, when you compare clouds versus on premises data centers, you can use the analogy of houses and hotels. So to give you an idea so you know, when we look at hotels, that's like the equivalent of a cloud, right? I can get everything I need from there. I can get my food, my water, my outdoor facilities. If I need to accommodate more people, I can rent some more rooms. I don't have to maintain the hotel. It's all done for me. When you look at houses the equivalent to on premises infrastructure, I pretty much have to do everything myself, right. So I have to purchase the house. I have to maintain it. I have to buy my own food and water. Eat it. You have to make improvements myself. But then why do we all live in houses? No, in hotels. And the simple answer that I can I can only think of is, is that it's cheaper, right. It's cheaper to do it myself. But that's not to say that hotels haven't got their role to play. Um, you know? So, for example, if I've got loads of visitors coming over for the weekend, I'm not going to go build an extension to my house just for them. I will burst into my hotel into the cloud, um, and use it for, you know, for for things like that. So what I'm really saying is the cloud is great for many things, but it can work out costlier for certain applications, while others are a perfect >>It's an interesting analogy. I hadn't thought of that before, but you're right because I was going to say Well, part of it is you want the cloud experience everywhere, but you don't always want the cloud experience especially, you know, when you're with your family, you want certain privacy that I've not heard that before. He's out. So that's the new perspective s Oh, thank you, but but But Patrick, I do want to come back to that cloud experience because, in fact, that's what's happening. In a lot of cases, organizations are extending the cloud properties of automation on Prem. >>Yeah, I thought, as I thought, a really interesting point and a great analogy for the use of the public cloud. And it really reinforces the importance of the hybrid and multi cloud environment because it gives you the flexibility to choose where is the optimal environment to run your business workloads? And that's what it's all about and the flexibility to change which environment you're running in, either for more months to the next or from one year to the next. Because workloads change and the characteristics that are available in the cloud change, the hybrid cloud is something that we've we've lived with ourselves of pure, So our pure one management technology actually sits in hybrid cloud and what we we started off entirely cloud native. But now we use public cloud for compute. We use our own technology at the end of a high performance network link to support our data platform. So we get the best of both worlds and I think that's where a lot of our customers are trying to get to. >>Alright, I want to come back in a moment there. But before we do, let's see, I wonder if we could talk a little bit about compliance, governance and privacy. I think the Brits hung on. This panel is still in the EU for now, but the you are looking at new rules. New regulations going beyond GDP are where does sort of privacy governance, compliance fit in the data lifecycle, then, is that I want your thoughts on this as well. >>Yeah, this is this is a very important point because the landscape for for compliance, around data privacy and data retention is changing very rapidly. And being able to keep up with those changing regulations in an automated fashion is the only way you're gonna be able to do it. Even I think there's a some sort of Ah, maybe ruling coming out today or tomorrow with the changed in the r. So this is things are all very key points and being able to codify those rules into some software. Whether you know, Iot Tahoe or or your storage system or kohi city, it will help you be compliant is crucial. >>Yeah. Is that anything you can add there? I mean, it's really is your wheelhouse. >>Yeah, absolutely. So, you know, I think anybody who's watching this probably has gotten the message that, you know, less silos is better. And it absolutely it also applies to data in the cloud is where as well. So you know, my aiming Teoh consolidate into fewer platforms, customers can realize a lot better control over their data. And the natural effect of this is that it makes meeting compliance and governance a lot easier. So when it's consolidated, you can start to confidently understand who's accessing your data. How frequently are they accessing the data? You can also do things like, you know, detecting anomalous file access activities and quickly identify potential threats. >>Okay, Patrick, we were talking. You talked earlier about storage optimization. We talked to Adam Worthington about the business case, the numerator, which is the business value, and then the denominator, which is the cost and what's unique about pure in this regard. >>Yeah, and I think there are. There are multiple time dimensions to that. Firstly, if you look at the difference between legacy storage platforms that used to take up racks or aisles of space in the data center, the flash technology that underpins flash blade way effectively switch out racks rack units on. It has a big play in terms of data center footprint, and the environmental is associated with the data center. If you look at extending out storage efficiencies and the benefits it brings, just the performance has a direct effect on start we whether that's, you know, the start from the simplicity that platform so that it's easy and efficient to manage, whether it's the efficiency you get from your data. Scientists who are using the outcomes from the platform, making them more efficient to new. If you look at some of our customers in the financial space there, their time to results are improved by 10 or 20 x by switching to our technology from legacy technologies for their analytics, platforms. >>The guys we've been running, you know, Cube interviews in our studios remotely for the last 120 days is probably the first interview I've done where haven't started off talking about Cove it, Lester. I wonder if you could talk about smart data lifecycle and how it fits into this isolation economy. And hopefully, what will soon be a post isolation economy? >>Yeah, Come. It has dramatically accelerated the data economy. I think. You know, first and foremost, we've all learned to work at home. You know, we've all had that experience where, you know, people would have been all about being able to work at home just a couple days a week. And here we are working five days. That's how to knock on impact to infrastructure, to be able to support that. But going further than that, you know, the data economy is all about how a business can leverage their data to compete in this New World order that we are now in code has really been a forcing function to, you know, it's probably one of the few good things that have come out of government is that we've been forced to adapt and It's a zoo. Been an interesting journey and it continues to be so >>like Lester said, you know, we've We're seeing huge impact here. Working from home has pretty much become the norm. Now, you know, companies have been forced into basically making it work. If you look online retail, that's accelerated dramatically as well. Unified communications and videoconferencing. So really, you know the point here, is that Yes, absolutely. We're you know, we've compressed, you know, in the past, maybe four months. What already would have taken maybe even five years, maybe 10 years or so >>We got to wrap. But Celester Louis, let me ask you to sort of get paint. A picture of the sort of journey the maturity model that people have to take. You know, if they want to get into it, where did they start? And where are they going to give us that view, >>I think, versus knowing what you have. You don't know what you have. You can't manage it. You can't control that. You can't secure what you can't ensure. It's a compliant s so that that's first and foremost. Uh, the second is really, you know, ensuring that your compliance once, once you know what you have. Are you securing it? Are you following the regulatory? The applicable regulations? Are you able to evidence that, uh, how are you storing your data? Are you archiving it? Are you storing it effectively and efficiently? Um, you know, have you Nirvana from my perspective, is really getting to a point where you you've consolidated your data, you've broken down the silos and you have a virtually self service environment by which the business can consume and build upon their data. And really, at the end of the day, as we said at the beginning, it's all about driving value out of your data. And ah, the automation is is key to this, sir. This journey >>that's awesome and you just described is sort of a winning data culture. Lester, Patrick, thanks so much for participating in this power panel. >>Thank you, David. >>Alright, So great overview of the steps in the data lifecycle and how to inject smarts into the process is really to drive business outcomes. Now it's your turn. Hop into the crowd chat, please log in with Twitter or linked in or Facebook. Ask questions, answer questions and engage with the community. Let's crowdchat, right. Yeah, yeah, yeah.