>>Hi, everyone. Welcome to the cubes presentation of the AWS startup showcase the theme. This episode is data as code, and this is season two, episode two of the ongoing series covering exciting startups from the ecosystem in cloud and the future of data analytics. I'm your host, John furry. You're getting great featured panel here with AWS heroes, Lynn blankets, the CEO of Lindbergh Lega consulting, Peter Hanson's, founder of cloud Cedar and Alex debris, principal of debris advisory. Great to see all of you here and, uh, remotely and look forward to see you in person at the next re-invent or other event. >>Thanks for having us. >>So Lynn, you're doing a lot of work in healthcare, Peter you're in the middle of all the action as data as code Alex. You're in deep on the databases. We've got a good round up of, of topics here ranging from healthcare to getting under the hood on databases. So as we'll start with you, what are you working on right now? What trends do you see in the database space? >>Yeah, sure. So I do, uh, I do a lot of consulting work working with different people and, you know, often with, with dynamo DB or, or just general serverless technology type stuff. Um, if you want to talk about trends that I'm seeing right now, I would say trends you're seeing as a lot, just more serverless native databases or cloud native databases where you're seeing these cool databases come out that really take advantage of, uh, this new cloud environment, right? Where you have scalability, you have plasticity of the clouds. So you're not having, you know, instant space environments anymore. You're paying for capacity, you're paying for throughput. You're able to scale up and down. You're not managing individual instances. So a lot of cool stuff that we're seeing, you know, um, with this new generation of, of infrastructure and in particular database is taking advantage of this, this new cloud world >>And really lot deep into the database side in terms of like cloud native impact, diversity of database types, when to use certain databases that also a big deal. >>Yeah, absolutely. I like, I totally agree. I love seeing the different types of databases and, you know, AWS has this whole, uh, purpose-built database strategy. And I think that, that makes a lot of sense. Um, you know, I want to go too far with it. I would, I would more think about purpose-built categories and things like that, you know, specialize in an OLTB database within your, within your organization, whether that's dynamo DB or document DB or relational database Aurora or something like that. But then also choose some sort of analytics database, you know, if it's drew it or Redshift or Athena, and then, you know, if you have some specialized needs, you want to show some real time stuff to your users, check out rock site. If you want to, uh, you know, do some graph analytics, fraud detection, checkout tiger graph, a lot of cool stuff that we're seeing from the startup showcase here. >>Looking forward to unpacking that Lynn you've been in love now, a healthcare action with cloud ops, the pandemic pushes hard core on everybody. What are you working on? >>Yeah, it's all COVID data all the time. Uh, before the pandemic, I was supporting research groups for cancer genomics, which I still do, but, um, what's, uh, impactful is the explosive data volumes. You know, when you there's big data and there's genomic data, you know, I've worked with clients that have broken data centers, broken public cloud provider data centers because of the daily volume they're putting in. So there's this volume aspect. And then there's a collaboration, particularly around COVID research because of pandemic. And so you have this explosive volume, you have this, um, need for, uh, computational complexity. And that means cloud the challenge is it, you know, put the pedal to the metal. So you've got all these bioinformatics researchers that are used to single machine. Suddenly they have to deal with distributed compute. So it's a wild time to be in this space. >>What was the big change that you've seen with the, uh, the pandemic and in genomic cloud genomic specifically what's the big change has happened. >>The amount of data that is being put into the public cloud, um, previously people would have their data on their local, uh, capacity, and then they would publish their paper and the data may or may not become available for, uh, reproducing the research, uh, to accelerate for drug discovery and even variant identification. The data sets are being pushed to public cloud repositories, which is a whole new set of concerns. You have not only dealing with the volume and cost, but security, you know, there's federated security is non-trivial and not well understood by this domain. So there's so much work available here. >>Awesome. Peter, you're doing a lot with the data as a platform kind of view and platform engineering data as code is, is something that's being kicked around. What are you working on and how does platform engineering change as data becomes so much more prevalent in its value proposition? >>Yeah. So I'm the founder of cloud Cedar and, um, we sort of built this company out, this consultancy all around the challenges that a lot of companies have got with getting their data sorted, getting it organized, getting it ready for other use cases, such as analytics and machine learning, um, AI workloads and the like. So typically a platform engineering team will look after the organization of a company infrastructure, making sure that it's coherent across the company and a data platform, engineering teams doing something similar in that sense where they're, they're looking at making sure that, uh, data teams have a solid foundation to build upon, uh, that everything's quite predictable and what that enables is a faster velocity and the ability to use data as code as a way of specifying and onboarding data, building that, translating it, transforming it out into its specific domains and then on to data products. >>I have to ask you while you're here. Um, there's a big trend around data meshes right now. You're hearing, we've had a lot of stuff on the cube. Um, what are practical that people are using data mesh, first of all, is it relevant and how are people looking at this data mesh conversation? >>I think it becomes more and more relevant, uh, the bigger the organization that you're dealing with. So, you know, often times in the enterprise, you've got, uh, projects with timelines of five to 10 years often outlasting technology life cycles. The technology that you're building on is probably irrelevant by the time that you complete it. And what we're seeing is that data engineering teams and data teams more broadly, this organizational bottleneck and data mesh is all about, uh, breaking down that, um, bottleneck and decentralizing the work, shifting that work back onto, uh, development teams who oftentimes have got more of the context and a centralized data engineering team. And we're seeing a lot of, uh, Philocity increases as a result of that. >>It's interesting. There's so many different aspects of how data is changing the world. Lynn talks about the volume with the cloud and genomics. We're hearing data engineering at a platform level. You're talking about slicing and dicing and real-time information. You mentioned rock set, Alex. So I'd like to ask each of you to answer this next question, which is how has the team dynamics changed with data engineering because every single company's impacted. So if you're researchers, Lynn, you're pumping more data into the cloud, that's got a little bit of data engineering to it. Do they even understand that is that impacting them? So how has data changed the responsibilities or roles in this new emerging area of data engineering or whatever you want to call it? Lynn, we'll start with you. What do you, what do you see this impact? >>Well, you know, I mean, dev ops becomes data ops and ML ops and, uh, you know, this is a whole emergent area of work and it starts with an understanding of container technologies, which, you know, in different verticals like FinTech, that's a given, right, but in bioinformatics building an appropriately optimized Docker container is something I'm still working with customers now on because they have the concept of a Docker container is just a virtual machine, which obviously it isn't, or shouldn't be. So, um, you have, again, as I mentioned previously, this humongous skill gap, um, concepts like D, which are prevalent in ad tech FinTech, that's not available yet for most of my customers. So those are the things that I'm building. So the whole ops space is, um, this a wide open area. And really it's a question of practicality. Um, you know, I have, uh, a lot of experience with data lakes and, you know, containerizing and using the data lake platform. But a lot of my customers are going to move to like an interim pass based solutions. If they're using spark, for example, they might use to use a managed spark solution as an interim, um, step up to the cloud before they build their own containers. Because the amount of knowledge to do that effectively is non-trivial >>Peter, you mentioned data, you mentioned data lakes, onboarding data into lake house architectures, for instance, something that you're familiar with. Um, this is not obvious to some verticals obvious to others. What do you see this data engineering impact from a personnel standpoint? And then ultimately how things get built, >>You know, are you directing that to me, >>Peter? >>Yeah. So I think, um, first and foremost, you know, the workload that data engineering teams are dealing with is ever increasing. Usually there's a 10 X ratio of, um, software engineers to data engineers within a business and usually double the amount of analysts to data engineers again. And so they're, they're fighting it ever increasing backload. And, uh, so they're fighting an ever increasing backlog of, of, uh, tasks to do and tickets to, to, to churn through. And so what we're seeing is that data engineering teams are becoming data platform engineering teams where they're building capability instead of constantly hamster wheels spinning if you will. And so with that in mind, with onboarding data into, uh, a Lakehouse architecture or a data lake where data engineering teams, uh, uh, getting wins is developing a very good baseline of structure where they're getting the categorization, the data tagging, whether this data is of a particular domain, does it contain some, um, PII data, for instance, uh, and, and, and, and then the security aspects, and also, you know, the mechanisms on which to do the data transformations, >>Alex, on the database side, those are known personas in an enterprise, a them, the database team, but now the scale is so big. Um, and there's so much going on in databases. How does the data engineering impact organizations from your standpoint? >>Yeah, absolutely. I think definitely, you know, gone are the days where you have a single relational database that is serving operational queries for your users, and you can also serve analytics queries, you know, for your internal teams. It's, it's now split up into those purpose-built databases, like we've said. Uh, but now you've got two different teams managing it and they're, they're designing their data model for different things. You know? So L LLTP might have a more de-normalized model, something that works for very fast operations and it's optimized for that, but now you need to suck that data out and get it elsewhere so that your, your PM or your business analyst, or whoever can crunch through some of that. And, you know, now it needs to be in a more normalized format. How do you sort of bridge that gap? That's a tough one. I think you need to, you know, build empathy on each side of, of what each side is doing and, and build the tools to say, Hey, this is going to help you, uh, you know, LLTP team, if we know what, what users are actually doing, and, and if you can get us into the right format there, so that then I can, you know, we can analyze it, um, on the backend. >>So I think, I think building empathy across those teams is helpful. >>When I left to come back to, you mentioned a health and informatics is coming back. Um, but it's interesting, you know, I look at a database world and you look at the solutions that are out there. A lot of companies that build data solutions don't have a data problem. They've never, they're not swimming in a lot of data, but then you look at like the field that you're working in right now with the genomics and health and, and quantum, they're always, they're dealing with data all the time. So you have people who deal with a lot of data all the time are breaking through New Zealand. People who are don't have that experience are now becoming data full, right? So people are now either it's a first time problem, or they've always been swimming in a ton of data. So it's more of what's the new playbook. And then, wow, I've never had to deal with a lot of data before. What's your take? >>It's interesting. Cause they know, uh, bioinformatics hires, um, uh, grad students. So grad students, you know, use their, our scripts with their file on their laptop. And so, um, to get those folks to understand distributed container-based computing is like I said, a not non-trivial problem. What's been really interesting with the money pouring in to COVID research is when I first started, some of the workflows would take, you know, literally 500 hours and that was just okay. And coming out of FinTech, I was, uh, I could, I was blown away like FinTech is like, could that please take a millisecond rather than a second? Right. And so what has now happened, which makes it, you know, like I said, even more fun to work in this domain is, uh, the research dollars have really gone up because of the pandemic. And so there are, there are, there's this blending of people like me with more of a big data background coming into bioinformatics and working side by side. >>So it's this interesting sort of translation because you have the whole taxonomy of bioinformatics with genomics and sequencers and all the weird file types that you get. And then you have the whole taxonomy of dev ops data ops, you know, containers and Kubernetes and all that. And trying to get that into pipelines that can actually, you know, be efficient, given the constraints. Of course, we, on the tech side, we always want to make it super optimized. I had a customer that we got it down from 500 hours to minutes, but they wanted to stay with the past solution because it was easier for them to go from 500 hours to five hours was good enough, but you know, the techies want to get it down to five minutes. >>This is, this is, we've seen this movie before dev ops, um, edge and op operations, you know, IOT, world scenes, the convergence of cultures. Now you have data and then old, old school operations kind of coming up. So this kind of supports the thesis. That data as code is the next infrastructure as code. What do you guys, what's the reaction there for you guys? What do you think about that? What does data's code mean? If infrastructure's code was cloud and dev ops, what is data as code? What does that mean? >>I could take it if you like. I think, um, data teams, organizations, um, have been long been this bottleneck within the organization and there's like this dark matter of untapped energy and potential waiting to be unleashed a data with the advent of open source projects like DBT, um, have been slowly sort of embracing software development, lifecycle practices. And this is really sort of seeing a, a big steep increase in, um, in their velocity. And, and this is only going to increase and improve as we're seeing data teams, um, embrace starter as code. I think it's, uh, the future is bright for data. So I'm very excited. >>Lynn Peter reaction. I mean, agility data is code is developer concept CICB pipeline. You mentioned it new operational workflows coming into traditional operations reaction. >>Yeah. I mean, I think Peter's right on there. I'd say, you know, some of those tools we're seeing come in from, from software, like, like DBT, basically giving you that infrastructure as code, but applied to that data realm. Also there have been a few, like get for data type things, pack a derm, I believe is one and a few other ones where you bring that in and you also see a lot of immutability concepts flowing into the data realm. So I think just seeing some of those software engineering concepts come over to the data world has, has been pretty interesting >>What we'll literally just versioning datasets and the identification of what's in a data set. What's not in a data set. Some of this is around ethical AI as well, um, is a whole, uh, area that has come out of research groups. Um, mostly AI research groups, but is being applied to medical data and needs to be obviously, um, so this, this, this, um, metadata and versioning around data sets is really, I think, a very of the moment area. >>Yeah, I think we, we, you guys are bringing up a really good kind of direction that's happening in data. And that is something that you're seeing on the software side, open source and now dev ops. And now going to data is that the supply chain challenges of we've been talking about it here on the cube and this, this, um, this episode is, you know, we've seen Ukraine war, but some open source, you know, malware hitting datasets is data secure. What is that going to look like? So you starting to get into this what's the supply chain, is it verified data sets if data sets have to be managed a whole nother level of data supply chain comes up, what do you guys think about that? >>I'll jump in. Oh, sorry. I'll jump in again. I think that, you know, there's, there's, um, some, some of the compliance requirements, um, around financial data are going to be applied to other types of data, probably health data. So immutability reproducibility, um, that is, uh, legally required. Um, also some of the privacy requirements that originated in Europe with GDPR are going to be replicated as more and more, um, types of data. And again, I'm always going to speak for health, but there's other types as well coming out of personal devices and that kind of stuff. So I think, you know, this idea of data as code is it's, it goes down to versioning and controlling and, um, that's, uh, that's sort of a real succinct way to say it that we didn't used to think about that. We just put it in our, you know, relational database and we were good to go, but, um, versioning and controlling in the global ecosystem is kind of, uh, where I'm focusing my efforts. >>It brings up a good question. If databases, if data is going to be part of the development process has to be addressable, which means horizontally scalable. That means it has to be accessible and open. How do you make that work and not foreclose it with a lot of restrictions? >>I think the use of data catalogs and appropriate tagging and categorization, you know, I think, you know, everyone's heard of the term data swamp, and I think that just came about because that everyone saw like, oh, wow, S3, you know, infinite storage. We just, you know, throw whatever in there for as long as we want. And I think at times, you know, the proliferation of S3 buckets, um, and the like, you know, we've just seen, uh, perhaps security, not maintained as well as it could have been. And I think that's kind of where data platform engineering teams have really sort of, uh, come into the, for, you know, creating a governance set of buckets like formation on top. But I think that's kind of where we need to see a lot more work with appropriate tags and also the automatic publishing of metadata into data catalogs so that, um, folks can easily search and address particular data sets and also control the access. You know, for instance, you've got some PII data, perhaps really only your marketing folks should be looking at email addresses and the like not perhaps your finance folks. So I think, you know, there's, there's a lot to be leveraged there in formation and other solutions, >>Alex, let's back up and talk about what's in it for the customer, right. Let's zoom back and saying reality is I just got to get my data to make sure it's secure always on and not going to be hackable. And I just got to get my data available on river performance. So then, then I got to start thinking about, okay, how do I intersect it? So what should teams be thinking about right now as I look up all their data options or databases across their enterprise? >>Yeah, it's, it's a, it's a good question. I just, you know, I think Peter made some good points there and you can think of history as sort of ebbing and flowing between centralization and decentralization a lot of times. And you know, when storage was expensive, data was going to be sort of centralized and Maine maintained, sort of a, you know, by the, uh, the people that are in charge of it. But then when, when S3 comes along, it really decreases storage. Now we can do a lot more experiments on it. We can store a lot more of our data, keep it around and do different things on it. You know, now we've got regulations again, we were, we gotta, we gotta be more realistic about, about keeping that data secure and make sure we're, we're doing the right things with it. So it's, we're gonna probably go through a period of, of centralization as we work out some of this tooling around, you know, tagging and, and ethical AI that, that both Peter. And when we're talking about here and maybe get us into that, that next wearable world of de-centralization again. But I, I think that ebb and flow is going to be natural in response to, you know, the problems of the, the other extreme, >>Where are we in the market right now from progress standpoint, because data lakes don't want to be data swamps. You seeing lake formation as a data architecture, as an example, where are we with customers? What are they doing right now? Where would you put them in the progress bar of, of evolution towards the Nirvana of having this data sovereignty? And this data is code environment. Are they just now in the data lake store, everything real-time and historical? >>Well, I can jump in there. Um, SQL on files is the, is the driver. And so we know when Amazon got Athena, um, that really drove a lot of the customers to really realistically look at data lake technologies, but data warehouses are not going away. And the integration between the two is not seamless. No, we, we are partners with AWS, but we don't work for them. So we can tell you the truth here. Um, there's, there's work to it, but it really, for my customers, it really upped the ante around data lake, uh, because Athena and technologies like that, the serverless, um, SQL queries or the familiar quarry, um, uh, libraries really drove a movement away from either OLTB or OLAP, more expensive, more cumbersome structures, >>But they still need that. Oh, LTP, like if they have high latency issues, they want to be low latency. Can they have the best of both worlds? That's the question. >>I mean, I w I would say we're getting, you know, we're getting closer. We're always going to be, uh, you know, that technology is going to be moving forward, and then we'll just move the goalpost again, in terms of, of what we're asking from it. But I think, you know, the technology that's getting out there, you can get, get really well. And then, you know, just what I work in the dynamo DB world. So you can get really great low latency. So, you know, single digit millisecond LLTP response times on that. I think some of the analytics stuff has been a problem with that. And there, there are different solutions out there to where you can export dynamo to S3, and then you can be doing SQL on your FA your files with Athena Lakeland's talking about, or now you see, you know, rock set of partner here that that'll just ingest your dynamo, DB data, you know, make all those changes. So if you're doing a lot of, uh, changes to your data and dynamo is going to reflect in Roxanna, and then you can do analytics queries, you can do complex filters, different things like that. So, you know, I, I think we continue to push the envelope and then we moved the goalpost again. But, um, you know, I think we're in a, a lot better place than we were a few years ago, for sure. >>Where do you guys see this going relative to the next level? If data as code becomes that next agile, um, software defined environment with open source? Well, all of these new tools with serverless things happening with data lakes are built in with nice architectures with data warehouses, where does it go next? What happens next? If this becomes an agile environment, what's the impact? >>Well, I don't want to be so dominant, but I have, I feel strongly, so I'm going to jump in here. So, so I, um, I feel like, you know, now for my, my, my most computationally intensive workloads, I'm using GPS, I'm bursting to GPU for TensorFlow neural networks. So I've been doing quite a bit of exploration around Amazon bracket for QPS and it's early. Um, and it's specialty. It's not, you know, for everybody. And the learning curve again is pretty daunting, but, um, there are some use cases out there. I mean, I got ahold of a paper where some people did some, um, it was a Q CNN, um, quantum convolutional neural network for lung cancer images, um, from COVID patients and the, the, uh, the QP Hugh, um, algorithm pipeline performed more accurately and faster. So I think, um, bursting to quantum is something to pay attention to. >>Awesome. Peter, what's your take on what's next? >>Well, I think there's still, um, that, that was absolutely fascinating from Lynn, but I think also there's, there's, uh, you know, some more sort of low-level, uh, low-hanging fruit available in, in the data stack. I think there's a lot of, there's still a lot of challenges around the transformation there, getting our data from sort of raw landed data into business domains, and that sort of talks to a lot of what data mesh is all about. I think if we can somehow make that a little more frictionless, because that that's really where the like labor intensive work is. That's, that's kinda dominating, uh, data engineering teams and where we're sort of trying to push that, that workload back onto, um, you know, software engineering teams. >>Alice will give you the final word. What's the impact. What's the next step? What's it look like in the future? >>Yeah, for sure. I mean, I've never had the, uh, breaking a data center problem that wind's had, or the bursting the quantum problem, for sure. But, you know, if you're in that, you know, the pool I swim and of terabytes of data and below and things like that, I think it's a good time. It just like we saw, you know, like we were talking about dev ops and, and pushing, uh, you know, allowing software engineers to handle more of, of the operation stuff. I think the same thing with data can happen where, you know, software engineering teams can handle not just their code, not just, you know, deploying and operating it, but also thinking about their data around the code. And that doesn't mean you won't have people assist you within your organization. You won't have some specialists in there, but I think pushing more stuff, even onto the individual development teams where they have ownership of that. And they're thinking about it through all this different life cycle. I mean, I'm pretty bullish on that. And I think that's an exciting development >>Was that shift, what left with left is security. What does that mean to >>Shipped so much stuff left, but now, you know, the things that were at the end are back at the end again, but, uh, you know, at least we think we can think about that stuff early in the process, which is good, >>Great conversation, very provocative, very realistic and great impact on the future data as code is real, the developers I do believe will have a great operational role and the data stack concept and impacting things like quantum, it's all kind of lining up nicely. Um, and it's a great opportunity to be in this field from a science and policy standpoint. Um, data engineering is legit. It's going to continue to grow and thanks for unpacking that here on the queue. Appreciate it. Okay. Great panel D AWS heroes. They work with AWS and the ecosystem independently out there. They're in the trenches doing the front lines, cracking the code here with data as code season two, episode two of the ongoing series of the 80, but startups I'm John for your host. Thanks for watching.